AI Development & Consulting, Engineer, Developer, Traveler, Technologist, @fastdotai Fellow, @matdmiller@bsky.social
United States
Joined March 2008
- Tweets1.4K
- Following2.9K
- Followers1.1K
- Likes77.6K
Pinned Tweet
In my latest blog post, I tackle @karpathy's 'Let's build GPT' to construct a transformer 🤖 model from scratch, leveraging the foundational knowledge I gained from the fantastic @fastdotai courses by @jeremyphoward.
Curious about how transformers work?
blog.matdmiller.com/posts/20…
Mat Miller retweeted
I want to show you guys something really wild.
Claude Opus 5.5 basically one-shotted an explainer video for this paper, and it's incredible!
After watching the video, I feel like I have a decent understanding of how JEM works!
Check it out:
This paper went completely under the radar!
Meta researchers have developed a novel and simple self-supervised learning framework competitive with DINOv2 and DINOv3!
"Scalable Patch-Level Self-Supervised Learning"
The authors introduce JEM, a principled student-teacher method that learns to align patch representations across different views. Additionally, it incorporates explicit regularization to ensure representations are informative and prevent structural collapse.
The framework outperforms DINOv2 in a controlled setting on dense tasks, and even beats the DINOv3 checkpoint despite it being trained on 12x less data.
Mat Miller retweeted
i've been to 132 conferences in 11 countries.
this is the best talk I've ever seen in my life.
a coffin on stage
projection mapping
auto-advancing hands-free slides
and fascinating storytelling
@davekiss's talk had it all.
Mat Miller retweeted
Cancer patients with the same diagnosis often have very different tumor biology.
Molecular testing can reveal underlying differences, but that's slow and expensive.
Introducing Holon-S1, our platform to predict genes and biomarkers from routine pathology slides.
• beats two other models at predicting spatial gene activity in a controlled comparison
• has identified potentially novel biomarkers in breast cancer and colon cancer
• evaluated across 16 organ types and more than 15 scanners, using slides from both formalin-fixed, paraffin-embedded and fresh-frozen tissue.
We believe Holon-S1 has the potential to help researchers find new biomarkers in slides they already have, discover treatment targets, and bring more personalized cancer care within reach!
Mat Miller retweeted
Want to run Qwen3.8 Flash at FP4 w/QAD to maintain high accuracy at low precision?
We have it optimized for RTX 6000, DGX Spark and other Blackwell sm12x hardware.
Check out our Qwen3.8-Flash-Next QAD weights on @NVIDIAAI's page at @huggingface 👇😎
huggingface.co/nvidia/Qwen3.…
Mat Miller retweeted
Dusting off this tweet from April because this gap in shared understanding of LLM capability is *widening*. It's now less "two groups of people speaking past each other" and more a sharp funnel.
- Napkin math somewhere around 6B people (~75% of the population) have barely come in contact with LLMs at all.
- Around 1-2B (~20%) are casual and infrequent users of free-tier ChatGPT-like products. This group sees derpy chatbots and treats them a bit like a better Google search, a writing aid, or etc. Maybe an agent tries to book you a flight. I have non-tech friends who (reasonably, imo) say they have not much use for it at all. Even many professionals outside of math&code are in this tier. For example, execs and many other functions spend a lot of their time talking to other people, so while they understand what is happening intellectually, it is still second-hand and a bit abstract.
- Now we get to professional use of frontier-grade LLMs in math&code. Somewhere around 20M people (0.2%) see first-hand that large, complex projects that used to take them weeks/months can now be completed by agents with a prompt. Building apps, copying apps, translating apps, decompiling apps from binaries... This has all happened very quickly and recently - less than 1 year ago, I was writing code manually by hand, typing memorized computer code commands into a code editor character by character, occasionally pressing Tab to autocomplete a little chunk of code.
- And finally we get to the ~5,000 people (~0.00006%) with access to frontier-grade systems internally. The external world has seen the preview. It looks like swarms of thousands of agents collaborating over weeks on software mega projects: minting zero days, running cyber attacks and defenses at machine speeds, discovering new science, advancing the frontier of mathematics. Things that would have taken top professionals in the industry years of work. Meanwhile, human review and comprehension are starting to fall behind. For example, people are still involved in the "archeology" of the OpenAI-HF incident from many months ago. Mathematicians may be poring over the 722 manuscripts on frontier mathematics for a while.
The funnel is driven by a combination of factors:
1. The impact scales with ambition, problem size, and horizon. A question with a paragraph answer barely stresses the system. You need a reservoir of big, difficult problems that you really care about. This aspect drives the consumer / professional dimension of the funnel.
2. The jaggedness of the system (which I have written about a lot separately). Capability peaks in domains that are digital, verifiable and economically valuable. This is because LLM capability emerges from reinforcement learning on verifiable rewards on a curated environment mixture driven by revenue potential. This aspect primarily drives the area (e.g. math&code) dimension of the funnel.
3. Access. Free-tier, paid-tier, internal.
So this is the weirdness of the moment. The general public has mostly not interacted with these systems. When they have, it looks like a derpy chatbot. The majority of professionals still see only a modest uplift. And a small sliver of professionals are experiencing the vertigo of the curve going vertical. And it is all happening at the same time.
Judging by my tl there is a growing gap in understanding of AI capability.
The first issue I think is around recency and tier of use. I think a lot of people tried the free tier of ChatGPT somewhere last year and allowed it to inform their views on AI a little too much. This is a group of reactions laughing at various quirks of the models, hallucinations, etc. Yes I also saw the viral videos of OpenAI's Advanced Voice mode fumbling simple queries like "should I drive or walk to the carwash". The thing is that these free and old/deprecated models don't reflect the capability in the latest round of state of the art agentic models of this year, especially OpenAI Codex and Claude Code.
But that brings me to the second issue. Even if people paid $200/month to use the state of the art models, a lot of the capabilities are relatively "peaky" in highly technical areas. Typical queries around search, writing, advice, etc. are *not* the domain that has made the most noticeable and dramatic strides in capability. Partly, this is due to the technical details of reinforcement learning and its use of verifiable rewards. But partly, it's also because these use cases are not sufficiently prioritized by the companies in their hillclimbing because they don't lead to as much $$$ value. The goldmines are elsewhere, and the focus comes along.
So that brings me to the second group of people, who *both* 1) pay for and use the state of the art frontier agentic models (OpenAI Codex / Claude Code) and 2) do so professionally in technical domains like programming, math and research. This group of people is subject to the highest amount of "AI Psychosis" because the recent improvements in these domains as of this year have been nothing short of staggering. When you hand a computer terminal to one of these models, you can now watch them melt programming problems that you'd normally expect to take days/weeks of work. It's this second group of people that assigns a much greater gravity to the capabilities, their slope, and various cyber-related repercussions.
TLDR the people in these two groups are speaking past each other. It really is simultaneously the case that OpenAI's free and I think slightly orphaned (?) "Advanced Voice Mode" will fumble the dumbest questions in your Instagram's reels and *at the same time*, OpenAI's highest-tier and paid Codex model will go off for 1 hour to coherently restructure an entire code base, or find and exploit vulnerabilities in computer systems. This part really works and has made dramatic strides because 2 properties: 1) these domains offer explicit reward functions that are verifiable meaning they are easily amenable to reinforcement learning training (e.g. unit tests passed yes or no, in contrast to writing, which is much harder to explicitly judge), but also 2) they are a lot more valuable in b2b settings, meaning that the biggest fraction of the team is focused on improving them. So here we are.
Mat Miller retweeted
new blog post: why taking inference scaling seriously leads to swarms.
contrary to popular belief, I don't think multi-agent systems are a new scaling axis. they're just structured inference scaling.
hallerite.com/posts/on-the-n…
Mat Miller retweeted
Using @pangram on LinkedIn is like seeing the AI matrix
Mat Miller retweeted
Full GLM-5.3 (753B) on 4× DGX Spark ran the Tower of Babel Bench:
one brief, then 6 hours of active time with no human to answer questions.
The task: build the Tower of Babel in Blender "in extreme, exquisite detail" and host a web app showing its progress.
wesche.com/lab/glm5.3local/t…
What it did on its own:
-planned milestones with time boxes and kept a dated journal
-wrote the whole scene as procedural Python: 2,378 objects, 340 unique meshes, 128k faces
-rendered low-res drafts to check its work, then four 1920×1080 Cycles finals
-built and hosted its own web app: renders, 3D viewer, history, plan, live stats
It also realized it had no vision, so it wrote a tool that turns its renders into ASCII brightness maps to "see" them.
@matdmiller bench isn't "can it write code." It tests:
-endurance: working hours alone and pacing against a clock
-time management: slow renders in the background, no waiting around
-context management: 262K window, compaction, helper agents
-3D/spatial reasoning, coding and the final quality
Every call, token, tool use and cache hit is logged.
The run:
-6:00 active hours, 0 crashes, 0 errors
-274 model calls
-178,868 tokens generated (97,653 reasoning)
-10.36M prompt tokens, 92.3% served from cache
-4 finals, 53–168 s each on CPU
GLM-5.3 EXL3 2.75bpw · TensorFold TP4 (@u1tra_instinct image) · 262K context
Mat Miller retweeted
Grateful to see my new book Incorruptible in the #1 Best Seller in Business Management slot on Amazon!
Amazon US has the Kindle at $1.99 and hardcover 21% off right now for Prime Day if you've been meaning to pick up a copy for yourself or a friend.
Kindle: geni.us/4yjqS
Hardcover: buff.ly/8w7ImXn
Mat Miller retweeted
I joined the @NVIDIARTXSpark club and wanted to kick the tires this weekend so I set up a little 3D modeling benchmark for fun and sent @Zai_org GLM 5.3 Flash and @Alibaba_Qwen Qwen 3.8 Next Flash off to create the Tower of Babel. Check it out here: huggingface.co/spaces/matdmi…
I used the @PrimeIntellect agent harness and @MiaAI_lab and @ashxhart Spark Tensor Fold recipes for the runs. Check it out and feel free to run it yourself and contribute your results back if you want.
The main idea for this benchmark was to try out a zero shot agentic task to test out performance, stability and capability of the models, hardware and recipes.
Overall things went pretty well. There were some issues with image prompt caching and the recipes I used but I'm sure they will get smoothed out over time. It'll be fun to try this out on new recipes and tasks to see how things improve over time!
I joined the @NVIDIARTXSpark club and wanted to kick the tires this weekend so I set up a little 3D modeling benchmark for fun and sent @Zai_org GLM 5.3 Flash and @Alibaba_Qwen Qwen 3.8 Next Flash off to create the Tower of Babel. Check it out here: huggingface.co/spaces/matdmi…
I used the @PrimeIntellect agent harness and @MiaAI_lab and @ashxhart Spark Tensor Fold recipes for the runs. Check it out and feel free to run it yourself and contribute your results back if you want.
The main idea for this benchmark was to try out a zero shot agentic task to test out performance, stability and capability of the models, hardware and recipes.
Overall things went pretty well. There were some issues with image prompt caching and the recipes I used but I'm sure they will get smoothed out over time. It'll be fun to try this out on new recipes and tasks to see how things improve over time!
Mat Miller retweeted
My hot take from chatting to @poteto is that we should use MORE abstractions in the AI age
You can use them (combined with harsh lint rules) to reduce the design space available to the agent and constrain them only to good decisions.
Combined with the fact that high-leverage abstractions let you do more with less code - so, more token efficient.
Plus, unwinding the damage from a bad abstraction is much cheaper with agents.
This runs counter to a lot of folks thinking that agents just want to read the raw code. They can, but they're not maximally efficient that way.
Be braver! Design abstractions.
Mat Miller retweeted
Continuation of my old video-as-3d-object demo
Always wanted to extend it further with segmentation so you can search for any type of object within the scene
Mat Miller retweeted
Major update dropping soon, with a lot of fixes, improvements, additions, and a completely new EXL3 quant.
Worth the wait!
Run GLM 5.3 Flash EXL3 with TensorFold ⚡️
This is a completely new recipe that ushers a whole new level of performance for @NVIDIAAI 2x DGX Sparks.
Conservative default for stability:
- 1M context by default
- 2.7M KV cache pool (!)
- Yes, it's not a typo - 2.7M KV in just two Sparks
- 4 concurrent streams by default
Performance:
- 60 tok/s on prose, single stream.
- 108 tok/s on prose, 4 concurrent streams.
~1,950 prefill tok/s for most context lengths.
Stress-tested to handle a variety of workflows.
This is by far the BEST model to run if you have two DGX Sparks. Extremely smooth experience!
Expect further improvements!
Thanks to @ashxhart for developing such a powerful engine! TensorFold will be used in many of my upcoming recipes.
Get it here:
github.com/MiaAI-Lab/GLM-5.3…
Mat Miller retweeted
Today we're unveiling Trillium Labs @trillium_labs, a new non-profit to foster the open science of frontier AI. We're building open post-training recipes and will expand into open infra to study RSI, reward-hacking, multi-agent systems, and whatever comes next.
We're built around the theory of change that you need more eyes to solve hard technical problems. We have faith in the scientific methods and communities that humanity has built, and worry that AI is becoming too closed to utilize them.
Trilliums are wildflowers that bloom briefly in the spring, before the forest canopies fill out. Though they are small, they lay the foundation for the cycles of growth and nourishment through the rest of the year. At Trillium Labs, the recipes will be the slow nutrients for the seasons and the model releases will be the blooms. Building an institution dedicated to this is needed because, much as nature’s trilliums are slow to expand and grow, the open-ecosystem needs time and dedicated resources to catch up.
I co-founded with with a long-time friend and collaborator Tom Zick (@thesezickbeats). We're hiring (full time + student collabs/interns), we're fundraising, and we're looking for compute. Please get in touch if you're interested in helping out. Offices based in the Bay Area and Cambridge MA, remote okay.
I’m in the Bay Area until for The Curve and COLM to connect with people who are interested. We’re thankful to have initial support from Halcyon Futures and Schmidt Sciences with more funding en route to enable our ambitions of scaling. Our advisors @Thom_Wolf, @HannaHajishirzi, @gneubig and @ctnzr have been instrumental to building the ecosystem that exists today, and I’m stoked to get to keep working with them.
Mat Miller retweeted
We let an agent self-improve by exploring and internalizing ”on this sort of task, keep this sort of thing in mind“ bits of self-feedback. It learned to solve tasks where the original policy failed 128 attempts in a row and GRPO flatlined.
Paper: arxiv.org/abs/2609.37633
🧵1/8
Mat Miller retweeted
GPT-6 Astra can build 3D scenes from video. But how can we make them more accurate, physically plausible, and richly interactive?
We tackle this by integrating 3D vision tools (Pi3X, SAM3, GVHMR/Kimodo, and Blender sim) into Astra’s agentic workflow.
👇
kevinxu02.github.io/real2sim…
Mat Miller retweeted
Come check out my website.
It includes a list of my latest & greatest recipes.
The "Install Model" button gives you a ready-to-use prompt for Claude, Codex, or others. It adapts the recipe to your setup, starts the server and tests it.
mia-ai.net/models
