
Research @cartesia. Prev. co-founder @outerport (YC S24), @UWaterloo intern. Erdos number = 4.
San Francsico, CA
Joined February 2014
- Tweets894
- Following765
- Followers503
- Likes6K
Allen Wang retweeted
that's nothing, check out what we have going on at Cartesia
This slide from the Anthropic IPO roadshow was a big factor in my decision to leave. It’s an insane plot and I’m surprised they made it public.
It shows the compute efficiency of pretraining over time. The y-axis is a **log scale**: they are seeing continued superexponential algorithmic progress.
Mythos was not a one-time deviation, but a point on a superexponential. They could already retrain Mythos with a fraction of the compute originally needed.
It doesn’t strictly imply a software-only intelligence explosion- they have been spending more on R+D. But algorithmic improvements do not require ridiculous creative genius and are on the verge of being automated. You do the math.
AI training data production is in parts starting to look like petroleum engineering.
Some approaches bring in more diverse raw data, and then rely on better filtering and refinement downstream and keep it high quality. Where there're money and skills there's ingenuity.
Allen Wang retweeted
René Girard on why the business world is more creative than the intellectual world:
“Intellectuals are often contemptuous of business and business people; but in business, you can at least distinguish the successful people from the unsuccessful people. If your business is losing money, you will immediately ask what your competitor is doing much better. And you'll start imitating your competitor. But if you are an intellectual, you will manage to rationalize things in such a way that you always want to do the opposite of your competitor. Therefore, you'll hide the fact of imitation better. A businessman who is responsible for a business will immediately imitate his competitor. That's why the world of business today is probably more creative in relative terms than the intellectual world. It is the last creative world because, in intellectual life, you don't dare to imitate others.”
From CBC Ideas
“... I think [this] the meaning of the sacrifice of Isaac, which marks the abandonment of human sacrifice. What's extraordinary about this Biblical text is that it first shows us an Abraham who is still obedient to the system of human sacrifice. It shows the obedience first: it shows that it's out of that obedience that true change becomes possible.”
— René Girard
An ideal world is one where AI is efficient, steerable and interpretable.
The current paradigm where the industry pours billions of dollars into data and compute to build capable but potentially uncontrollable machines is far from that ideal.
We should aim higher than that.
1
498
Bitter lesson just means your method should not be capped when you pore to it more compute and data. It doesn't mean you should not aim for more efficient methods.
Otherwise you would use an infinite look up table.
Allen Wang retweeted
People should always be in control of the future.
OpenAI is against the notion that it would be acceptable for people to be relegated to a role of doing things that “AI systems think makes sense to reward in humans”.
People should not cede control of the future to superintelligence, and that requirement should ultimately constrain AI progress.
Allen Wang retweeted
We asked GPT-6 Astra to edit Qwen3.5-0.8B weights to improve math.
It directly changed only 0.016% of weights, fixing 0.8B's reasoning loops & improving math up to 6x and even instruction following 3x, without gradients or training.
We call this Weight-Space Programming: frontier LLMs can now natively read/write neural network weights! 🧵
Blog: rikiyat.github.io/blog/weigh…
Readers added context they thought people might want to know
LLM was given weights explicitly as text in prompt and used code/forward passes (per authors' logs) to ablate Qwen looping direction, a known thinking-mode issue per model card. Primarily fixed repetition in that mode, not general 'natively' improving weights/math.
rikiyat.github.io/blog/weight-sp…
github.com/RikiyaT/lm_und…
huggingface.co/Qwen/Qwen3.5-0…
huggingface.co/RikiyaT/Qwen3.…
The free market breeds companies that are the most fit for success like the nature does with biological evolution.
2
2
460
When o3 came out, I envisioned a process where AI automatically expands the universe of known mathematics. That it'd put an end to "pure mathematics" as a discipline and the interesting work will be in applied mathematics - directing how the conquest of mathematics goes.
Much of the development of mathematics throughout history was already driven by applications in the natural sciences, finance and engineering. The new era of mathematics can accelerate that. There's an exciting time ahead on this front.
💥 Thrilled to share my new essay "A New Dawn For Mathematical Progress" out today in @NoemaMag! Read the essay here: noemamag.com/
As frontier AI systems solve some of the most difficult open questions in mathematics such as the Millennium Prize problems, many fear human mathematicians are becoming obsolete.
I argue that whether AI ends human mathematics or begins its golden age depends on which of two models of "mathematical progress" we embrace: the checklist model or the generative encounter model.
Right now, the impacts of AI systems on mathematics has left the mathematical community in deep disorientation. A PhD student recently wrote of a "profound spiritual crisis"; a Fields medalist called for "new theories of care and love."
These reactions make sense under the checklist model, where progress consists of resolving well-defined, pre-specified problems (Olympiads, famous conjectures, Millennium Prizes) to crown lone human heroes.
However, the checklist model was always a narrow view; one that elevated "pure" math as the only "real" math, while treating "applied" math as secondary.
As the mathematician George Dantzig once said: "I consider myself a mathematician. I don’t differentiate between pure mathematics and applied mathematics."
History shows that so much of our richest math comes from the generative encounter model on which mathematics makes breakthroughs when human mathematicians tackle complex and messy real world problems.
If we embrace the generative encounter model of mathematical progress, a golden age of mathematics, rooted in human purpose & new visions of education & collective reward, may have just begun.
4
634
Allen Wang retweeted
attempts to “shine intellectually before models overcome” are misguided
that’s a super self centered goal!
there are so many interesting missions to go on. aim for an outcome, rather than fighting the tide for personal recognition. listen to Ilya.
> at larger scale matches an open-source model trained on 11T tokens with ~10x less data and compute.
this is lowkey a huge deal.
I've always believed that intelligence should come from simple, general building blocks that learn their own abstractions directly from raw data; in other words, the bitter lesson. I think we're finally seeing a real step towards AI that can learn from all data.
Language modeling on raw bytes is a microcosm of this ideal, and to me it's a central problem in AI: if we can't build LLMs that learn end-to-end from bytes, how will we handle the rest of the world's data that we understand even less?
Last year we introduced hierarchical networks (H-Nets) as a first step. Our new version finds up to 3-5x efficiency gains in controlled scaling law experiments, and at larger scale matches an open-source model trained on 11T tokens with ~10x less data and compute. The posts below walk through what we found and why we think it works.
3
23
1,261





















