Learning important things deeply.
Tokyo-to, Japan
Joined April 2020
- Tweets601
- Following3.5K
- Followers395
- Likes2.1K
Xiaolong Yang retweeted
AI in Econometric Theory: An Example
Isaiah Andrews, Anna Mikusheva
arxiv.org/abs/2610.35831 [𝚎𝚌𝚘𝚗.𝙴𝙼]
ALT This note contains a story and a paper. The paper is about the admissibility of two-stage least squares with weak instruments, a topic we have worked on together since 2019. The story is about how we (or, rather, our LLMs) wrote the paper, and some thoughts on what this may mean for our field.
Xiaolong Yang retweeted
I'm posting a series of prediction questions from Stanford CS312 (Deep Learning Alchemy) exams to test deep learning intuition.
First up: optimization. If we increase an LM's batch size 8× (same token budget), can rescaling learning rate or weight decay compensate? Try below ↓
This quiz is part of our optimizer hyperparameter assignment and unit designed by @wen_kaiyue
Setup: We train an 8-layer, ~35M-parameter Transformer on 614.4M tokens using AdamW. At batch size 64, the best reference run has peak LR 0.003, weight decay (WD) 0.2, and final validation loss 2.9178.
Now increase the batch size from 64 to 512, which gives 8× fewer optimizer steps. Everything else stays fixed except the changes below.
Which recipe gives the lowest final validation loss?
A. Keep LR and WD unchanged
B. LR ×√8
C. LR ×8
D. WD ×8
E. LR ×√8 and WD ×√8
What would you predict, and why?
visited Berkeley campus back then on my phd admissions day, and the fact that I didn’t find much beauty in many architectures left a deep impression on me..
I visited Wurster Hall at UC Berkeley today. It was designed by, and currently houses, the architecture faculty.
The UC Berkeley website notes that "Wurster is often voted Berkeley's ugliest building."
I visited because I was curious about whether it would in person seem as unattractive as it does in photos. And, in short: it does. It's gaunt, austere, and filmed in grime. On an elysian campus, it stands out for its bleak hostility. It reminds me of the German submarine base in Bordeaux.
The UC Berkeley page agrees that it was never designed to be attractive, quoting the curator of their design archives: “Wurster and his handpicked architects didn’t want anything romantic, or sentimental — they hated the idea of ‘making pretty buildings. They wanted a serious building.”
I guess we have to concede success relative to their stated goals, but it does make one wonder what it all says about architecture as a discipline.
Xiaolong Yang retweeted
I visited Wurster Hall at UC Berkeley today. It was designed by, and currently houses, the architecture faculty.
The UC Berkeley website notes that "Wurster is often voted Berkeley's ugliest building."
I visited because I was curious about whether it would in person seem as unattractive as it does in photos. And, in short: it does. It's gaunt, austere, and filmed in grime. On an elysian campus, it stands out for its bleak hostility. It reminds me of the German submarine base in Bordeaux.
The UC Berkeley page agrees that it was never designed to be attractive, quoting the curator of their design archives: “Wurster and his handpicked architects didn’t want anything romantic, or sentimental — they hated the idea of ‘making pretty buildings. They wanted a serious building.”
I guess we have to concede success relative to their stated goals, but it does make one wonder what it all says about architecture as a discipline.
Xiaolong Yang retweeted
東大法の統計シークエンスのインタビュー、後編が最近出たようです。話すのは勝又さん。
todaishimbun.org/polscidatas…
Xiaolong Yang retweeted
(8/8) I’m not entirely sure why I'm sharing this story.
Maybe it’s to make myself seen, just like how a post back when I only had 20 followers brought me an internship.
Or maybe it can encourage someone else to hit ‘post’.
Still, I'm very grateful for the opportunity.
Xiaolong Yang retweeted
I had 5 papers accepted to NeurIPS 2026, with a total of 12 papers accepted from all DL Theory team members!
aip.riken.jp/news/neurips202…
Xiaolong Yang retweeted
Deep learning is often taught as a sanitized collection of facts and math. But work at the frontier of AI is messy and empirical.
Our new class, Deep Learning Alchemy, aims to teach the art and science of deep learning in the hands-on spirit of Language Modeling From Scratch.
Students build intuition by running experiments on small LMs, then test it through prediction quizzes: given an unseen code diff, can you predict what happens?
deep-learning-alchemy.github…
My co-instructor @kothasuhas and I think there's significant demand for classes that treat students as serious learners who want to be assessed for what they’ve learned, and so we're trying a new exam-heavy approach to the class (7 exams, 85% of the grade, with the remaining 15% in-person discussion assessments).
We’ve designed the class so students can use AI agents to run their experiments and do their assignment, but they then have to demonstrate that mastery on exams without access to the assignment, and any partial credit on the exam has to come through citations to the assignment, incentivizing them to know what’s in their write-up.
The same assignments and quizzes are public for anyone to try (lecture recordings will be released through @StanfordOnline with some delay). We have some very fun assignments and quizzes covering both classic topics (like NQM, run-to-run stochasticity, and scaling laws) and new and timely topics like muP or hyperball. Made by some real experts like @TristanThrush and @wen_kaiyue
Try some quizzes below!
Xiaolong Yang retweeted
Call for Proposals: 14th Annual Meeting of Asian Political Methodology ppa.hku.hk/news-and-events/c…
Xiaolong Yang retweeted
➕➕ New research ➕➕
On the design of agents for markets and markets for agents.
AI agents are increasingly entering markets, acting on behalf of people—
How do we make sure they represent their people well, and that these markets are set up safely and fairly for everyone?
Xiaolong Yang retweeted
I’m excited about the AllBirds AI pivot because this is the first completely pure act of retardmaxxing in the public markets and I’m so on board with watching that journey play out.
Xiaolong Yang retweeted
Exploring random topics, driven by your own curiosity, is the most valuable form of learning. This is where the real outlier ideas come from. Whereas running a startup is the intellectual equivalent of being dragged by a horse.
Xiaolong Yang retweeted
Statistical Rethinking 2026 is done: 20 new lectures emphasizing logical & critical statistical workflow, from basics of probability to causal inference to reliable computation to sensitivity. It's all free, made just for you. Lecture list & links: github.com/rmcelreath/stat_r…
Xiaolong Yang retweeted
広中平祐氏が死去 数学者、フィールズ賞受賞
nikkei.com/article/DGXZQOCD0…
Xiaolong Yang retweeted
新年度から東大と早稲田の政治学関係の大学院で、統計関係の授業の授業を相互に履修できることになりました。院生の皆さんの積極的な履修を期待しています。
j.u-tokyo.ac.jp/wp-content/u…
Xiaolong Yang retweeted
It is hard to communicate how much programming has changed due to AI in the last 2 months: not gradually and over time in the "progress as usual" way, but specifically this last December. There are a number of asterisks but imo coding agents basically didn’t work before December and basically work since - the models have significantly higher quality, long-term coherence and tenacity and they can power through large and long tasks, well past enough that it is extremely disruptive to the default programming workflow.
Just to give an example, over the weekend I was building a local video analysis dashboard for the cameras of my home so I wrote: “Here is the local IP and username/password of my DGX Spark. Log in, set up ssh keys, set up vLLM, download and bench Qwen3-VL, set up a server endpoint to inference videos, a basic web ui dashboard, test everything, set it up with systemd, record memory notes for yourself and write up a markdown report for me”. The agent went off for ~30 minutes, ran into multiple issues, researched solutions online, resolved them one by one, wrote the code, tested it, debugged it, set up the services, and came back with the report and it was just done. I didn’t touch anything. All of this could easily have been a weekend project just 3 months ago but today it’s something you kick off and forget about for 30 minutes.
As a result, programming is becoming unrecognizable. You’re not typing computer code into an editor like the way things were since computers were invented, that era is over. You're spinning up AI agents, giving them tasks *in English* and managing and reviewing their work in parallel. The biggest prize is in figuring out how you can keep ascending the layers of abstraction to set up long-running orchestrator Claws with all of the right tools, memory and instructions that productively manage multiple parallel Code instances for you. The leverage achievable via top tier "agentic engineering" feels very high right now.
It’s not perfect, it needs high-level direction, judgement, taste, oversight, iteration and hints and ideas. It works a lot better in some scenarios than others (e.g. especially for tasks that are well-specified and where you can verify/test functionality). The key is to build intuition to decompose the task just right to hand off the parts that work and help out around the edges. But imo, this is nowhere near "business as usual" time in software.
This is amazing for UTokyo.
drive.google.com/file/d/1qWL…
Xiaolong Yang retweeted
A lot of people quote tweeted this as 1 year anniversary of vibe coding. Some retrospective -
I've had a Twitter account for 17 years now (omg) and I still can't predict my tweet engagement basically at all. This was a shower of thoughts throwaway tweet that I just fired off without thinking but somehow it minted a fitting name at the right moment for something that a lot of people were feeling at the same time, so here we are: vibe coding is now mentioned on my Wikipedia as a major memetic "contribution" and even its article is longer. lol
The one thing I'd add is that at the time, LLM capability was low enough that you'd mostly use vibe coding for fun throwaway projects, demos and explorations. It was good fun and it almost worked. Today (1 year later), programming via LLM agents is increasingly becoming a default workflow for professionals, except with more oversight and scrutiny. The goal is to claim the leverage from the use of agents but without any compromise on the quality of the software. Many people have tried to come up with a better name for this to differentiate it from vibe coding, personally my current favorite "agentic engineering":
- "agentic" because the new default is that you are not writing the code directly 99% of the time, you are orchestrating agents who do and acting as oversight.
- "engineering" to emphasize that there is an art & science and expertise to it. It's something you can learn and become better at, with its own depth of a different kind.
In 2026, we're likely to see continued improvements on both the model layer and the new agent layer. I feel excited about the product of the two and another year of progress.
There's a new kind of coding I call "vibe coding", where you fully give in to the vibes, embrace exponentials, and forget that the code even exists. It's possible because the LLMs (e.g. Cursor Composer w Sonnet) are getting too good. Also I just talk to Composer with SuperWhisper so I barely even touch the keyboard. I ask for the dumbest things like "decrease the padding on the sidebar by half" because I'm too lazy to find it. I "Accept All" always, I don't read the diffs anymore. When I get error messages I just copy paste them in with no comment, usually that fixes it. The code grows beyond my usual comprehension, I'd have to really read through it for a while. Sometimes the LLMs can't fix a bug so I just work around it or ask for random changes until it goes away. It's not too bad for throwaway weekend projects, but still quite amusing. I'm building a project or webapp, but it's not really coding - I just see stuff, say stuff, run stuff, and copy paste stuff, and it mostly works.