
prev continual learning @google , @PrimeIntellect resident, @huggingface evals | opinions my own
San Francisco, CA
Joined September 2023
- Tweets1.4K
- Following1K
- Followers1.5K
- Likes3.6K
Sumuk retweeted
🧵[1/N] Are large language models more creative than humans at combining ideas? 🤔
Combinatorial creativity involves combining familiar ideas in unfamiliar ways, and is central to scientific discovery and technological innovation.
In fact, breakthrough scientific work is often associated with more distant combinations of methods, techniques, and concepts.
Here, we introduce Kombine, a new benchmark of combinatorial creativity for LLMs.
TLDR:
- LLMs are more combinatorially creative than the average human.
- However, conceptual blending is the hardest task for LLMs, and even frontier models struggle.
- LLMs also tend to homogenize, inventing concepts with the same properties across separate models.
- Moreover, humans and LLMs combine in distinct regions of semantic space.
- As a result, we anticipate greater conceptual diversity can be unlocked through human-AI collaboration. 🗣️
Thread below🧵⬇️
Sumuk retweeted
final week to apply to Speedrun Alpha fellowship! up to $250k investment in technical students + recent grads who want to build. apps close this sunday. 👀 👀 👀
apply: alpha.a16z.com
details below 👇
–
alpha is our fellowship for technical students and recent grads who want to start building. we have two tracks:
for future founders:
- up to $250K investment
- $1M+ in credits across AWS, GCP, Azure, OpenAI, Anthropic, Cursor and more
- office hours and fireside chats with speedrun partners
- 8 weeks in person in SF (jan 11 – mar 5)
if you'd rather get experience before starting a company, apply to our talent track:
- intros to speedrun portfolio companies to find your mutual fit
- office hours and fireside chats with speedrun partners
- 4 weeks in person in SF (feb 8 – mar 5)
Sumuk retweeted
a fellowship with $250k for recent grads/students - deadline next week
link: alpha.a16z.com
more info 👇
--
Applications are closing soon for a16z speedrun alpha, our new fellowship for technical students and recent grads who want to start building now
- Up to $250K investment
- $1M+ in credits across AWS, GPC, Azure, Open AI, Anthropic, Cursor and many more.
- Automatic final interview for speedrun, up to $1M investment
- Office hours and hosted sessions with speedrun partners
- Coworking space at the speedrun office throughout the fellowship
- 8-week, in-person experience (January 11 – March 5)
for folks that would rather get experience before starting their company, we also have a Talent Track that fast-tracks you toward roles at breakout speedrun startups
just over a week left to apply: alpha.a16z.com
Introducing Alpha (A001) - 13 fellows who spent the last 8 weeks building their companies with us in SF.
This is our earliest-stage program for students, dropouts, and recent grads with the conviction to run straight into building a startup.
Most started Alpha as solo founders, still figuring out what exactly to build. Within 8 weeks, they went all in on building, talking to customers, growing their teams, and even getting their first revenue through the door.
Meet our founders ↓
Sumuk retweeted
Say your goal is to steer an LLM to make it more creative — should you do this in weight space or in activation space? 🤔
🚨Excited to share our new work (CreativityNeuro), accepted at #COLM2026, which finds that weight space steering of creativity tends to generalize further OOD than activation steering! Thread below 🧵
Sumuk retweeted
very interesting post on chinese model progress; and, how to understand the gap between chinese compute and chinese model capabilities
every single person i know working at the frontier is incredibly talented and hardworking, and they're not wasting compute or capital, and neither is america. some thoughts / opinions on this
1/ new capability is infinitely harder. i'm not saying chinese labs are just mass distilling (although there's some of that), but even a few teacher model calls to 5.6 sol pro for math or fable agent rollouts is OOMs cheaper than hiring human experts, sourcing data, cleaning data, etc. this is why mercor, scale and the other data vendors do so well. paying physics phds to help teach the models is not cheap, and china does indeed get a leg up here.
this is also why i suspect most labs have the true frontier model internal only. the modus operandi seems to be to assume any model put into the hands of hyperscalers or released via api will be distilled or stolen. what the public sees is ~6 mo behind what's internal, and depending on how the chinese labs are operating, they're either ~1yr behind, or ~1.5yrs behind the true capability of the labs. there's also a fair amount of benchmaxxing from china, as they only have to show a good looking number to be considered great, not get sustained usage to stay alive (like the US labs), or be efficient at inference (which is also an optimization for the labs, more on this later). the modus operandi seems to be to assume any model put into the hands of hyperscalers or released via API will be distilled or stolen, which is 100% justified and right.
talent and ip bleed is a also real thing. a lab may frequently have to run/burn thousands of tests to figure out the optimal config/architecture for a model. anyone on the training, inference team or one of the hyperscalers who hosts the model can just look at one model implementation/config file and know the truth and understand and replicate. you don't need to know what didn't work, only what did.
2/ data privacy requirements (or lack thereof) in mainland china. assuming reasonably that the six little tigers in china (look it up if you don't know about this) have access to some form of additional ccp data that we do not know about, it is a moderate advantage. the labs are incredibly thirsty for novel data. the internet has mostly exhausted its utility, at least with current techniques, and much like what ilya said, there is only one internet (like fossil fuels!). people are also deploying the models to things never seen before. i.e how is a model supposed to learn how to order food with the doordash cli unless its taught how to behave in this scenario / is superhuman at generalization. US enterprises are also vary of this, and therefore cannot and do not rely on chinese model apis. i don't think this is cope, although i realize many may feel this way
3/ inference efficiency: the modern labs are extremely good at inference. the gb300 is a beast, and even older chips are being used very well to serve the models. api pricing is meant to also recover the cost on training, and inference cost for intelligence continues to fall. all labs have really large sparse moes that are OOMs cheaper than what api pricing reflects.
4/ profit motive: there is an interesting dwarkesh podcast about this. you must stop looking at china from the worldview of they're trying to make money. this is just not true. they're trying to dominate via scale and volume, which is true for the way china works, but not the us. the people of china are incredible, extremely sharp and are not to be underestimated. 996 work culture and strong stem is a great source of talent and data for the labs. american labs must pay around $1m/yr for entry level talent in the bay, which also increases capex. also there's an undisclosed amount of capex coming from the ccp. i do not claim to know what the scale is, but i would suspect there's a fair amount of chips flowing through the gulf, khazakstan etc. (do we seriously believe khazakstan of all places is building $10B worth of datacenters with nvidia gpus all for/by themselves?)
5/ datacenter and build out. xai had to spend a lot of money, time and effort to build out the supercomputer in tennessee. anthropic pays them 10 digits to lease out to serve customers etc because of the lack of supply, electricity, zoning, regulation etc. ccp does not have this issue, and there are claims that they are indeed introducing misinformation into the us that data centers are bad for us. we have to get much much faster about addressing this and dealing with "muh water use" kind of statements. it's oneshotting our population, much like nuclear energy did. electricity is the final moat, over chips, models, intelligence, and data. regardless of your timelines, if you believe in rsi, then you know the physical world is the only bottleneck.
china/the ccp has real, compounding structural edges on data access, deployment freedom, talent cost, speed of physical buildout, and now apparently on shaping the domestic narrative around our datacenters and energy. america is making the best use of it's most powerful resource, capital, to accelerate faster.
Sumuk retweeted
Suppose you're handed a fine-tuned LLM that secretly favors a certain entity. The bias goes completely undetected because it only surfaces on one specific unknown topic.
So how do you catch a bias you can't search for? You amplify it.
Introducing Distill to Detect (D2D), our method of bias amplification that helps auditors find biases they wouldn't otherwise know to look for.
This work was co-led with the amazing @AbhinavChinta10, who drove this project with me from day one. Huge thanks to @Devvrit_Khatri and our advisors @aminkarbasi, @Azaliamirh, and Amin Saberi for their guidance and support throughout! 🙏
📄 Paper: arxiv.org/abs/2607.01208
📝 Blog: distill2detect.github.io
💻 Code: github.com/abhinav-chinta/Di…
For more information, please see the thread below. 🧵
Sumuk retweeted
We compare the distributions of real and simulated user behaviors.
A few takeaways from our results across 24 LLMs:
- Scale alone isn't enough: Llama-3.1-8B-Instruct beats Llama-3.3-70B-Instruct
- 8B models specifically trained as user simulators rival the best closed-source models
- Open-source is competitive: gemma-4-31B-it and gpt-oss-120b outperform several closed-source models
Sumuk retweeted
What happens when you compare the distributions of real and simulated user behaviors?
🔍 The gap is large.
We introduce a method to measure this gap and evaluate 24 LLM-based user simulators across coding and writing tasks.
@convai_uiuc @MSFTResearch @berkeley_ai
🧵 1/N
Sumuk retweeted
🚀 Qwen3.5-397B-A17B is here: The first open-weight model in the Qwen3.5 series.
🖼️Native multimodal. Trained for real-world agents.
✨Powered by hybrid linear attention + sparse MoE and large-scale RL environment scaling.
⚡8.6x–19.0x decoding throughput vs Qwen3-Max
🌍201 languages & dialects
📜Apache2.0 licensed
🔗Dive in:
GitHub: github.com/QwenLM/Qwen3.5
Chat: chat.qwen.ai
API:modelstudio.console.alibabac…
Qwen Code: github.com/QwenLM/qwen-code
Hugging Face: huggingface.co/collections/Q…
ModelScope: modelscope.cn/collections/Qw…
blog: qwen.ai/blog?id=qwen3.5
Sumuk retweeted
Current long-term conversation benchmarks focus on recall.
But this ignores key skills like recognizing what user information is valuable & leveraging it to improve future interactions.
To address this gap, we present MultiSessionCollab to evaluate agents in a multi-session collaboration environment.
Sumuk retweeted
Thanks to @Lenovo for supporting our research with a gift! And, thanks to @ElibolOguz for initiating presentation sessions, the feedback and discussions were energizing! @convai_uiuc @uiuc_nlp
Sumuk retweeted
Replying to @sumukx @nikitabier
Truffle ambients do this
By the way
It will be a launch app
Sumuk retweeted
It's easier than ever to train, optimize and run your own models thanks to open-source (versus delegating all learning, control, capabilities to black-box APIs).
Cool to see @karpathy proving it once more by leveraging @huggingface fineweb (huggingface.co/datasets/karp…)!
Excited to release new repo: nanochat!
(it's among the most unhinged I've written).
Unlike my earlier similar repo nanoGPT which only covered pretraining, nanochat is a minimal, from scratch, full-stack training/inference pipeline of a simple ChatGPT clone in a single, dependency-minimal codebase. You boot up a cloud GPU box, run a single script and in as little as 4 hours later you can talk to your own LLM in a ChatGPT-like web UI.
It weighs ~8,000 lines of imo quite clean code to:
- Train the tokenizer using a new Rust implementation
- Pretrain a Transformer LLM on FineWeb, evaluate CORE score across a number of metrics
- Midtrain on user-assistant conversations from SmolTalk, multiple choice questions, tool use.
- SFT, evaluate the chat model on world knowledge multiple choice (ARC-E/C, MMLU), math (GSM8K), code (HumanEval)
- RL the model optionally on GSM8K with "GRPO"
- Efficient inference the model in an Engine with KV cache, simple prefill/decode, tool use (Python interpreter in a lightweight sandbox), talk to it over CLI or ChatGPT-like WebUI.
- Write a single markdown report card, summarizing and gamifying the whole thing.
Even for as low as ~$100 in cost (~4 hours on an 8XH100 node), you can train a little ChatGPT clone that you can kind of talk to, and which can write stories/poems, answer simple questions. About ~12 hours surpasses GPT-2 CORE metric. As you further scale up towards ~$1000 (~41.6 hours of training), it quickly becomes a lot more coherent and can solve simple math/code problems and take multiple choice tests. E.g. a depth 30 model trained for 24 hours (this is about equal to FLOPs of GPT-3 Small 125M and 1/1000th of GPT-3) gets into 40s on MMLU and 70s on ARC-Easy, 20s on GSM8K, etc.
My goal is to get the full "strong baseline" stack into one cohesive, minimal, readable, hackable, maximally forkable repo. nanochat will be the capstone project of LLM101n (which is still being developed). I think it also has potential to grow into a research harness, or a benchmark, similar to nanoGPT before it. It is by no means finished, tuned or optimized (actually I think there's likely quite a bit of low-hanging fruit), but I think it's at a place where the overall skeleton is ok enough that it can go up on GitHub where all the parts of it can be improved.
Link to repo and a detailed walkthrough of the nanochat speedrun is in the reply.
Sumuk retweeted
Wow this is a game-changer!
No. 1 question we get with +1.5M models on the hub: how to find the best one for your use-case?
Here comes YourBench 🪄 – work led by @sumukx @ailozovskaya and team
From a single document example, generate a custom eval with questions and test a series of models on it to find the best fit for your field!
Cherry on the cake: as always, it's all open-source and open-science with full publication, code, and more
Sumuk retweeted
Too much interest for #YourBench at #COLM2025 Congratulations @sumukx @huggingface @convai_uiuc @uiuc_nlp





























