Zac Liu (@zacqoo) header
Zac Liu (@zacqoo) profile picture

Building Causal Reflection @based_models — boost open models to frontier reliability without fine-tuning or RL | ML Engineer & Researcher | Stanford

CA
Joined June 2017
Photo by Zac Liu (@zacqoo)Photo by Zac Liu (@zacqoo)Photo by Zac Liu (@zacqoo)Photo by Zac Liu (@zacqoo)Photo by Zac Liu (@zacqoo)Photo by Zac Liu (@zacqoo)
Zac Liu retweeted
General agents have mastered code. Video is next. I quit my computer vision research job at Meta AI to spend two years building a video-first agent harness. Coding agents made building easy, so launching is now 10× more competitive. Meet AdaL v2, the video agent for GTM. • Generate and edit video, images and audio entirely through chat • Version control to explore ideas and go back to the one you like • Multiple videos in one project, from your main launch film to formats for each social platform HyperFrames and Higgsfield MCPs are add-ons inside Claude Code. AdaL v2 is our full coding agent harness, optimized natively for video, with its own desktop app. Compared with them: • 10× more control: you see and direct the whole production, not unreadable bash commands • 10× faster iteration: you reuse and reproduce any asset from past versions • 10× easier to use: one plan for video, image and audio models, zero setup, with browser use and deep research built in Launch films, product demos, explainers, UGC ads, every platform cut. Launching just got easy again.
60
22
12
90
7,680
Today we released the first results for Causal Reflection. It boosts open models to frontier performance and reliability without fine-tuning, RL, or routing to closed models. You can stay on open source and own the intelligence.
1
5
308
On Continual Learning Bench it ranked #1: 0.42 normalized reward / 0.49 mean gain at $2.32 — more than 2× the next best entry at >10× lower cost.
1
1
68
I’m joining the Boardy Pro Advisory Board! I’m working closely with @Boardyai and his team to shape the future of Boardy Pro. Boardy gave me 5 invites to share: 1 year of free access to Boardy Pro. DM if you want in!
1
3
310
Do you remember when you joined X? I do! #MyXAnniversary
55
LLMs are poor reasoners, everyone knows this. But most never ask why? A seminal critique comes from Mirzadeh et al. (2024) in their paper "GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Models" (Apple Machine Learning Research). The authors extended the GSM8K benchmark by introducing symbolic variations (e.g., altering names, numerical values, or adding irrelevant clauses) while preserving the underlying logical structure. They found that state-of-the-art models, including GPT-4o and o1-series, exhibited significant performance variance, drops of up to 65% when irrelevant but seemingly plausible clauses were added concluding : "We found no evidence of formal reasoning in language models. Their behavior is better explained by sophisticated pattern matching so fragile, in fact, that changing names can alter results by ~10%." This fragility is also echoed in Dziri et al. (2024) from MIT CSAIL, titled "Reasoning or Reciting? Exploring the Capabilities and Limitations of Language Models Through Counterfactual Tasks". By testing LLMs on counterfactual variants of tasks (e.g., altered chess positions or musical fingering), the authors observed consistent performance collapses in novel scenarios, attributing success on standard benchmarks to memorization rather than robust reasoning. Techniques like Chain-of-Thought (CoT), Tree-of-Thought (ToT), or hierarchical reasoning (e.g., planning + execution) improve performance by guiding models to generate intermediate steps, mimicking human deliberation. These are not true reasoning because: 1. Gains often stem from providing better context or "magic keywords" that align attention to relevant patterns, not internal logical processes. 2. Recent analyses describe CoT as a "mirage" models produce plausible-looking chains post-hoc, but these may not faithfully represent how the answer was derived (or could be illogical in smaller models). 3. Performance remains brittle: Models hallucinate, fixate on superficial cues, or degrade with depth/complexity. 4. Even advanced "reasoning models" (e.g., o1 series, DeepSeek-R1) scale inference-time compute for longer chains, yielding impressive benchmarks, but underlying failures persist in out-of-distribution or adversarial cases. 5. Most importantly, self-correction often fails or worsens errors without external feedback. In essence, these frameworks scaffold pattern matching into better approximations of reasoning, but don't instill systematic abstraction, causal understanding, or reliable generalization. LLMs "recite" trained paths more effectively, not derive logic. So excellent for compression i.e. knowledge retrieval but awful for reasoning i.e. decision making.
1
4
421
No one teaches this, but this is what really happens when you hit `run` on an LLM. User → API → Engines → Multi-GPU → CUDA → Hardware I mapped every layer (100+ components) of the LLM Inference Stack so you can finally see the full picture. Full blogpost coming soon!
3
7
1
68
4,548
The 2023 @Kaggle AI report is out! I was honored to be part of this project, and I served as the area chair for Tabular and time series data. There are many incredible insights in this report, and I urge you all to take a look at it. Here are some links: Kaggle AI Report Essay Competition: kaggle.com/competitions/2023… pdf version of the 2023 Kaggle AI report: storage.googleapis.com/kaggl… Slides for the report: kaggle.com/AI-Report-2023 #AI #ArtificialIntelligence #MachineLearning #ML
9
126
5
682
134,060
Anyone interested in VC and early stage startups, check out @ventureco_op, a first-of-its kind program by @LaconiaCapital that expands financial access into VC & sheds light on the inner workings of venture firms Cohort 5 applications are open until 9/21 ventureco-op.medium.com/quin…
1
4
347
2,921
6,167
1,710
40,737
12,891,450
wow. coming from @runwayml #Gen2 research.runwayml.com/gen2 While on the topic of video generation I was also mildy mind-blown a few days ago by multiControlNet and friends: reddit.com/r/StableDiffusion… And the earlier, bit more professional take, "anime rock paper scissors": youtube.com/watch?v=GVT3WUa-… The barrier to entry for creating animations/movies is evaporating quickly.
Would you watch this movie? I made this intro using @runwayml and their new #Gen2 Txt2Video the visuals in this clip are 100% created from Text Prompting! #GenerativeAI is improving so fast. I'll continue my journey and share my result with y'all! #animation #art #ai #aiart
26
160
8
1,204
346,480
Random note on k-Nearest Neighbor lookups on embeddings: in my experience much better results can be obtained by training SVMs instead. Not too widely known. Short example: github.com/karpathy/randomfu… Works because SVM ranking considers the unique aspects of your query w.r.t. data.
100
477
85
4,344
1,036,688
Next frontier of prompt engineering imo: "AutoGPTs" . 1 GPT call is just like 1 instruction on a computer. They can be strung together into programs. Use prompt to define I/O device and tool specs, define the cognitive loop, page data in and out of context window, .run().
Massive Update for Auto-GPT: Code Execution! 🤖💻 Auto-GPT is now able to write it's own code using #gpt4 and execute python scripts! This allows it to recursively debug, develop and self-improve... 🤯 👇
95
837
137
4,743
1,735,308
🎉 GPT-4 is out!! - 📈 it is incredible - 👀 it is multimodal (can see) - 😮 it is on trend w.r.t. scaling laws - 🔥 it is deployed on ChatGPT Plus: chat.openai.com - 📺 watch the developer demo livestream at 1pm: youtube.com/live/outcGtbnMuQ…
Announcing GPT-4, a large multimodal model, with our best-ever results on capabilities and alignment: openai.com/product/gpt-4
96
622
53
3,960
806,899
There is a high disaster potential in the Banda Sea due to mega-thrust #earthquake and #tsunami threat. The region can produce Mw 8.8 or larger earthquakes, as in 1629. Unfortunately, this threat to the densely populated region is highly underestimated. link.springer.com/article/10…
1
2
8
9,892
I will be presenting “AutoDC + AutoML = your AI dev superpower” at DataConLA 2022 (@DataConLA), with @TarlowScott #dataconla #dataconference #datascience #artificialintelligence #machinelearning #datavisualization
2
1
3
or get 50% off when you register using this link. eventbrite.com/e/30306554620… DataConLA is the largest data conference in Southern California and is celebrating its 10 year anniversary!!