Hard Work and Guts
Austin, TX
Joined March 2025
- Tweets1.8K
- Following895
- Followers4.7K
- Likes7.2K
Pinned Tweet
2 months ago, AI was a chatbot that I occasionally used for random questions. Then I came across Ahmad’s posts about how it works and what it can do, and I knew immediately this was something I wanted to dedicate all my time and resource into and finally, my own sovereign AI.
I started out with the goal of having my own AI assistant that is 100% private, that I can access anywhere using text, voice and vision. Currently building out the second brain infrastructure that can continuously improve itself, with actual long-term memory so it doesn’t just end up being another glorified chatbot. Afterwards, tie it into genomics research and VCF automation, and design my private console to drive and monitor the entire infrastructure.
I’ll start sharing my journey in a way that isn’t just technical jargon, my hope is that it’ll inspire more people to get into AI to improve their everyday lives.
RAM vs Gen5 vs Gen4 NVMe Engram Offload
Offloading to RAM gives significant increase in prefill and decent rise in decode over Gen5 NVMe. Gen5 outperforms Gen4 by good amount as well
Bonus data, Gen5 loads models roughly 20% faster than Gen4, but no benefits outside of that I can measure
How would you test a model for intelligence? During my swarm experiments with generational agents, saw some incredible behaviors by 300B tier models that 10-100B models didn’t show, which made me want to see what even larger models will do. While amazing to see, it sort of happens at unpredictable times so trying to come up with a controlled setting where I can test for complex behavior.
I understand ‘intelligence’ could mean many different things, which is why I’d greatly appreciate your feedback on how I can best test for it once I get TP8 up and running
Examples of how PCIe switch handle various GPU/model configurations
Dense model behind same switch, zero difference.
MoE model behind same switch, actually faster.
TP that combines GPUs inside + outside switch (goes through CPU) loses any switch benefits and risks collapse, avoid at all cost
Switch with 100 lanes supports 4 GPU at full x16 over a single PCIe slot, perfect for running multiple single models or TP2/4. Worth it if you’re out of slots and/or want to maximize moe TP performance, or want to shoot for proper TP8 which is whole another ballgame requiring 3 switches
You can have multiple GPU over a single PCIe slot without bifurcation, using a PCIe switch. It works like this.
The amount of data inference creates is measured in KB, nowhere near enough to saturate a Gen5x16 link. The bulk of the traffic comes from GPU-to-GPU hitting the CPU root complex uplink, and it can be avoided altogether using devices such as C-Payne switch.
As long as you aren’t offloading to RAM, or have P2P disabled, each slot can support many GPUs in various configurations for inference
Been working on the mother of all agentic task that covers majority of the real use cases y'all provided and sort of hitting a wall. Main goal was to
1. Can this model do X
2. If so how well, and wheres the limit
3. How does it fare against other models
4. Be as authentic to real use cases possible
Problem I'm finding is, a task cannot be 'authentic' and 'exact' without pulling against each other. Real work is messy using several skills at once, while exact answer needs a controlled task where only one thing change.
Also the master task is 12 tickets long that covers multitude of areas that spans 300+ tool calls. Takes about 40~50min to run, so breaking it into smaller chunks covering specific use cases. That way you can run only the tasks that measures thing you care about without spending an hour.
Anyway, its coming along. Will either have the entire agentic suite up on a website you can run, or I can run a model of your choice and put up the results/recording, whichever will be more feasible.
Based on Vera Rubin NVL insane spec (288GB HBM4 with 22TB/s), I was hoping for another huge jump for next gen 5090/6000 but think we might get smaller bump than expected.
Biggest jump will be MSRP, whatever that even means anymore
Recursive Flint Results and THIS is the closest thing to Qwen 3.8 35B if existed, so far
There is no thinking mode, but not in traditional sense. Thinking is always ON, no way to control level or off, which means how your harness work around its reasoning is critical to results
This model actually matched 27B on both tier2/3 agentic tests, which I haven’t seen done in quite some time. It also runs faster due to architecture, so I was ready to celebrate a new champion but..
Long context. Seeing a pattern lately where models do great until context grows, then starts to fall apart to different degree. Payload fidelity starts to fall behind a little bit, but agentic results speak for themselves and this thing can get it gone with proper harness
Fantastic model, good to bench one in same tier that can keep up with 27B for once.
Results for baby version of MiniCPM-V 4.7, compared against Bonsai 2 2B.
It’s small, but still x6 bigger than 2B. It will not serve you well as an agent, coding/tool-call/anything long simply won’t work
Crazy decode/prefill. It does shine for very specific short tasks, such as multilingual Q&A, classification, quick math or routing, on very small hardware. Vision might be a use too, but still working on comprehensive vision bench
Thinking mode is broken, use it on instruct mode only. On tasks it does well, it’ll do it with x10 less thinking than 2B so it has its moments. Quantizes very cleanly just like bigger sibling as well
Someone out there way smarter than me would know how to properly leverage the speed and size of these baby models, would love to hear about it
