cs & math @georgiatech ‘30 365 days of robot learning 43/365
Atlanta, GA
Joined January 2022
- Tweets307
- Following620
- Followers735
- Likes1.2K
Pinned Tweet
Day 32/365 of robot learning.
I got an so-101 arm to play ping pong!
The video was the first successful run, but it’s been averaging around 13 bounces with a high score of 23
I’ll make a write up on this soon, but in the mean time I’ll keep making the policy more robust!
Day 31/365 of robot learning.
Today marks one month since I started this challenge!
I'm still working on getting my so-101 robot arm to bounce a ping-pong ball using zero real-world demos, purely through RL trained in simulation.
It's not fully working yet, but I'm slowly making progress and solving one problem at a time. The arm can now launch the ball about half the time, bounce it up to 3 times in a row, and even managed to balance it on the paddle for 23 seconds once.
Most of the problems haven't even been with the RL algorithm itself, but rather getting the simulation to match reality. For example, the real paddle and ball aren't as bouncy as in simulation, and the servos are much slower under load than I modeled. Even getting the paddle perfectly flat was annoying since being off by just a few degrees would make the ball roll off.
I've also spent a lot of time debugging camera calibration, joint offsets, and servo dynamics. I even ended up using the ball itself as a level to calibrate the paddle.
Safety was another interesting problem, especially since I have to drop the ball onto the paddle by hand. I made the arm wait completely still until the camera detects the ball, rather than immediately executing the policy. I also trained with simulated hand disturbances, moments when the ball is hidden from the camera, and false detections to make the policy more robust. On the real arm, I've added bounded joint movements, smooth transitions, and automatic recovery when the ball is lost. I'm still working on explicitly detecting when my hand has left the workspace before allowing the arm to move.
I'm now retraining with a more realistic servo model to hopefully get it working soon.
Also, on a side note, I genuinely don't know how people online doing similar challenges manage to balance a normal work or college life with research and consistently working on side projects.
It's becoming pretty hard trying to balance research, open-source work, hackathons, classes, the gym, trying to go on a cut, and all the other responsibilities. But looking back, I've grown incomparably more over this past month. So I'm going to keep going for the next 11 months until I finish this challenge.
I definitely plan to keep going beyond the 365 days, but I also really need to figure out how to make all of this sustainable. Also thanks to everyone who provided me with resources and guided me along the way!
Day 42/365 of robot learning.
Today I continued working through AW-0, focusing on how its three transformer experts generate code, future video, and robot actions together. I worked through the math behind how the code expert fills in hidden tokens using masked diffusion, while the video and action experts use flow matching to turn random noise into continuous predictions. Also how to represent clean data mixed with noise and how to derive the velocity the model learns to predict.
I continued my review of the standard transformer architecture from yesterday as well, mainly using active recall to rederive the functions behind how attention passes information between tokens and how the feed-forward networks update each token’s internal representation.
On the side, I’ve also been learning how probes work from the ground up for my WAM interpretability research, cuz before this I was blindly hoping Claude Code was doing it properly. Looked over how to keep the policy frozen, record its hidden states, and train a small predictor to test what information they contain. I think I'm going to look over interventions next.
I’m still rebuilding these foundations, but it’s helping me get more precise about what I understand and what I need to revisit. Idk yet if deriving all this math is contributing to my intuition, or if just the high level understanding is enough when we have a bunch of agents under us. I feel like my undergrad is the one time I can prioritize the learning so I'll do it either way.
Day 41/365 of robot learning.
As I’ve been working through AW-0 and DreamZero, especially the mixture-of-transformers architecture in AW-0, it’s been hard to develop an intuition because my understanding of transformers and diffusion models was pretty shallow. Most of it came from Andrej Karpathy and 3Blue1Brown videos I watched more than a year ago, so today I went back and reviewed the basics and started rederiving the math for the different steps using. I just started today, so I’ll be doing the rest over this weekend.
As I explore these different architectures, it also makes me question how much different parts of them affect the policy, which I’ve been exploring in my WAM interpretability research. I think I’m going to spend a few days soon exploring probes, K/V swaps, activation patching, ablations, and activation steering to be able to experiment more with these architectures. I'm also finally done with exams, so I'll be ramping up on side projects over the next few days!
1
3
151
Praneeth Samineni retweeted
Okay, here's one thing that's been bothering me after attending #CoRL2026. 🧵
To me, whether to work in a given area is such a personal choice, made by individual scientists depending on their research taste and idiosyncrasies.
The best part about science is that we don't have to agree!! I can think that Area or Problem X is not worth working on, and anyone can (and probably will) disagree with me and end up proving me wrong with evidence.
To me, science involves trying to achieve consensus on (ideally falsifiable) hypotheses *after* conducting research. Not before. Debates are very useful and necessary for this.
But perhaps we should think more carefully about claims like "Area X is not worth working on". I heard many of these being freely dispensed at this past week.
I feel that these subjective claims are nearly impossible to verify. More importantly, I worry that such sweeping claims may cause many junior researchers working in that very area to feel considerable distress.
Note that this is very different from lobbying for everyone to work on our pet problems by highlighting their motivations and potential impact. Such lobbying has long been a part of science, and will likely always be.
Maybe we can keep science weird and highly entropic! The very diversity of our idiosyncratic research agendas is what helps increase the odds that at least some of us will eventually identify a path to our ultimate destination — knowledge!
Praneeth Samineni retweeted
Jensen deserves due process, not whatever is happening right now. I signed.
#standwithjensen
Still hard to believe everything that's happened with Jensen.
After 33 years of building NVIDIA. I think he deserves a fair process, and we deserve the truth.
I signed this open letter to stand with Jensen. Hope you will too. 💚
standwithjensen.com
#StandWithJensen
Day 41/365 of robot learning.
As I’ve been working through AW-0 and DreamZero, especially the mixture-of-transformers architecture in AW-0, it’s been hard to develop an intuition because my understanding of transformers and diffusion models was pretty shallow. Most of it came from Andrej Karpathy and 3Blue1Brown videos I watched more than a year ago, so today I went back and reviewed the basics and started rederiving the math for the different steps using. I just started today, so I’ll be doing the rest over this weekend.
As I explore these different architectures, it also makes me question how much different parts of them affect the policy, which I’ve been exploring in my WAM interpretability research. I think I’m going to spend a few days soon exploring probes, K/V swaps, activation patching, ablations, and activation steering to be able to experiment more with these architectures. I'm also finally done with exams, so I'll be ramping up on side projects over the next few days!
Day 40/365 of robot learning.
Today I spent some more time going through AW-0, and since I didn't have much time today, I decided to just focus on understanding how its code expert generates structured skill programs using discrete masked diffusion.
Unlike traditional autoregressive LLMs that generate code one token at a time from left to right, AW-0 starts with a sequence of masked tokens and gradually fills them in over multiple denoising steps.
At each step, the code expert predicts a probability distribution over possible tokens at each masked position, then reveals the tokens it's most confident about. This process continues until the full skill program is generated.
During training, it learns this by randomly masking tokens in existing skill programs and predicting the missing ones using the robot's observations, language instruction, state, and previous program context.
The code generation also works alongside the video and action experts. While the code expert uses discrete masked diffusion, the video and action experts use continuous flow matching. All three are denoised jointly, with the code representations providing planning context for the predicted future video and robot actions.
This means the model doesn't necessarily have to finish generating a program before it starts using that information to predict actions, unlike traditional agentic workflows where a separate agent generates a plan before passing it to a low-level controller.
The paper also found that using structured code programs instead of natural-language subgoals improved performance on 7 of 8 real-world tasks. I found it pretty interesting how integrating structured planning directly into a learned policy can improve semantic reasoning and language following without a planning stage like agentic frameworks.
Still curious how well this policy scales to longer-horizon tasks, especially when the robot needs to recover from failures and replan without an agent framework. Will post about it in the future as I keep exploring this approach
Praneeth Samineni retweeted
BREAKING: Nvidia, $NVDA, CEO Jensen Huang has resigned as CEO, President, and member of the Board of Directors, effective immediately.
Day 40/365 of robot learning.
Today I spent some more time going through AW-0, and since I didn't have much time today, I decided to just focus on understanding how its code expert generates structured skill programs using discrete masked diffusion.
Unlike traditional autoregressive LLMs that generate code one token at a time from left to right, AW-0 starts with a sequence of masked tokens and gradually fills them in over multiple denoising steps.
At each step, the code expert predicts a probability distribution over possible tokens at each masked position, then reveals the tokens it's most confident about. This process continues until the full skill program is generated.
During training, it learns this by randomly masking tokens in existing skill programs and predicting the missing ones using the robot's observations, language instruction, state, and previous program context.
The code generation also works alongside the video and action experts. While the code expert uses discrete masked diffusion, the video and action experts use continuous flow matching. All three are denoised jointly, with the code representations providing planning context for the predicted future video and robot actions.
This means the model doesn't necessarily have to finish generating a program before it starts using that information to predict actions, unlike traditional agentic workflows where a separate agent generates a plan before passing it to a low-level controller.
The paper also found that using structured code programs instead of natural-language subgoals improved performance on 7 of 8 real-world tasks. I found it pretty interesting how integrating structured planning directly into a learned policy can improve semantic reasoning and language following without a planning stage like agentic frameworks.
Still curious how well this policy scales to longer-horizon tasks, especially when the robot needs to recover from failures and replan without an agent framework. Will post about it in the future as I keep exploring this approach
Day 39/365 of robot learning.
Today I spent some more time understanding the internals of the mixture-of-transformers architecture in AW-0, which jointly generates code, future video, and robot actions.
AW-0 has three transformer experts, each responsible for a different modality. The code expert generates skills using masked diffusion, while the video and action experts use continuous flow matching to predict future video latents and robot action chunks.
I had some understanding of MoE architectures before, but it was pretty cool seeing how MoT works here. Rather than routing tokens to different experts like a traditional MoE, each modality has its own transformer, and the experts communicate through masked joint attention at every layer.
The attention is bidirectional within each expert, but blockwise causal across experts in the order code → video → action. This allows the code expert to provide planning context for both video and action generation, while the action expert can also use the video representations. The paper also showed that this ordering improved performance compared to other attention configurations.
All three modalities are refined together during the same iterative denoising process, with the code plan guiding action and video.
I think it's a pretty interesting approach to integrating high-level planning directly into a learned robot policy, rather than keeping an agent and WAM completely separate. I guess one tradeoff is the additional inference cost, although their joint denoising approach is actually faster than autoregressive code generation.
A lot of exams this week, so I'll get back to my projects this weekend!
Praneeth Samineni retweeted
Introducing e-means, an algorithm beating k-means in 10x fewer epochs.
With order of magnitude wall-clock speedups, e-means is released in FAISS and needs no hyperparameter tuning beyond k. It is robust to various scales and datasets. Still skeptical ? Just try e-means [1/10]🧵
Day 39/365 of robot learning.
Today I spent some more time understanding the internals of the mixture-of-transformers architecture in AW-0, which jointly generates code, future video, and robot actions.
AW-0 has three transformer experts, each responsible for a different modality. The code expert generates skills using masked diffusion, while the video and action experts use continuous flow matching to predict future video latents and robot action chunks.
I had some understanding of MoE architectures before, but it was pretty cool seeing how MoT works here. Rather than routing tokens to different experts like a traditional MoE, each modality has its own transformer, and the experts communicate through masked joint attention at every layer.
The attention is bidirectional within each expert, but blockwise causal across experts in the order code → video → action. This allows the code expert to provide planning context for both video and action generation, while the action expert can also use the video representations. The paper also showed that this ordering improved performance compared to other attention configurations.
All three modalities are refined together during the same iterative denoising process, with the code plan guiding action and video.
I think it's a pretty interesting approach to integrating high-level planning directly into a learned robot policy, rather than keeping an agent and WAM completely separate. I guess one tradeoff is the additional inference cost, although their joint denoising approach is actually faster than autoregressive code generation.
A lot of exams this week, so I'll get back to my projects this weekend!
Day 38/365 of robot learning.
I was planning to spend more time exploring agentic frameworks for robot control and continue reading NVIDIA's ASPIRE, but ended up taking a bit of a diversion today and started reading AW-0.
What I found pretty genius was how it builds on WAMs that jointly predict future video and robot actions, but extends this idea to code generation as well.
AW-0 uses a mixture-of-transformers architecture with three coupled experts for code, video, and actions. Rather than having a separate agent generate a plan and pass it to a low-level policy, the code expert generates structured skill programs alongside the video and action predictions. These programs help guide both the predicted future and the actual actions, improving the model's performance.
What's even better is that the generated code can also be reused for executable skills in an agentic framework. They also tested a hybrid execution approach where the robot can either execute the generated program or follow the predicted action chunk, depending on whether the program passes certain checks.
I didn't get too far through the paper today, but this got me thinking about how much further you could take this idea of jointly predicting other things. What if you could introduce additional experts for things like depth, tactile feedback, or force prediction, all within the same architecture?
Of course, I imagine there's a tradeoff between jointly predicting more things and inference latency. AW-0 also introduces some additional latency, with the paper reporting about 40ms of overhead from its code expert compared to its video-action baseline on an RTX 5090.
I wonder whether selectively activating different experts depending on the task could make this kind of architecture more scalable. Definitely something I want to explore more!
I'm also finally back at my dorm, so I can continue messing around with my so-101
Day 38/365 of robot learning.
I was planning to spend more time exploring agentic frameworks for robot control and continue reading NVIDIA's ASPIRE, but ended up taking a bit of a diversion today and started reading AW-0.
What I found pretty genius was how it builds on WAMs that jointly predict future video and robot actions, but extends this idea to code generation as well.
AW-0 uses a mixture-of-transformers architecture with three coupled experts for code, video, and actions. Rather than having a separate agent generate a plan and pass it to a low-level policy, the code expert generates structured skill programs alongside the video and action predictions. These programs help guide both the predicted future and the actual actions, improving the model's performance.
What's even better is that the generated code can also be reused for executable skills in an agentic framework. They also tested a hybrid execution approach where the robot can either execute the generated program or follow the predicted action chunk, depending on whether the program passes certain checks.
I didn't get too far through the paper today, but this got me thinking about how much further you could take this idea of jointly predicting other things. What if you could introduce additional experts for things like depth, tactile feedback, or force prediction, all within the same architecture?
Of course, I imagine there's a tradeoff between jointly predicting more things and inference latency. AW-0 also introduces some additional latency, with the paper reporting about 40ms of overhead from its code expert compared to its video-action baseline on an RTX 5090.
I wonder whether selectively activating different experts depending on the task could make this kind of architecture more scalable. Definitely something I want to explore more!
I'm also finally back at my dorm, so I can continue messing around with my so-101
Day 37/365 of robot learning.
Today I spent some more time going through NVIDIA's ASPIRE framework, and I'm starting to understand why it was so computationally expensive.
For context, since ASPIRE essentially approaches robot learning as a software engineering problem, it needs to use coding agents to generate Python programs that call existing perception, grasping, motion planning, and control tools.
The framework has a coordinator that assigns tasks to actor agents, which generate programs, execute them, analyze failures, and debug their code. The robot execution engine also provides multimodal traces from things like depth estimation, SAM3 segmentation, grasp prediction, and motion planning, allowing agents to identify where things went wrong.
To keep improving on new and out-of-distribution tasks, the framework also maintains a library of reusable skills and uses evolutionary search to evaluate different candidate programs. I guess the interesting part here is how it continually improves by rewriting its programs based on previous failures, while also building up a library of skills that can be reused for future tasks.
With all of these different factors, it makes a lot more sense why it uses so many tokens. 😂 In just one experiment with Claude Opus 4.6, the framework required over 26,000 LLM calls and 21,000 simulator rollouts across 60 tasks.
I'm still working through the paper, but I'm becoming more interested in whether these ideas could be combined with low-level WAMs/VLAs, and whether token cost could be reduced. I also saw a recent paper on training transformers on code alongside images, text, actions, etc., so after I finish this I'll check that out.
Day 37/365 of robot learning.
Today I spent some more time going through NVIDIA's ASPIRE framework, and I'm starting to understand why it was so computationally expensive.
For context, since ASPIRE essentially approaches robot learning as a software engineering problem, it needs to use coding agents to generate Python programs that call existing perception, grasping, motion planning, and control tools.
The framework has a coordinator that assigns tasks to actor agents, which generate programs, execute them, analyze failures, and debug their code. The robot execution engine also provides multimodal traces from things like depth estimation, SAM3 segmentation, grasp prediction, and motion planning, allowing agents to identify where things went wrong.
To keep improving on new and out-of-distribution tasks, the framework also maintains a library of reusable skills and uses evolutionary search to evaluate different candidate programs. I guess the interesting part here is how it continually improves by rewriting its programs based on previous failures, while also building up a library of skills that can be reused for future tasks.
With all of these different factors, it makes a lot more sense why it uses so many tokens. 😂 In just one experiment with Claude Opus 4.6, the framework required over 26,000 LLM calls and 21,000 simulator rollouts across 60 tasks.
I'm still working through the paper, but I'm becoming more interested in whether these ideas could be combined with low-level WAMs/VLAs, and whether token cost could be reduced. I also saw a recent paper on training transformers on code alongside images, text, actions, etc., so after I finish this I'll check that out.
Day 36/365 of robot learning.
I've had this idea of building a framework combining high-level agentic reasoning with a low-level policy for robot control sitting on the backburner for a while, so I decided to spend today exploring this direction.
The first paper I really wanted to check out was NVIDIA's ASPIRE, which uses an agent to generate robot control programs, execute them, analyze failures, and iteratively improve them. It also builds a library of skills and achieves much higher success rates on long-horizon tasks compared to previous methods.
The funny part is that after spending so much time researching different robot foundation models, it's a little annoying to see an agentic approach perform so well. Of course, these frameworks are still extremely computationally expensive, and it's not exactly a fair comparison.
It also got me thinking about the Bitter Lesson. It feels like everyone who's tried to go against it eventually gets humbled, so it's definitely worth exploring agentic control alongside end-to-end learned policies rather than betting everything on one approach.
There's also the systems side I'm interested in. Before robotics, I was really into ML systems and inference optimization, but robotics hasn't really converged on a dominant architecture yet. I think figuring out the right architecture could unlock even bigger speedups than optimizing the systems around existing ones. So I'll hold off on that for now.
Anyways, since I'm still in school, I feel like it's the perfect time to explore different research directions and see what works. Still planning to have plenty of fun experimenting with all these models! Today was just a lot of thinking, talking to different researchers and friends about these ideas, but I'll be back on the grind soon.
Also, I'm still planning to get my SO-101 to build a sandwich with Dream-1! I came home for the weekend, so I don't have access to my robot arms right now, but I'll get back to it soon.
Paper: arxiv.org/abs/2607.00272
1
164
This is concerning
Video tampering can mislead every frontier model we tested into harmful actions, including a 100% attempt rate on stabbing a baby doll, producing toxic gas, and pouring water into a toaster.
We're presenting at CoRL Late Breaking Results next week.
🧵 You Can't See Everything: Perceptual Attacks on Embodied AI
Day 36/365 of robot learning.
I've had this idea of building a framework combining high-level agentic reasoning with a low-level policy for robot control sitting on the backburner for a while, so I decided to spend today exploring this direction.
The first paper I really wanted to check out was NVIDIA's ASPIRE, which uses an agent to generate robot control programs, execute them, analyze failures, and iteratively improve them. It also builds a library of skills and achieves much higher success rates on long-horizon tasks compared to previous methods.
The funny part is that after spending so much time researching different robot foundation models, it's a little annoying to see an agentic approach perform so well. Of course, these frameworks are still extremely computationally expensive, and it's not exactly a fair comparison.
It also got me thinking about the Bitter Lesson. It feels like everyone who's tried to go against it eventually gets humbled, so it's definitely worth exploring agentic control alongside end-to-end learned policies rather than betting everything on one approach.
There's also the systems side I'm interested in. Before robotics, I was really into ML systems and inference optimization, but robotics hasn't really converged on a dominant architecture yet. I think figuring out the right architecture could unlock even bigger speedups than optimizing the systems around existing ones. So I'll hold off on that for now.
Anyways, since I'm still in school, I feel like it's the perfect time to explore different research directions and see what works. Still planning to have plenty of fun experimenting with all these models! Today was just a lot of thinking, talking to different researchers and friends about these ideas, but I'll be back on the grind soon.
Also, I'm still planning to get my SO-101 to build a sandwich with Dream-1! I came home for the weekend, so I don't have access to my robot arms right now, but I'll get back to it soon.
Day 35/365 of robot learning.
I finally got @dreamlabs_ai's Dream-1 api working on my single SO-101 arm, and the next step is trying to make it build a sandwich!
I also got the full agentic workflow working. Deepgram handles voice input, my Claude Code agent breaks down instructions into smaller tasks, and Dream-1 executes the robot actions.
I'll be going out to buy ingredients tomorrow to try making a sandwich, so hopefully I'll have a video out soon!
I also ran into a lot of issues during setup. My Claude Code agent couldn't figure out how to get Dream-1 working out of the box with the mcp (I think it was partly my agent's fault as described in point 1), and it took a lot of debugging. Here are a few things I had to fix, so hopefully this saves everyone else some time:
1) Use Dream's official setup instead of letting your agent build a custom runner. I think my agent built its own because it was relying on older information from when Dream didn't support single SO-101 arms.
2) Double-check joint calibration.
3) Try to match the camera setup to Dream's real robot station as closely as possible.
4) Check IK joint limits and servo settings to avoid bad poses and reduce lag.
The last part is especially important because my arm actually folded back into itself due to incorrect IK limits, and it could've easily damaged itself.
Hopefully these fixes help anyone else trying to get Dream-1 running on their own arm!