Human in the loop
Amsterdam
Joined February 2012
- Tweets3.9K
- Following172
- Followers312
- Likes8.1K
Can't stop laughing at this one. Expectation when joining a robotics startup: advanced algorithms, bleeding edge machine learning and engineering. Reality: pretending to be a little duck
This live demo of four Microducks with Jensen Huang went completely viral, and we're ecstatic about it. But there's also a lot of confusion, so I want to clarify what Microduck can and cannot do, and what it will realistically be able to do in the future.
I count 60M+ views on this demo alone, based on the few big posts I see on social media, and everything has been buzzing. Even my own simple emotional design post on a newly created Instagram account got 1M+ views. We are selling Microducks at an incredible rate, even faster than during its historic launch. I've been trying to answer all the questions I see on our socials: people are asking if X or Y is possible on the robots, and this post aims to answer most of them.
First of all, the demo you've seen was not fully autonomous. We had one engineer piloting each Microduck. The motions on the robots were trained with reinforcement learning, using a sim-to-real pipeline: walking, standing up after a fall, sitting, doing a somersault, roller skating. All of those are policies trained this way. What the pilots did during the live event was send commands: directions (move forward, rotate), head orientation (look here), quacks, and special actions (do the somersault, sit down).
If you buy a Microduck, will you get an autonomous companion right out of the box? The current answer is no. But there is no impossible technical barrier to get there. I'm not sure how much time we'll have to work on this before shipping, but since everything is open source, I fully expect the community to build it, and to build it fast.
Now let's understand what's already done, what will be easy to add, and what's an open problem. What you'll be able to do out of the box, and you can already do it in simulation right now, is train new policies with reinforcement learning. You could teach your duck to walk differently from the default gait we provide, to do a weird dance, or to walk on stilts. The community has been crazy creative with this, even though they only have the simulator.
Note that you don't need reinforcement learning when the movements are simple. For example, the early emotional design I did is mostly coded movements plus pre-generated sounds. A simple mental model: use reinforcement learning when balance is part of the movement. If you try to code a cinematic walk by naively putting one foot in front of the other, your robot will realistically never walk, because you need to account for the dynamics of the movement, the inaccuracies of the sensors, the friction with the ground, and potential perturbations from the outside, like being pushed.
So training new policies is definitely doable, but you can't train a policy to get a genuine long-term companion in your home (well, technically you could, but that's a whole other range of policy: a research-grade problem, and the resulting neural network would probably be huge and never able to run on such a small compute). Instead, you need a few more building blocks: voice recognition, facial detection/recognition, localization, some simple navigation, and a behavior layer, which is less technical and harder to describe.
As a rule of thumb, a prototype of any of these is within reach of anyone if they have a modern coding agent and if the computation doesn't run on the robot (meaning the robot can reach the internet or an external computer and use external APIs). A rough prototype for each of the above can be a chill weekend project. As always, going from the prototype to a high-quality implementation is another story, but still doable for most if you put in the time and test a lot.
However, having all of this run locally, on the onboard chip, I would call a very hard task. You'd need some very clever engineering to get a good enough version of everything working together. But I can't say it's impossible, because I've seen some crazy efficient implementations of all of the above. And here we don't need a super high success rate: it's okay if the duck sometimes does something a bit random. So I wouldn't be surprised if folks managed a fully local, cool, simple version of this autonomous companion. For example, you don't need Whisper-grade open-vocabulary transcription: being able to recognize a few keywords would already be great for interaction.
One of the fastest ways to turn this robot into an autonomous companion would be to stream Microduck's sensors (the camera, the depth sensor, the IMU and the joint positions) to a computer, and use a software stack like the one we use for the conversation app on Reachy Mini: a local VAD, then a cloud speech-to-text model, then a call to a frontier LLM that decides on actions. Actions could be quacking, looking at things/faces/a direction, walking to a given position, sitting down, playing an emotion, kicking or doing a somersault. A SLAM algorithm would run in parallel so the robot can localize itself in your home, and you would prompt the LLM for a lifelike behavior: moving to random places, sometimes following people, and having emotional responses that fit what you're saying and the duck's environment. Of course someone is going to build a Tamagotchi-like layer on top of this, and it will be insane.
How much demand is there for a software suite that would do all of this? I for one think it would be super cool.
A word about the humans. We had five engineers backstage, and they are pretty much the core team behind Microduck. Their impact per head is absolutely off the charts. Antoine Pirrone piloted Graphite, Augustin Crampette Lavender (the rebel one), Matthieu Lapeyre Sky, Coralie Deplane Cream, and Pierre Rouanet was stressing over the Wi-Fi. Give them a follow! (Antoine @antoinepirrone, Augustin on LinkedIn, Matthieu @matth_lapeyre, Coralie @Drenaya_, Pierre on LinkedIn).
I get visibility because I communicate a lot, but I mainly work on Reachy Mini (you might want to check it out: it's a very cool robot, and it ships much faster than Microduck :) ). My contribution to Microduck has been minimal: I'm mostly a first power user. My main added value at GTC was having hands big enough to carry four Microducks below the stage.
Where I take pride is that Antoine and Coralie are former students and former interns of mine. I'm therefore entitled to 3% credit on anything (good) they do, for life. If they do something bad, however, it was clearly the fault of their parents.
About the super smart engineering that can be done on board: Antoine just released two very cool blog posts (links below). The first is on odometry, the robot estimating its own movement from its own internal sensors. The second is about SLAM, which allows long-term localization: it adds the depth sensor to the odometry to get a stable position over time. Think of it as recognizing known landmarks in your home to know where the robot is. The crazy part is that all of this runs on the robot. So this gives me hope that we'll be able to do a lot of cool stuff on board.
51
What people call software factory is often a software sweatshop now
Underdesigned software delivered quickly for the next demo day, and just as quickly left in a software landfill
Factories plan ahead and control quality
I'd wager that 99% of companies do NOT need a "software factory."
They just need to figure out WHAT to fix, and WHAT to build.
This example - Substack's search has been poor since forever, never addressed, even though it's easy to fix now (in hours, max days) a good example
75
The first two are programmer jargon for decades now. The last two I have never heard. Granted I am not in sf
saying “isomorphic” and “orthogonal” now is almost normie in sf. you need to be using 2010s cuda compiler speak e.g., instead of saying “we need to align,” say “we need to reconverge” instead of saying “ultimately,” say “the postdominator.” this will maximize your insufferability
48
Exactly. Selling in an interview is a two way street. If you apply to a high profile startup, they have already done their part. Now it's your turn.
In general, to have any shot at having a decent career in tech, learn to be proactive even when it’s not as painfully obvious (like here) that you need to.
Do your research before applying to a new place: you’ll learn a bunch if you do (and research w AI is much easier). If applying for a startup, build stuff that helps them, don’t just shoot an application (again, with AI if you know how to use it easier). Etc
Or just don’t, and complain how the market is tough… yea if you still work the same way you did in 2020
47
I might agree, but side projects are not just code, they could be hardware, crafts, sports, community work, politics etc. etc.
From a startup founder: “it’s now a red flag for me if I don’t see side projects on someone’s resume / application, because it signals lack of autonomy. Because it’s now so trivial to build anything on the side, vs before.”
1
106
Single responsibility microservices were always a bad pattern. With AI development they are now fully deadly. So much new surface for attacks, and the constant need for deployment coordination. But a whole generation of developers doesn't know another way.
1
84
On YouTube a monologue of a guy who was fired from a "people manager" position. He says his job was to protect the team from noise, and his team underused AI because they were not given guidance on how to use it. Oh, buddy...
79
There is a big battle happening rn for enterprise and national AI budgets. So much that b2c market is basically left alone for now, that's why for a regular consumer it looks like nothing is changing.
This quoted post is unavailable.
1
2
85
There's that beautiful artisan code that we had so much fun writing manually over the decades
Announcing 3 Critical vulns for hundreds of millions of IoT, Firewalls, OT, and Military devices including: Palo Alto Networks, Comcast, D-Link, HP, Canon, Honeywell, GE, Oracle, Siemens, Rockwell, Digi, Lockheed Martin, Boeing...
The vulns are with GoAhead Web Server which is a backend library used by these devices. There's a patch available from GoAhead but that doesn't mean there's a patch available for the devices yet -- that's up to the manufacturer.
- TLS traffic over DHE ciphers have a 1-byte secret, "04", meaning anyone can decrypt connections to these devices and steal login credentials and session tokens if the client negotiated with DHE ciphers like the latest TLS_DHE_RSA_WITH_CHACHA20_POLY1305_SHA256, which is extremely likely.
- LibreSSL mTLS has been completely bypassed, meaning anyone can connect over mTLS. If you have one of these devices listening on the internet and are expecting mTLS to protect it from attackers, take it offline or patch asap.
- OpenSSL Certificate Revocation Lists (CRLs) are not enforced, meaning if a user device is stolen or compromised, and you revoke that mTLS cert, nothing actually happens. There are a lot of other attacks this can be used for.
These are critical because these types of devices run our world. Water systems, electricity, military, firewalls, critical infrastructure.
foxio.io/blog/choose-your-ba…
1
47
Shameless ripoff of the famous search and rescue drones from Half Life 2
Purdue University's IDEAS Lab built an autonomous drone that tracks runners through dense forests completely without GPS or cloud networks.
By running 3D LiDAR mapping and obstacle-avoidance AI entirely on-chip, it can weave under tree canopies and predict target movement through blind spots in real time.
Massive implications for next-gen search & rescue and edge AI.
1
1
109
Yury Liavitski retweeted
Working on the battlefield look and feel today. Love how the backdrop looks now, but maybe the battle area itself needs a bit of detail
The reasons SR pays a lot more are:
1. It is infinitely automatable, so productivity of a programmer grows almost without limits, quickly.
2. It is an elastic target for investment. If you invest in a physical business you face much more initial costs and time before making returns.
I think many of us working in tech tend to forget that the reason software engineering vs other engineering fields pays (a lot!) more, is a lot more flexible (WFH/remote), and has better perks:
Because this field changes much faster than other engineering fields. The premium is because fewer people are able/willing to pick up new technologies, methodologies so quickly.
But the field had ~10-20 years of stability, and so lots of folks forgot this is why a premium exists. And that AI is just another tech wave: software engineers who pick it up and master it will generally fare better; those not will struggle a lot more.
81
