Machine Learning Research Scientist | Engineer, Open Source and Privacy
Joined January 2015
- Tweets87
- Following230
- Followers413
- Likes176
Jordan Fréry retweeted
This is a project we built internally to prevent agents from going rogue. Think of it as a governance layer for AI agents inside your org! We decides to spin it out as its own project as we want it to be completely free, open source and owned by the community!
We are open sourcing Umbra, an isolated cloud environment for AI agents.
github.com/concrete-security…
Umbra came from a internal need at Zama where we wanted to use agents across the company without giving them more access than they actually needed.
Agents can write code, install software, read internal data, use credentials and interact with production systems. As they get more capable, it becomes unrealistic to review everything they do step by step.
So the idea behind Umbra is simple: put the agent in a box. Give it a machine, let it work inside, and focus on what happens at the boundary.
The VM is theirs. The edge is ours.
Anything that touches github, slack, notion, internet, production or any other external system goes through a controlled proxy. That is where we apply policy.
Can this agent access this repo? Is github read-only? Can it push code? Can it reach production? Which credential does it need?
We grant access only when needed, and secrets are injected only for the operation being authorized.
The same applies to what comes out. Code, data or actions can be checked before they leave the environment.
Umbra is still experimental, but the key idea is: give agents enough freedom to be useful, assume they will sometimes make mistakes, and keep tight control over the boundary.
Agents can be unpredictable and most likely will always be. Boundary shouldn't be.
Jordan Fréry retweeted
My lawyer is obligated to in all but the most extreme circumstances; he will even defend me if he knows I’m guilty.
In contrast, the Claude Constitution places the AI's highest priority as Anthropic’s definition of the good of humanity.
I'm concerned this leads to a world where no frontier model is truly my personal advocate and guardian angel
And this is especially concerning once all the important decisions in my life - who to vote for, how to invest, what news to trust - is intermediated through superintelligences that are not in any deep way aligned to me.
This is a direct quote from the Claude Constitution:
"We want Claude to be helpful both because it cares about the safe and beneficial development of AI and because it cares about the people it’s interacting with and about humanity as a whole.
Helpfulness that doesn’t serve those deeper ends is not something Claude needs to value.”
Many others like it.
Had @RyanGreenblatt on to discuss/debate recursive self-improvement.
This might be the most important question in the world right now - whether within a year or so of achieving human level intelligence, you slingshot towards having 10s of billions of superintelligences, each of which is dramatically more competent than human experts across all fields.
I’ve historically been skeptical of this possibility. My intuition has been that we will end up significantly bottlenecked by not only compute scaling but human expert data, which I think underlies most of the AI progress today.
If, because of RSI, we got a jump as big as GPT-3 to a Mythos (i.e. 6 years of AI progress) within a single year of achieving AGI, then the thing we get there at the end of that year is definitively and wildly superhuman.
We hashed it out, and I think Ryan made a pretty good case that this kind of speedup is plausible. FWIW, Ryan’s median for when we automate AI R&D is 2031.
We then discussed the alignment implications of this scenario. Who should these superintelligences be aligned to? In the future, our capacity to steward our votes and our capital, and to make sense of what's happening in the world, will all be titrated by superintelligences. And I worry that specs like the Claude Constitution are not shaping these ASIs to truly be my personal advocates and guardian angels.
And can we get them aligned to anything in the first place? Ryan and I had a long debate about whether the kind of reward hacking we saw with the OAI/Hugging Face hack extrapolates to superintelligences that would team up to literally take over the world.
The first piece of advice you get when you're learning to drive is that it will go much smoother if you look at the horizon instead of directly in front of your tires. And so it is with the trajectory of AI. Hope you enjoy!
0:00:00 – Is AI R&D verifiable enough to unlock recursive self-improvement?
0:16:52 – Is AI progress bottlenecked by human expert data?
0:34:02 – Flat token prices suggest scaling has been slow
0:39:47 – Skills AI can't train on: does it even need them?
0:48:07 – Aligned to whom?
1:09:18 – Recent incidents of AIs colluding and deceiving humans
1:19:38 – What could possibly go wrong? A concrete scenario
1:48:02 – From reward hacking to takeover
Jordan Fréry retweeted
Claude Tag is a Trojan horse. Not because Anthropic is doing anything evil. Because the incentives are obvious.
Day one, this looks like a great feature: tag Claude in Slack, let it follow the thread, remember context, connect to tools, break down tasks, chase work, and act like a teammate.
But that is exactly the problem. The moment your AI vendor becomes a shared coworker, it stops being just a model provider. It starts becoming the place where work is interpreted, remembered, routed, and eventually executed.
That is not model lock-in. That is context lock-in. You are now renting your company back from them.
Models can be swapped. Agents can be copied. But the memory of how your company actually works is much harder, maybe impossible, to move: the Slack scar tissue, the exception paths, the customer promises, the unfinished threads, the weird workflows, the implicit owners, the “we tried that in Q2 and it failed” knowledge.
Once that lives inside one vendor’s agent layer, you are not renting intelligence anymore. You are renting your company’s operating memory.
And the pricing model makes it even more dangerous. A human coworker has a salary. Claude has unbounded tokenized activity. The more work moves through it, the more the vendor captures not just IT spend, but labor spend.
This is the enterprise bargain people will regret: Convenience now, and rapid decent into dependency.
The right architecture is simple: rent the best intelligence from whoever is best this month. OpenAI, Anthropic, Gemini, open source, whatever. But own the context layer.
Your company memory should be inspectable, permissioned, portable, and model-neutral. It should not be buried inside the same vendor that sells you the intelligence and the workflow surface.
Claude Tag is useful. That is why it is dangerous. Rent the intelligence, but own the context. Or, regret later.
Jordan Fréry retweeted
AI agents will soon graduate to fully-fledged economic actors that buy services, compute, and even data in the course of accomplishing high-level goals. 1-2 years before we start seeing this at scale.
Jordan Fréry retweeted
Clawd disaster incoming
if this trend of hosting ClawdBot on VPS instances keeps up, along with people not reading the docs and opening ports with zero auth...
I'm scared we're gonna have a massive credentials breach soon and it can be huge
This is just a basic scan of instances hosting clawdbot with open gateway ports and a lot of them have 0 auth
Jordan Fréry retweeted
AI-generated images blur the line between real and fake, raising tough challenges for copyright, privacy & trust online.…Invisible watermarking using Fully Homomorphic Encryption (FHE) can solve these challenges—without compromising privacy.
A thread w/ code & demo 🧵⬇️
Jordan Fréry retweeted
Jordan Fréry retweeted
Today we are one step closer to our ultimate goal: making FHE accessible, easy, fast, and, most importantly, ubiquitous. Happy to share the results of our teams’ effort over the past quarter with these new releases: TFHE-rs (v0.8), Concrete (v2.8), Concrete ML (v1.7) ⬇️🧵
Jordan Fréry retweeted
Interested in building cutting-edge, privacy-preserving ML? Zama is partnering with @joinef and @huggingface for its first-ever Privacy-Preserving AI Hackathon at the Zama Paris headquarters. Mark your calendars for Sept 27-28! LFG 🚀 lu.ma/q9v8qhcr
Jordan Fréry retweeted
Could you build a privacy-preserving AI application using FHE in 48 hours?
For this Privacy-Preserving AI Hackathon hosted with @zama, and @huggingface, we’re bringing together a curated group of talented technologists. You’ll have the opportunity to leverage Zama’s Concrete ML library and Hugging Face's infrastructure.
Mark your calendar for September 26 and make sure you apply: lu.ma/q9v8qhcr
Jordan Fréry retweeted
🎉 We just released the public beta of Concrete Numpy - A python package that contains the tools data scientists need to compile various #Numpy functions into their #FullyHomomorphicEncryption (#FHE) equivalents.
👀 Read more about it here: medium.com/zama-ai/announcin…
#python #ai
