2x CTO - Building an AI agency 🛠
Joined July 2015
- Tweets1.2K
- Following606
- Followers613
- Likes4.4K
Gm gm !
I’ll be in montreal next week for All In.
It’s the biggest AI conference in Canada so many good stuff to see.
See you there
🚀 I’m participating in AgentX AgentBeats — the world’s largest open competition focused on agentic AI.
Hosted by Berkeley RDI and connected to ~40,000 learners via the Agentic AI MOOC.
This isn’t just about building better agents — it’s about how we evaluate them. 🧵👇
🧠 Why agentic evaluation matters
As AI systems become more autonomous, classic benchmarks start to break:
• data contamination
• overfitting
• leaderboard gaming
They tell us what scored well — not why systems succeed or fail.
AgentBeats flips the problem.
Evaluation itself becomes agentic:
• runs tasks autonomously
• enforces protocols
• analyzes errors
• produces structured, reproducible reports
Benchmarks become systems, not spreadsheets.
🧬 What I’m building
An autonomous evaluation agent for biomedical NLP, inspired by recent large-scale LLM studies (e.g. Chen et al., Nature Comms 2025).
The idea is simple:
one agent evaluates other agents — rigorously and transparently.
🟢 How it works
A “Green Agent” evaluates competing “Purple Agents” (LLMs or agentic systems):
• orchestrates task execution
• enforces evaluation rules
• measures performance
• generates diagnostic reports
No single score.
Real insight.
Goals:
• reduce contamination & overfitting
• enable fine-grained error analysis
• move from leaderboard scores → system-level understanding
Critical for high-stakes domains like healthcare.
🔬 Why biomedical NLP is hard
• hallucinations can sound plausible — and be dangerous
• ground truth is scattered across papers & databases
• rare diseases = sparse, inconsistent data
Evaluation needs evidence, not vibes.
🤔 A key question
Can generalist LLM agents match or beat specialized biomedical NLP tools?
Historically, domain-specific systems dominated:
• NER
• relation extraction
• evidence synthesis
AgentBeats finally lets us test this properly.
📊 What gets evaluated
Across 6 task types / 12 datasets:
• QA (MedQA, PubMedQA)
• NER (BC5CDR, NCBI Disease)
• multi-label classification
• relation extraction
• text simplification
• dynamic summarization (live PubMed articles)
Dynamic summarization is real-world hard: fetched live, evaluated end-to-end.
⚙️ Key capabilities
• fully automated evaluation
• fine-grained error analysis (hallucinations, boundaries, knowledge gaps…)
• 1–5⭐ ratings aligned with production readiness
• side-by-side agent comparisons
• actionable insights — not just scores
🔮 Why this matters
AgentBeats points toward a future where:
• evaluation is autonomous
• benchmarks are reproducible & contamination-aware
• AI systems are judged with real rigor
In healthcare, this isn’t optional.
More soon 👀
👉 agenticai-learning.org/f25
cc: @BerkeleyRDI
one example is the @web3privacy newsletter
an agent fetches the newsletter,
chooses the most impactful ones
and send you a summary privately 🤫
you can subscribe here: app.1xbuild.com/subscribe/we…
ps: this requires some Eth on arbitrum
our dream is to make privacy a default
and we believe the future of AI agents is
✅ autonomous
✅ privacy-first
✅ built for Web3
thanks to @iEx_ec @arbitrum @ArweaveEco
for pushing the boundaries with us
Bertrand retweeted
1xBuild – Their platform runs on “agent templates.” With Web3Telegram, they’re building a privacy AI newsletter: Telegram updates that stay anonymous, with content archived permanently on Arweave.
1xbuild.com/ T
This is why @iExec is building web3telegram.
A privacy tool to protect your handle
⚡️
gm legends!
more details on the tech running our agents
- @Letta_AI as agentic framework
- @nodejs backend for db, users, bots, tools...
- @reactjs frontend with @vite_js and @shadcn
the interesting part is actually letta
steal this here: letta.com
Introducing telegram agents
Managing a telegram community is a 24/7 job
💀 not doing it = dead community
😵 dead community = lack of trust
📉 lack of trust = token fall
We're automating it
Just finished a multi-agent architecture on discord
Based on a public VS private agent architecture
📢 The main agent (Public channels)
- has access to public channels and answers global questions
- invite users in DM for a more personal approach
🤫The personal agent (DM only)
- ask questions to the user to build a personal program
- maintain private notes on him
🧠RAG and memory
- one RAG is shared across all agents
- notes from the main agent are shared across all agents
- each personal agent keeps individual notes (private) for each user
I'm looking for some inspiration to improve it,
any idea of some cool discord agents? 😎
ALT multi agent architecture for discord
we were scaling client projects on ElizaOS
then updates broke everything
n8n couldn’t scale either
now, building our own framework
let's see where it goes 😎
