
human-in-the-loop data infra | Prof, Ohio State CS | Co-founder, @thesteamfactory, @hackohio, @mobikitinc (acq.)
Columbus, OH
Joined March 2008
- Tweets295
- Following3.6K
- Followers3.1K
- Likes7.8K
Arnab Nandi ⭐️ retweeted
We ran @Elorian's new Glimpse model on RealityBench's physical work tasks: it reads labels as well as Argon, but has gaps in fine-grained visual reasoning: tape measures, lag bolts. Also, reasoning adds up quickly: our run cost $18 for a score that Gemma 4 31B matches for $0.70.
Arnab Nandi ⭐️ retweeted
Why build worlds? A work photo is the first mile of AI: if the capture is wrong, no model can reason its way back.
But real field data is locked away and barely labeled: thousands of photos per job, one label for the whole packet. Nobody knows what a model should see in each.
Sharing something we've been working on! @relanticai's RealityBench is a portfolio of 3D worlds which we used to evaluate 30+ frontier models on physical work tasks, incl. Argon, Astra, Fable, and Muse Wonder. Because these are 3D worlds, we can "bend" reality for new GT data.
Replying to @relanticai
The worlds are simulators, not image sets: built to extend to new tasks. Early access is on Hugging Face and relantic.com/realitybench Check out our video below.
Arnab Nandi ⭐️ retweeted
Task-specific routing can pay off hugely. For example, GPT-6 Astra, a top model, read only 46% of VINs correctly, whereas even Sonnet 5 gets 86%.
Arnab Nandi ⭐️ retweeted
The pace of improvement has been shocking. I half-expected that the focus on coding and RSI would mean that progress on vision had slowed. Instead, it seems likely that VLMs will be clearly superhuman in performance and cost in the next year or so.
Arnab Nandi ⭐️ retweeted
The models have a key structural advantage: superhuman attention. When you are looking at photos for details for a long time, it is easy to miss some.
Arnab Nandi ⭐️ retweeted
What did we learn about current models? The models that performed best are essentially price and performance competitive with a single human rater on average.
Arnab Nandi ⭐️ retweeted
We split the tasks into two categories: acceptability, where a VLM must judge if an image meets certain properties (e.g., car is in frame, license plate text is legible) and documentation, where the VLM must extract info from the scene (e.g., read a measuring tape).
Arnab Nandi ⭐️ retweeted
We build programmatic scorers for each. Humans disagree substantially on these tasks. We only release environments and scorers where the scorer closely matches the human majority in our testing.
Arnab Nandi ⭐️ retweeted
In some cases, we can compare VLM performance in these environments to real-world data from the same tasks. Model rankings transfer (Spearman ρ ≥ 0.63), suggesting the simulated environments capture what makes these tasks easy or hard for models.
Arnab Nandi ⭐️ retweeted
We created simulated environments in Blender for benchmarking VLMs on real world physical work perception and reasoning tasks. Think construction and maintenance of houses and cars.
Excited to share what we've been working on! @relanticai #RealityBench uses 3D worlds to evaluate 30+ frontier models on physical work understanding.
Arnab Nandi ⭐️ retweeted
Request early access to environments and images here: relantic.com/realitybench. Paper: arxiv.org/abs/2610.14397. Joint work with @arnabdotorg, @anmoltakiar and Dominik Winecki at @relanticai.
.@cbinnig should have titled this paper "welcome to the jungle"
"real-world DBs = highly normalized, even simple NL questions to require 4 joins/query, vs Spider (0.5J/query) ...state-of-the-art models experience up to a 70% drop in execution accuracy" openreview.net/pdf?id=NZLm4v…
Couldn't agree more. Recent "AI interaction" paradigms unlocked a huge wave of productivity. However, it's time to reclaim what we left behind.
One big regret about the LLM and agents wave is that subsecond interaction has taken a backseat. After decades of building towards “speed of thought” flow states, we’re now back to “brb, Claude’s compiling”.
Nice "Databases Wrapped" by @andy_pavlo. MCP is awesome, but MCP is also the fastest accelerating tech debt that databases have had to endure since MongoDB. Looking forward to the new wave of security, data loss prevention, access control, and perf recovery offerings for MCP.
Here is my latest article on the world of databases: cs.cmu.edu/~pavlo/blog/2026/…
All the hot topics from the last year:
• More Postgres action!
• MCP for everyone!
• MongoDB gets litigious with FerretDB!
• File formats!
• Market movements!
• The richest person in the world!
"In the AI-native university, professors become architects of learning, curating data, tuning models, and teaching students how to interrogate machine reasoning".
In 2026, I expect we’ll see the birth of the first AI-native university.
Over the past several years, universities have dabbled in AI-enabled grading, tutoring, and scheduling. But what’s emerging now is deeper, an adaptive academic organism that learns and optimizes itself in real time.
Picture an institution where courses, advising, research collaboration, and even building operations continuously adapt based on data feedback loops. Schedules optimize themselves. Reading lists evolve nightly and rewrite themselves as new research appears. Learning paths shift in real time to meet each student’s pace and context.
We’re already seeing precursors. ASU’s campus-wide partnership with OpenAI produced hundreds of AI-driven projects across teaching and administration. SUNY now embeds AI literacy into its general education requirements. These are the building blocks for more foundational deployment.
In the AI-native university, professors become architects of learning, curating data, tuning models, and teaching students how to interrogate machine reasoning
Assessment shifts, too. Detection tools and plagiarism bans give way to AI-aware evaluation, grading students on how they use AI, not whether they used it. Transparency and tactful application replaces prohibition.
And as every industry struggles to hire people who can design, govern, and collaborate with AI systems, this new university becomes the training ground, producing graduates fluent in orchestration who help augment a rapidly shifting workforce.
This AI-native university will become the talent engine for a new economy.
Do you think an AI native university will arise in 2026?
Leave your thoughts in the comments.
Arnab Nandi ⭐️ retweeted
Replying to @arnabdotorg
I use it so much, I even wrote a blog post about it (and a few other tools):
austinhenley.com/blog/prompt…










