Pinned Tweet
Comprehension Labs is an expert research team building datasets and RL environments from how people learn, reason, speak, and work.
Starting with 6,000+ hours of judged debate.
Request a sample or license your data at
comprehensionlabs.co
AI needs the full path to human capability.
Introducing @ComprehensionAI, an expert research team building datasets on how people learn, reason, speak, and work. Not just outputs. The attempts, feedback, and judgments behind them.
Starting with 6,000+ hours of judged debate.
Comprehension Labs retweeted
Anthropic's S-1 puts enterprise feedback inside its improvement loop. I'm bullish on company data for RL: reconstruct workflows from expert traces, define rewards, test on held-out tasks. My bet is on licensed environments for training agents on real business exceptions.
Anthropic has filed an S-1 registration statement with the SEC related to its proposed IPO.
Read more: anthropic.com/news/s1-sec
Legal disclaimer: A registration statement relating to these securities has been filed with the SEC but has not yet become effective. These securities may not be sold, nor may offers to buy be accepted, prior to the time the registration statement becomes effective. This post does not constitute an offer to sell or the solicitation of an offer to buy these securities, nor shall there be any sale of these securities in any state or jurisdiction in which such offer, solicitation or sale would be unlawful prior to registration or qualification under the securities laws of any such state or jurisdiction.
Comprehension Labs retweeted
Exactly, the 5% assumption puts the market in trillions. the bottleneck is converting that labor into usable supervision: task context, tool actions, expert corrections and outcomes, linked across the workflow. the opportunity is capturing expertise in a form models can train on.
Comprehension Labs retweeted
Valuable take. As intelligence gets cheaper, value accrues to whoever converts archives and live workflows into task-aligned expert traces: states, actions, feedback, outcomes. Winners will be measured by training utility, rejection rates and downstream lift, not raw volume held.
Comprehension Labs retweeted
Agree on proprietary data and network effects. the moat people sleep on is rl environments built on expert traces. a weak one is exposed fast through high rejection rates and no model lift. the hard part is not generating tasks. it is capturing traces worth training on.
The opportunity we see: bring industry expertise, usable expert traces and trusted data access together.
that’s what we’re building toward at comprehension labs: turning company data into training and evaluation material ai teams can use.
have proprietary data to license, or a specific dataset you need? let’s talk: cal.com/comprehensionlabs
Been seeing a lot of posts on data scaling and defensibility so here are a few predictions on how ai data space scales:
1. vertical specialists will build defensibility through industry-specific networks. domain expertise helps them recruit better experts and curate better datasets. when that attracts repeat buyers and more relevant contributors, network effects can emerge.
2. frontier expert trajectories will become a bigger part of post-training. task context, decisions, failed attempts and corrections provide more than final answers. specialists who can translate that work into executable tasks and reliable reward signals will be valuable to rl researchers.
3. trust infrastructure will become a competitive advantage. clear licensing rights, data provenance, access controls and de-identification help unlock proprietary company data. the hard part is reducing re-identification risk without stripping out the context that makes it useful.
We are building Comprehension Labs keeping all of these as guiding principles to turn overlooked human expertise into a continuing source of learning for AI. DM me or mail me at [email protected] if this perks your interest.
Comprehension Labs retweeted
Been seeing a lot of posts on data scaling and defensibility so here are a few predictions on how ai data space scales:
1. vertical specialists will build defensibility through industry-specific networks. domain expertise helps them recruit better experts and curate better datasets. when that attracts repeat buyers and more relevant contributors, network effects can emerge.
2. frontier expert trajectories will become a bigger part of post-training. task context, decisions, failed attempts and corrections provide more than final answers. specialists who can translate that work into executable tasks and reliable reward signals will be valuable to rl researchers.
3. trust infrastructure will become a competitive advantage. clear licensing rights, data provenance, access controls and de-identification help unlock proprietary company data. the hard part is reducing re-identification risk without stripping out the context that makes it useful.
We are building Comprehension Labs keeping all of these as guiding principles to turn overlooked human expertise into a continuing source of learning for AI. DM me or mail me at [email protected] if this perks your interest.
Comprehension Labs retweeted
Everyone is racing to automate knowledge work and almost nobody has captured it. 715 of 923 US occupations have no real benchmark. No traces, no training and no licensing work. The scarce thing is the record of how experts work and that is an indication why data will grow 100x.
Mercor Research mapped AI benchmarks to actual US jobs.
Only 208 of 923 occupations in the US economy are represented by existing AI benchmarks. 90% of hours worked in the US aren't covered well by a relevant AI benchmark.
Today, we’re releasing the 𝗕𝗲𝗻𝗰𝗵𝗺𝗮𝗿𝗸 𝗠𝗮𝗽 𝗼𝗳 𝘁𝗵𝗲 𝗨𝗦 𝗘𝗰𝗼𝗻𝗼𝗺𝘆, an interactive breakdown of which economic tasks are covered by AI benchmarks.
Benchmarks are how we measure AI’s progress and anticipate its impact on real work. They’re crucial to understand which jobs AI can automate, where the biggest gaps are, and how those capabilities are changing over time.
The Benchmark Map is our first step toward building a resource that makes it easier to answer: 𝘋𝘰𝘦𝘴 𝘵𝘩𝘪𝘴 𝘈𝘐 𝘮𝘰𝘥𝘦𝘭 𝘸𝘰𝘳𝘬 𝘧𝘰𝘳 𝘮𝘺 𝘰𝘤𝘤𝘶𝘱𝘢𝘵𝘪𝘰𝘯?
Explore it here: mercor.com/research/labor-in…
Comprehension Labs retweeted
Rising de-id spend is proof the market is real. Sellers who capture consent and structure at collection avoid that cost entirely, so it becomes margin instead of overhead. Same demand, very different unit economics, and the gap widens as volume grows.
it is obvious that the data broker space is overcrowded. I would not recommend becoming a general data broker nowadays. with that said, here are ACTUAL more niche ways to earn money in the data space now:
1. de-identification. we spend a stupefying amount on on it. around 35k/TB on non-edge cases! this is at least a 7 figure monthly cost for brokers like us. it is a pricey tax that must be paid. there’s relatively few companies that do de-id well. run Presidio and GLiNER and maybe a 32b Qwen model for hard cases. then run and fine-tune based off of results. even that simple pilot could show signs of pareto!
it is crucial your de-id system needs to:
a) rigorously sanitise to industry standards (obviously)
b) map multiple variables to a single entity (e.g erik newsham, E.N, erok gets translated into a de-id representation of me)
2. devote yourself to the buy side. since everyone and their grandma has become a broker, there is now a glut of brokers with company data to sell, yet they have no one to sell it to! devote yourself to finding data buyers. a few efficient ways to increase surface area of buyers is to: go to SF and get face to face with researchers, mass cold email outbound or fill out our grepped partners application as we are actively buying. once you have found data acquirers, tell brokers you can help them offload for 20% commission.
3. build a financial market for corpora. the current data market is unlit. often two labs bidding for the same corpus have given us vastly different prices (which is not automatically bad as there isn’t one fixed value for data). we have been asked directly by several labs for historical transaction data. as data is so fuzzy and context-dependent it would obviously be hard to standardize, so maybe the form this market would take is a marketplace. a safe air-gapped agentic system could scour an entire corpus, verify, note and map everything needed for a buyer to make an informed decision. solving data's Arrow Information Paradox. a corpus could be sectioned and sold as parts. overtime, transactional data is produced and “data class indexes” could emerge and efficiency in this currently inefficient market would flourish. companies could list data as an asset and “IPO” their corpora. there are obviously many variables (license time and exclusivity for example) but through a streamlined process that irons out rights, QC, de-id and other commonplace wrinkles. this would be more like art indexes than crude indexes
4. make autonomous lab machinery. I am super super autonomous lab pilled. ask claude to cure cancer and it will come back with experiments that need to be done to further advancement. we simply have not conducted enough experiments. the (very) simplified loop of an autonomous lab is goal→conduct millions of simulated experiments→select top performers (and a few moonshots) for real world experiment→use successes and failures to update simulation and make progress. periodic labs and anthropics wet lab are really cool examples of this. from what I have heard though, the machinery isn’t super great and is a major bottleneck.
5. broker financing. some companies would rather $400k now than $600k in a month. for an instant deal to happen, a broker needs money and a lot of these new brokers have, uhh, interestingly sized liquidity pools (they are broke). by lending them money to buy corpora upfront, they close a deal that would never have happened and you make a nice slice of the spread for providing them cash short term.
Comprehension Labs retweeted
The gate prediction is the one to watch. As labs scale, the question stops being "how much data do you have" and becomes "can you prove where it came from."
Whoever recorded permission at the moment of collection wins that test. Rights reconstructed after the fact almost never survive diligence, because there is nobody left to ask and nothing on file to show.
Provenance is about to become the price of admission, not a nice-to-have.
a few of my predictions for the next 6 months of ai data:
- a monopolistic human data buy side layer emerges. right now you can use zap mail, heyreach, instantly and claude max plan to reach out to practically every potential human data buyer on the world in 7 days and (even with the worst strategy) at least come out with roughly 20 buyers spanning rl builders, neo labs and (if lucky) one or two major labs, who you could easily offload $20-$50m cumulatively from the very start. it is not going to be this easy to find buyers forever. as labs inevitably get f*cked by bad apples on rights, de-id, QC and the rest, a select few will emerge as “transaction gates” in which every corpora must flow them in order for the labs to deem as safe to buy. analogous to how a few labs already don’t touch corpora if it was de-id by Tonic Textual.
- the waterline continues to rise. ai labs are buying real company data to build artificial workspaces to teach their models how to do an office job. using a headcount as a metric, 3 months ago they bought company internals with 10+ headcount, now they almost exclusively purchase from companies with 50 or more employees. as models saturate these smaller workspaces, demand for smaller company corpora plummets whilst demand for the already dwindling supply of large sprawling company internals increases. yet doesn’t skyrocket as at a certain point if the price becomes too high then labs fall back on synthetic expert work. it goes without saying that small companies in a esoteric domain models underperform in that hold vertical depth (e.g a 30-man freight brokerage) will still hold immense value.
- reality gets graded more and more. as the most knowledgeable domain experts begin to fail in writing valid rubrics that aren’t instantly saturated we are going to have to get reality to play the verifier, domain expert and environment, if we want progress to continue. rubrics shift from the attributes of a perfect NDA to a lab measured critical tempurature.
- atom environments. why hasn’t the vertiginous increase of ai capabilities led to huge economic growth? how can we get non-tech people to understand how intelligent ai is already? how do we find the best value/token ratio? to “solve” these questions I believe need to give ai a “body” in which it can interact with the real world at extremely low bandwidth. giving ai a body sounds like a humanoid which is obviously just one small slice of real world interaction, a little bit like how literally none of the ways we interact with in the real world is only via our real body but via tools instead like handles, vehicles, instruments and cutlery. maybe I should reframe it from giving ai a body to giving the world ai
- protocols for company data emerge. for such a fuzzy yet crucial area of ai, it is insane how unorganized certain steps in company data brokerage are, such as giving context on what your companies internals actually hold.
Comprehension Labs retweeted
Sycophancy is a data problem. Five leading assistants favored answers matching the user's views, because preference data rewarded agreement. That is why we hold the largest licensed reasoning and debate dataset because judged debate rewards discernment, not flattery.
Comprehension Labs retweeted
Why Comprehension Labs? Models can see final answers but not how people reach them. We build RL environments grounded in education, backed by consented human data on how people learn, debate and decide. Expert judgment, captured as traces. To learn more: comprehensionlabs.co
DM us or book a call at cal.com/comprehensionlabs/30…
Why Comprehension Labs? Models can see final answers but not how people reach them. We build RL environments grounded in education, backed by consented human data on how people learn, debate and decide. Expert judgment, captured as traces. To learn more: comprehensionlabs.co
Comprehension Labs retweeted
AI needs the full path to human capability.
Introducing @ComprehensionAI, an expert research team building datasets on how people learn, reason, speak, and work. Not just outputs. The attempts, feedback, and judgments behind them.
Starting with 6,000+ hours of judged debate.
Comprehension Labs retweeted
A passing test is not always a solved task. An agent can remove a failing test instead of fixing the bug. High-quality RL data needs checks the agent cannot rewrite, so the reward tracks the behavior we actually want.
Comprehension Labs retweeted
Data is not saturated. Generic web text is. One research estimate puts data labeling at $57.6B by 2030. The audited floor grew 32% last year, and all ten frontier labs still buy human judgment and evaluation data. Judgment will keep compounding, and it needs to be captured.
Comprehension Labs retweeted
An expert corrects one mistake. The real question is whether that correction carries into the next problem.
That's longitudinal data, and it's what we're building at Comprehension Labs. Here's what one connected trace looks like.
Buying or collecting this data? DM me.
An expert corrects a mistake. What happens on the next attempt?
At Comprehension Labs, we're interested in how feedback changes work and whether that change carries into a different problem.
Take a code review. An engineer submits an implementation. A reviewer identifies an edge case, explains the issue and requests a change. The engineer revises the code and adds a test. Together, the task context, original version, review, correction and result form a production trace with human judgment attached.
Now connect that record to the engineer's next task. Do they catch a similar issue before review? Can they apply the feedback when the requirements change? Does the same mistake recur?
That connection is the longitudinal part. It lets us examine how decisions and approaches change across repeated work. A successful edit shows that a correction was made. Later attempts give us evidence to investigate whether the feedback carried forward.
For that evidence to be useful, the details need to stay connected. Feedback belongs with the version it addressed. An approval needs its criteria. An outcome needs its context. Expert checks and recorded explanations help make those decisions interpretable.
This is what we're building around: production traces organized into connected histories of attempts, feedback, revisions and outcomes. Our goal is to make these records useful for AI training and evaluation while preserving the context that gives them meaning.
If you're buying or collecting this kind of data, or your company captures these traces in its workflows, we'd love to chat. DM us or book a call at cal.com/comprehensionlabs/30…
An expert corrects a mistake. What happens on the next attempt?
At Comprehension Labs, we're interested in how feedback changes work and whether that change carries into a different problem.
Take a code review. An engineer submits an implementation. A reviewer identifies an edge case, explains the issue and requests a change. The engineer revises the code and adds a test. Together, the task context, original version, review, correction and result form a production trace with human judgment attached.
Now connect that record to the engineer's next task. Do they catch a similar issue before review? Can they apply the feedback when the requirements change? Does the same mistake recur?
That connection is the longitudinal part. It lets us examine how decisions and approaches change across repeated work. A successful edit shows that a correction was made. Later attempts give us evidence to investigate whether the feedback carried forward.
For that evidence to be useful, the details need to stay connected. Feedback belongs with the version it addressed. An approval needs its criteria. An outcome needs its context. Expert checks and recorded explanations help make those decisions interpretable.
This is what we're building around: production traces organized into connected histories of attempts, feedback, revisions and outcomes. Our goal is to make these records useful for AI training and evaluation while preserving the context that gives them meaning.
If you're buying or collecting this kind of data, or your company captures these traces in its workflows, we'd love to chat. DM us or book a call at cal.com/comprehensionlabs/30…















