Ohana means 'control the intelligence explosion'
Joined February 2013
- Tweets3K
- Following1.5K
- Followers1.1K
- Likes14.3K
Pinned Tweet
One of my battles in this world is to get people away from building SaaS slop and into building real, impactful companies that will change the world.
Listen to the whole argument in this clip from the show:
Esben Kran retweeted
as far as I can tell, here are the stated views of tier 1 VC firms on extinction risk from AI so far:
- Denied/dismissed explicitly: @a16z, @khoslaventures, @benchmark
- Silent: @sequoia, @generalcatalyst, @ThriveCapital, @kleinerperkins, @IndexVentures, @Accel, @benchmark, @lightspeedvp
- Acknowledged the possibility: Thiel / @foundersfund, @GreylockVC
- Firm-level stance that avoiding human extinction is an active priority: none yet.
Who is going to lead from the front here?
We've all read the Hugging Face report, we are all staring down the cyber-offensive exponential, biological threats and attacks on physical infrastructure are not far off, and we are still early in navigating the threat landscape. There is no later time at which it will make more sense to act than today.
The frontier labs are already pulling out all the stops. Investors are a major power bloc at the frontier, they can see what's happening more clearly than almost anyone else. They need not see themselves as spectators to an unfolding drama, but instead can be -- and ought to be -- active participants in hardening civilizational defenses during the critical window (now).
That starts by taking ownership of the problem, in no unclear terms, and orienting towards sufficient (not cosmetic) solutions.
Not many people seem to know about Metaplanet, Jaan Tallinn's VC firm.
They are the most AGI-pilled firm on the planet, and are about to flip many of the tier 1s.
> Invested in @GoogleDeepMind in 2011
> Led @AnthropicAI series A in 2021
> Doubled down on Anthropic in every round since. IRR hurdle for deploying capital into anything else was {absurdly high number}% for years
> Anchored the premier AGI hedge funds (+Mythos) + in-house public equities strategy
4-person team in Estonia now managing 10s of billions AUM, liquid after $ANT IPO.
Rationalist roots --> zero bureaucracy, cares about human survival & flourishing as north star, doesn't care who else is investing or how high status a founder / idea is
Founders: congrats! you no longer need to play status games to raise a mega round for your civilization-scale moonshot. You can just pitch @metaplanetfund
I used a probe to post-train unwanted behaviors away and when the probe disappears, the model switches the unwanted behavior on in production.
It's proven on 0.6B but with more complex models and more complex RL environments, having a probe attached and finding this behavior may be near-impossible.
This can be mitigated directly with bit-exact probes and the effect is nearly as big as literally just including a verbal cue in the post-training data, which should be surprising.
Read the full analysis here blog.kran.ai/numeric-chamele…
Esben Kran retweeted
In case anyone's wondering, the expectation at the frontier labs is basically this:
1) We grit our teeth as the world slogs its way through the cyber apocalypse. (They've been expecting it for years, and calling it the "vulnpocalypse" internally.)
A model can be trained to know when a probe is watching (see @Sauers_' recent work) but the ability can be isolated and removed with two methods:
1. The "Precision emulation" setting in the PyTorch compiler
2. Switching and intervening on the specific heads that (accidentally, in frontier models) learn that probes are watching
Read the full investigation: blog.kran.ai/probe-switch-he…
Love this!
What is a philosopher-builder?
A philosopher-builder is someone who starts from a thesis about human flourishing.
Rather than simply looking for a gap in the market, they start out with a vision of the human capacities that are worth developing, what constrains them, and how we can expand them.
They reason from first principles about human potential and translate those insights into products or institutions.
To understand what a philosopher-builder is, it’s important to grasp what they are not:
1. They are not philosopher-kings. They do not have a blueprint for a society or a theory about how everyone out to live. Some of our most importance capacities are independent judgment, creativity, and the pursuit of knowledge. They seek to expand these capabilities, not prescribe outcomes
2. Open to disconfirming evidence. A strong philosophical starting point is open to rational criticism and has to be tested against reality. Their hypothesis is open to challenge and evidence should not only inform how they build their product, but potentially cause them to revise their underlying thesis. Philosophical seriousness demands intellectual humility.
3. Not a nihilistic iterator. “Make something people want” is a powerful entrepreneurial discipline, but by itself, it cannot establish whether satisfying people’s immediate wants advances human flourishing. You cannot outsource your belief in which ends are worth pursuing to market feedback.
The philosopher-builder is someone who combines philosophical conviction with empirical discipline. They have a vision, ambition, and an openness to revision.
We believe that AI isn’t a normal technology like the internet or the smartphone that society will simply absorb. Instead, it will create the biggest opening for institution-building since the American founding and philosopher-builders will be more important than ever.
If this vision of how to build resonates with you, we want to hear from you.
Esben Kran retweeted
🚨New paper (possibly the most important one I have written yet).
We think that as early as 2027 or 2028, there might be a major AI-enabled terrorism event. We conducted a pre-mortem, made 9 falsifiable predictions, and wrote 2 concrete scenarios.
Led by @KamacheeMax.
Esben Kran retweeted
I just read Anthropic's draft for a federal bill that was leaked a few days ago. You've probably seen other people say similar things, but I will add my voice to say that it's pretty bad.
In addition to being textbook regulatory capture by watering down key proposals of existing legislative drafts, the most striking thing of all is Anthropic's gall in doing one thing with their mouth and another with their hands. It was just a few weeks ago that Dario Amodei publicly called for embedded auditors and pacing the frontier. At the same time, however, Anthropic has privately been lobbying for slightly weakening access provisions for auditors and greatly weakening the federal government's ability to issue emergency orders to AI companies (both relative to the FRONTIER Act draft).
There's no two ways to put it. This is a tremendously dishonest attempt at regulatory capture.
Esben Kran retweeted
Registering my prediction that this turns out to be irrelevant for the case that interpretability can be a "Normal Science" but that the strong superposition argument (that UNRELATED concepts are stored in single neurons) is incorrect and training-time trajectories of neuron encodings will show incoherent superpositions as adjacent through the neuron's feature tree from early training at 90%+.
That last part is a major update, I believe, but does not solve our problem. See xitter.cf/JacobHHilton/status/21…
Hey @nickcammarata, have you interpreted my 432-parameter model yet? Do you think you'll manage it by year end? That's 4 hours per parameter
Esben Kran retweeted
The europoor meme can be funny, but it's also midwit and wrong.
As someone with both EU and US passports who's lived in both sets of elite poles (SF/NYC and BCN/BER/PAR), let me just give you the TLDR:
-The percent of GDP spent on welfare in the US is the about the same as the EU (smidge under).
-The effective tax rate in the coastal US is about the same as the EU (smidge under).
-Both are (former) middle-class societies where much of society feels the social contract is broken. In the US, elites abandoned the middle-class decades ago, and the well-to-do in states like CA/NY have reconciled themselves to paying almost half their paychecks and getting nothing in return, planning to insulate themselves from the suck by paying for private everything: neighborhoods, schools, medical care. If their earnings keep on rising, they're OK with it.
In France, that bourgeoisie hasn't been decimated yet, and they still expect (delusionally perhaps) basic social infra like schools to function (which is what these protests were initially about). The infra in Europe is a lot better than in the US, despite the GDP delta (New Yorkers can only dream of a subway like Madrid's). But that's clearly eroding in time, without the wage growth (or private social infra) to allow for a paid workaround.
-The free healthcare blah blah meme is fake: the public health systems in Europe have collapsed to the point it's mostly a public/private hybrid, for anyone with a solid paycheck, and fuck you if you're poor, go to the public system (which is the ER in the US). Guess what? In Europe you've got deductibles, in/out of network and all the same shit as in the US (though yes, the cost is still lower, though more or less correlated with wages).
-Boomers on both sides of the Atlantic are living in a dated lala land, but they vote to keep the system in place. In Europe it's legislation and printing money to prop up pensions; in the US, it's legislation and printing money to prop up housing prices and the stock market, which is the American 'free market' version of pensions. Rampant inflation and cost-of-living problems in both, with youth locked out of housing and living in precarity.
-Both are struggling with having been historically middle-class societies
US better positioned only because it has more economic growth (hello 80 hour weeks), and (obviously) leadership in critical fields like AI. But the foundational problems of an out-of-touch elite, incoherent immigration policy, and general social malaise that sees no meaning or purpose at the End of History is absolutely the same in both polities.
Americans who mock what's happening in Europe simultaneously believe some X mirage about what Europe is (and have never lived there), as well as don't understand what's going on in their own country. The French Revolution happened a few years after the American one, and once again, the two countries are echoes of each other, as similar countries tend to be. France is probably the most 'American' country in Europe, in all the republican senses of the word (and no the Yookay isn't in Europe).
So fascinating!
1/ We discovered something totally unexpected about the neural structure of consciousness that overturns decades of thinking on its biological basis and brings us significantly closer to the recipe for generating conscious awareness.
New preprint: biorxiv.org/content/10.64898… (led by @HesseJanis, with @FrankLanfranchi, @_conor_mcgrory, @Yuelin_Shi, and @EngelTatiana)
Esben Kran retweeted
The Lab ran some experiments to replicate the thesis - overall positive, though there may be caveats to how precise a feature classifier may be in terms of SNR
blog.kran.ai/random-network-…
Esben Kran retweeted
interpretability might be the most important scientific problem in the world right now, as I don't think we can solve alignment if we don't know what we're building. I think I have good news about it. I co-authored early work in mechanistic interpretability at OpenAI and came away pessimistic, because neural networks appeared to fold into arbitrary tangled knots. years later, I started openai's first large-scale automated interpretability project, finding about 2% of neurons legible with the primitive tools we had
with modern agentic tools and a much closer look, the picture looks totally different. the vast majority in models up to trillions of parameters are legible, and their circuits beautiful. many famous examples of tangled neurons turned out to be misunderstood. The roadblocks I expected mostly aren't there. I now think ambitious interpretability will be solved by the time we need it most, and possibly quite soon, as automated scientists keep improving. it may be a bitter lesson for interpretability: the simple thing, at absurd scale, works
already-clean.goodfire.com/
