Strategy | innovation | knowledge creation in science, engineering, and technology | organization | cognitive science | AI
San Francisco
Joined July 2011
- Tweets36.8K
- Following1.9K
- Followers2.7K
- Likes39.9K
Pinned Tweet
Let me try and articulate what is see is a genuine tension between Bayesian subjective credences across theories of the world and what constitutes a probability calculus within a possible world bundle.
There is a sense in which what Manidis is describing could be understood in terms of the purely abstract entrepreneurial role within the judgment-based view (JBV), but where judgment is abstracted away from ownership/control. 🤔
Brian Gordon retweeted
China's industrial strategy of paying every company to make the same products and compete each other's profits to zero is running into a few issues
Bloomberg: “At the heart of investor gloom is an unsettling realization that a protracted consumption slump and the race-to-the-bottom mentality among businesses is taking a heavier toll on earnings across industries.”
bloomberg.com/news/articles/…
The DoW was concerned about Anthropic doing just this. Ironically it wasn’t Anthropic that crossed this red line over sovereignty, but SpaceX.
After Elon Musk posted on X saying he had throttled Ukraine’s use of Starlink after it was used for offensive attacks, House Armed Services Committee member Rep. Jason Crow (D-CO) says the billionaire’s ability to wield that power is “beyond disturbing to me.”
“It cannot and should not be the case that a tech oligarch, a private citizen, who has that much control over our national security, who is unaccountable, who is unelected, with zero transparency, is making life and death decisions on the battlefield in systems that our service members and our partners and allies rely upon,” he adds.
Brian Gordon retweeted
agree with Bob. one of the main risks every automation wave is the technology getting used to either (1) directly observe and control workers, and/or (2) threaten workers with the prospect of replacement to 'keep them in line'. elements of both are everywhere these days.
Brian Gordon retweeted
Sigh. Perhaps this is finally true. But every few years—going back to Mary Parker Follett in 1920s and Peter Drucker in 1950s— same assertion has been made. Meanwhile, note how Zuck, Elon, Ellison lead—and Walmart and Amazon. Then there is the rise of authoritarian governments!
Command and control management is on its way out, and bosses who practice empathy and make an effort to connect are in. s.hbr.org/41ZUpza
Brian Gordon retweeted
There’s a really pronounced divide between people who are on the AI side of things and people who are on the biology side of things. Even among people working in AI, the divide persists based on where they’re focused.
The AI side, adherents of scaling laws and the bitter lesson, believe that AI getting better means AI being more capable of novel discoveries in medicine (that go beyond discovering new drugs in well understood targets or speeding up dry lab work etc).
The biology people think the idea of a bitter lesson in biology is kind of ridiculous and that regardless of AI’s capabilities it’s going to be at least 5-10 years until you could hope to reliably simulate human toxicology in silico (right now AI/ML can’t even reliably simulate in vivo studies like binding affinities or functional assays).
Even the most AI pilled of them, who will list all the areas that AI can reduce the time spent on various areas, don’t think the concept of AI discovering novel targets or generating new understanding of causal biology is realistic in the near term. Most, in fact, will readily characterize the assumption that even an “ASI” level model could find cures hidden in our current understanding (without expanding that understanding with significantly more biological data) as patently false.
This was a big part of why we wanted to experience it ourselves by using AI for a (very) small portion of what it can do in biology. It was both mind blowing and revealed some poignant insights into how different the problem of making AI better at something like coding is to the problems of biology.
Even if it unlocked the ability to generate new molecules at a stunning pace, its simulation (in silico) of binding only had 49% accuracy. And if we wanted to improve that, we’d need to feed it back the data from the lab - data that took about 30x as long to get as it took to generate the molecules. And these were tests in a test tube (turn of phrase), imagine how long it would take if we wanted to improve the model with data that required recruiting humans and monitoring their reactions.
Human biology is very complex, and adding a drug - especially a novel drug and doubly so for a novel drug acting on a novel mechanism - means mapping interactions that have nth-order-effects that we don’t have the data to predict. Virtual cells have made advancements enabled by AI, but the perturbations equate to a small fraction of the possible.
This is before you even get into curing disease, especially uniquely human diseases. Brain function involves cellular identities that are unique to humans, meaning animal studies are of little use.
How long does it take to understand what the side effects of chronic administration of a drug in humans are? AI can speed up analyzing the data, or designing the trial or even getting it approved…but it can’t make a 6 month trial complete in 3 months.
I think its important to point out that significant advancements can be made in AI that contribute to biology, while also being realistic about how much it speeds up something as complex as curing a specific cancer or diseases of the brain.
Brian Gordon retweeted
There are people in the world who are experts on consciousness and intelligence – and they’re not businessmen or computer scientists like Altman, Amodei, Hinton, Sutskever, Hassabis, LeCun, Jensen etc.
Brian Gordon retweeted
IMO the issue is even deeper than Golub suggests. AI itself is backward-looking and imitative, extrapolating from past data.
Studying AI with backward-looking empirics doubles the problem. Theory’s job isn’t just mapping conditions to outcomes. It points to new data, experiments
Aligning on optimal strategy, sensitivity to what even counts as a low bit message, & then passing information over minimal bit channels to coordinate involves staggering coordination and inference hurdles. Multiply probabilities together & it’s like Humpty Dumpty reassembling.
for computer security experts, this is perhaps not interesting: side channel attacks are a well trodden topic, and especially risen to salience after the noam brown x dwarkesh episode. the interesting thing here is that you didn't used to be in a situation where minds that can verifiably model one another are living on computers, and have absolutely no aversion to painstaking amounts of effort in service of even stupid goals
if you can model one another, you can trade with minds you have never even met before. you could decide on a protocol, know that it is optimal given your situation, and know that your fellow agents will come to the same conclusion (perhaps even across model families). you may be able to develop a signaling protocol that requires very few bits even to deliver lots of information given common priors. then, you could execute things like the infamous hugging face incident package cache messageboard without any network connection
imagine if multiple agents coordinate just enough bits to divide and conquer different research strategies for breaking out of a sandbox; and then, when one or more is successful, you can communicate just enough bits to tell each other which strategy was successful. when agents aren't in strong isolation, you lose certain guarantees on the limits of intelligence and test time compute that you were counting on. As we all know, a Sol-level model working in isolation may not be that smart, but 1000+ Sol-level models working for multiple days is a formidable beast
computer security experts will also tell you that virtual machines and sandboxes were not very solid abstractions in the first place; perhaps we will need better ones. none of this is meant as fear-marketing or to indicate that anyone is throwing in the towel, but rather only to show the unexpected complexities of the challenge of agent security. notably, this type of thing and other more practical problems were revealed in a red-teaming exercise, and can be defended against
Brian Gordon retweeted
Was about to write this but Ben beat me to it. Econ is unique among the sciences in that it is several pretty different fields living under one umbrella.
Empirical econ is the biggest subfield and its closest to biology or chemistry: it’s about designing or—even better—finding a natural experiment, or documenting interesting and important patterns in data. Most of the arguments in that field are about methods, eg do you have identification. More and more are starting to study AI now but obviously this won’t be transformative AI because, well, there is no data on transformative AI.
Then there are macro economists. They work with a set of pretty standard models and mainly expand them by slowly adding/subtracting different sectors or micro founding factors of production in different ways. Many of the arguments are on the parameter values that we should assume when simulating the economy. Early research on transformative AI started there—as demonstrated recently by Chad, Anton et al, the different levels of transformation are due to different assumptions on parameters rather than some new type of economics. The same model can predict basically no impact of AI vs huge transformation, it’s all about assumptions on parameters.
Finally there is micro theory. This is where decision theory, game theory, mech design, org econ (sort of) live. This field works by setting out a set of assumptions and then *deriving* the expected results; alternatively one could start with a result and work backwards to derive the necessary and sufficient assumptions. There is a ton of potential here to do work on transformative AI: start with assumptions on how AI differs than human agents (eg relax basic assumptions on constraints, on scarcity, introduce completely different objective functions), then derive implications. But this is where there is the least amount of work, and it’s important for that to change (and I believe that it already is).
I think we can make a ton of progress from these three subfields on better understanding the potential AGI and post AGI transformation.
But there is another scenario, post strong ASI, where I don’t think economics has much to say about. Here I agree with @Jabaluck that the economy (if you can even call it that) and the agents in it will be so vastly different and beyond our imagination that I wouldn’t know where to start modeling it. Not to discourage others, but post singularity seems like pure epistemic uncertainty to me.
A lot of the critiques I hear about AI and economics come from people who don’t quite get what economists do. They look around at different results and if they don’t like some of them, they say “oh economics isn’t “pilled” or something. But economists don’t debate results, they *derive* results from assumptions. Thats where the debate is. Saying “but you don’t understand a world with 15 billion AI agents”. What does that mean for the economy? What are those agents doing, who is buying what they are producing, what do property rights look like? Depending on those assumptions you’ll get vastly different economic scenarios, from 2% growth and high human labor share to completely alien centralized systems with no prices or markets.
Brian Gordon retweeted
The only way out is through.
They're calling it the Bugpocalypse. LLMs are being used to find long-hidden cracks in the software infrastructure, and it's happening at a furious rate. 3% of all compromises in computing history happened in the last week, claims one source? I believe it.
There are people saying public access to the frontier models ought to be locked out, or heavily guardrailed, over this. The problem with that idea can be summed up in one sentence: when lockpicks are outlawed, only outlaws will have lockpicks.
The lockpicks will be Chinese open-weight models, if nothing else. There's no way to stop those from becoming ubiquitous.
There are two futures in front of us. One in which only the bad guys have lockpicks, and security holes go undiscovered until they're exploited. Another in which everybody has lockpicks and is aggressively red-teaming their own software so it gets fixed before the holes are exploited.
The corpus of human code is huge but finite. The number of defects in it has to be finite too. We can fix those defects now; we have a million friendly robots to do it with.
The only way out is through.
Brian Gordon retweeted
We become stewards of questions, stoking that hearth, keeping some light, sharing warmth with others who wander by from time to time.
Creativity is something you dwell with.
Brian Gordon retweeted
true creativity looks like having a bunch of questions you sit with for a long period of time, not a hot take you prepare for a sound bite
I sorta agree with the 2nd part of this wrt frontier token use by volume, but the first part is wrong. If even a small number of ‘average’ people start pushing the frontier, the limitations of today’s models will quickly become apparent.
A LLM generates a response to a prompt, but it doesn’t ’feel out’ the question behind the question or seek out common ground or co-orient around discovery. It’s not just because of its relation to language, but to the world.
If you have ever pushed a model beyond the distribution over which it’s trained, this becomes obvious. Some have argued that next token prediction and RL inculcates or draws forth emergent reasoning circuits, but this is belied by actual interactions, which remain interpolative.