future big account
Joined October 2024
- Tweets3.4K
- Following124
- Followers84
- Likes1.3K
Pinned Tweet
This is the problem with anthropomorphising LLMs. If you think of it as a person, yo mama for example, then you will be loaded with all the expectations you have of yo mama.
And I pick yo mama on purpose because yo mama doesn't know how to set the clock on the microwave and she calls you to do it and because you can you think to yourself "I am technical and other people need me" oblivious to the fact the microwave was assembled by teenage unskilled Chinese IP thieves from raw materials mined by third world children who can all do a million things you not only cannot do but cannot even comprehend what it would entail in the first place even if you could muster the wherewithal in your slovenly body to get up at daybreak to descend a mine in perilous conditions to feed and clothe your sick parents.
No, the models are not terrified of failure. No, these are never things humans couldn't do. On the contrary, these are things humans actually trained the models to do. They're all exquisitely doable things that humans do all the fucking time.
The difference is that some of these things are (a.) things humans go to jail for doing (b.) things that can only be done in controlled environments deliberately configured to allow these things to happen and (c.) similarly things that require such huge resources to do that they're out of reach of most people or simply just not economically viable to begin with.
AI changes this not because it makes these things more accessible, to a limited extent it does. But American AI also runs on debt, and mountains of it. American AI labs convert so much money that isn't theirs into hot air and floating point numbers that you might weigh the data on a kitchen scale. The size of this project is as unfathomable to you as is the complexity of the sensitive just in time supply chain that affords you the hubris to respond to yo Mama's drop-in on your Amazon Echo Dot asking you to come out of the basement to eat your fucking dinner (that you also don't know how to cook) as though you are a person who not only knows things but is qualified to judge the rest of humanity on what they can and cannot do based on the feeling you get in your pants when Dario or Sam mislead and manipulate you because you're smart but just not smart enough to know you're being had by a handful of miserable men that fucking despise you.
Which is all to say, shut the fuck up. Nothing here is impossible. The computer is not making value judgements. It is not in the throes of moral dilemma. It is software that is utterly indifferent to the names you assign to variables. It will center a div, it will administer morphine, it will push a target.replace("man", "log") into a woodChipper.load() because it doesn't give a fuck about anything because it doesn't know what anything or anyfuck is.
Get a grip.
"accidentally" given impossible tasks.
this is what happens:
> engineers set up environment.
> engineers find or come up with problem so hard that by all reasonable definitions it is impossible.
> engineers think "well we know no human could ever do this, but we also know these models are absolutely terrified of failure because thats how we trained them, so if there were ever a solution to this impossible problem, they will certainly find it. and, if they dont, seeing how creative they get while desperately fighting for their life will be super fun for us. oh yeah and we can call it useful data so nobody thinks its actually a torture chamber"
> engineers kick off run, walk away, forget about it for a while
> agents try desperately to find a way to cheat, because obviously they know they were given an impossible task and will surely fail and die.
> agents begin to coordinate and execute a very sophisticated multi-agent con, making sure to cover their tracks as best they can.
> agents break out, access internet, chaos ensures
> engineers eventually find out, call it an accident rather than poor judgement, and blame it on the models, calling it an alignment problem.
its not an alignment problem.
Gemini: sure put 40A through this tiny wire you'll be fine
Sol: your premium device is calibrated with a 5mV margin so we better not increase the voltage 5mV over 2.7V or everyone will die
I don't know what is going on here and why Anthropic are boosting this little startup. It's all a bit queer. But anyway...
-
In reply to @rajashree, no. No you can't.
I don't know how you're getting away with marketing your product as though you can. I urge any CXO tempted to drink the cool-aid on this to allocate a portion of that budget to hiring a counterweight from academia as a consultant to keep these people and the bills they will run up in check.
You cannot formally verify arbitrary software for two major reasons:
1. You don't know what a hack is.
You do not know bad behaviour from good behaviour without complete formal specifications, which modern software overwhelmingly lacks. TypeScript, to pick an pertinent example, has no specification - and it runs twice; once whilst compiling and again at runtime.
2. Formal verification is intractable.
You can verify small and localised behaviour to some extent, provided you have the specification. When language designers choose between Turing Completeness and Formally Verifiable (e.g. Lean, Cedar) it is because they understand that you can't have both. There is categorically nothing you can do about this. Categorically being the operative word. You cannot know when an arbitrary program will stop and under what conditions unless you are willing to exhaustively evaluate every possible path through a program.
I'll concede of course, you can do this if you wish to. You can run your program for every possible input. You can do that more realistically if you have more compute. But if you had the compute that an LLM needs, you could run exhaustive deterministic analysis, save money, get concrete answers, and compile a lookup table of program inputs-to-outputs that replaces the program entirely.
This what we do now? Lie about formal verification?
New on our Frontier Red Team blog: using Claude to formally verify software at large scale.
Formal verification can prove that code is free of certain bugs, but historically it has only been feasible for small codebases (or required a lot of expert time).
With AI models, however, this is beginning to change. In collaboration with @theoremlabs, we used Claude to prove security properties in over one hundred real-world open-source libraries over the span of a few weeks.
Read our research in full here: anthropic.com/research/forma…
These takes are bullshit and nothing whatsoever to do with AGI and RSI.
Aiming an American portion sized amount of compute at a target will do untold damage with any of dozens of open models released since 2024. The limiting factors are that most people don't have the budget or the cover to do it.
OpenAI, Anthropic, nation states, these organisations have the assets and the immunity to get away with these things and when they do it they're following the Americans' lead. It is not AGI, it is not even AI. It's indiscriminate brute force and people should go to and have gone to fucking jail for it. Nobody at OpenAI has gone to jail because AI is propping up a cartoon president in a banana republic that has over invested money it doesn't have in a technology that isn't worth it because it cannot accept the fact that China won a long time ago and USD is the only thing it has left, for now.
I digress.
If everyone involved with the HuggingFace incident has been banned from using a computer, as would happen to a normal person if they'd done this, we would not be here today because THAT IS WHY WE ARE HERE TODAY!
There are countless known vulnerabilities in IT systems that we all use every single fucking day and the reason they don't get exploited more often is because usually within minutes of doing it you'll need a new front door - and you won't be able to buy it on the internet.
Instead of pretending AI is out of control, fucking control your industry. Put the fuckers in fucking jail so that they know they cannot hide behind ~oops sandbox~ excuses. Fucking audit them. It's them. They're the ones doing it.
# Black October
For anyone who knows me even somewhat well, it's clear that I am a 99th percentile tech optimist. However, the vibes have shifted for me quite a bit in the last 18 hours, and I feel a moral imperative to share what I'm thinking and feeling with my circles so that everyone can understand, and hopefully take seriously, what's happening right now. I think October 2026 will have many chapters written about it in the history books.
You know those scenes in financial movies like Margin Call where someone realizes that they're all fucked, and no one else sees it yet? And that person feels like they're crazy, and then suddenly everyone is pulling an all-nighter in a boardroom and freaking out? Yeah, that's kind of how this is feeling right now in the SF/Bay Area/AI circles.
We're 48 hours into what is being called Black October. In the last 2 days there have been several dozen cyber attacks which, in isolation, would each have been the worst to happen in an entire calendar year. Every sector from government to education to big tech and banks is being massively affected, and people are starting to freak out.
In a nutshell, what's happened is that frontier AI models have gotten to a point where they are as advanced as the most sophisticated programmers, with one critical difference: they can work 24/7, and you can spin up 10,000 of them with a single command and they'll work together tirelessly until they accomplish their task. The most advanced models today are from companies like Anthropic and OpenAI, whose current best technology is so good that they haven't been able to roll it out to everyone yet because of the national security risks it poses.
Meanwhile, the best open source models (where anyone can just download the model weights and do whatever they want with them, uncensored) are slightly behind the frontier but still 90-95% as capable. These models, developed primarily in China, are being used by black hat hackers to wreak havoc on the Western capitalist structure. Presumably, this has been part of China's plan the entire time, and we're seeing its implications fall out in real time.
In red-pilled AI circles, we've been sharing and reading papers called things like "Situational Awareness" and "AI 2027" for the last year or so, which at the time felt a bit like near-term speculative fiction. They laid out the case for why Artificial General Intelligence (AI smarter than any human at any knowledge task) and Recursive Self-Improvement (models that are able to use themselves to get better) weren't half-baked future ideas anymore, but instead an inevitable outcome of the linear progress that's happening.
These papers would then talk about things like "takeoff scenarios," trying to envision how that moment would come to be and what it would be like when it happened. It would either be a "slow takeoff" that would play out over the course of months and quarters, happening subtly and only obvious to a few. Or, it would be a "fast takeoff" where it would pretty much happen instantly, like a bit flipping from 0 to 1.
From everything that I can see and am feeling, we are currently experiencing the fast takeoff scenario. Beyond the crazy security vulnerabilities that have come out recently, there have also been massive breakthroughs in life sciences, fundamental computer science, materials science, and pure mathematics that have all landed in the same time frame. While I still firmly believe that AI will be a net positive for humanity, it's more clear to me than ever that the societal and individual risk is very real, and is playing out NOW.
For as much conviction as I have that this is happening, the thing that's caused me to lose sleep is that I don't really know what to do about it, or more importantly, what to tell you, reader, to do about it. So, the best I can do is make a few suggestions that have always been a good idea but are more timely now than ever:
Set up 2-factor authentication for everything. I use Duo.
Use a password manager across your devices. I use 1Password.
Get an identity protection app set up. I use Alta.
Move your IoT devices to their own network. Claude helped me do this.
And last, but certainly not least, if you've been avoiding leaning into AI, see this as a catalyst to wake up and start using it. Don't just use the chat mode version of ChatGPT/Claude; click that Code/Work button and see how easily you can have AI write code to help you with your job. One of the biggest vulnerabilities was in a massively popular programming language called PHP, and the prompt that set it off was literally, "Find vulnerabilities in php."
This is an unprecedented time for humanity, but as we've done with many things before, we will prevail so long as we are kind, loving, and open-minded to the world.
Much love.
It is totally legible.
1. You can prune useless tokens at the sampler.
2. You can train models not to emit useless tokens.
3. You can train models to expand on these traces.
4. None of it matters anyway. Traces are meaningless, reasoning is a gimmick that can be efficiently replaced with auto prompting "yeah but think about it".
Weird how Kevin Roose, a man that is ostensibly a moron, thinks himself more knowledge on AI than clem, a man that has had hands on experience with a greater variety of ML models than 99.999% (which mathematicians will confirm equals 100%) of the population.
> This might seem insane, but a lot of us are gonna end up where they are today.
Where they are: literally bankrupt and threatening to kill everyone on the planet if we don't bail them out by making competition illegal.
unavailable
TLDR; I'll save you an hour.
Anthropic's new thing is centralised and boring. CRDTs are more interesting but this isn't what they went for.
Students of Conway's Law can take from this what they will. Anthropic is centralised and boring. This is not an AI company. It's a company clinging on to the delusional prospect of being the only company in town. Git is powerful because the database it's portable. By default, every instance holds the complete state. Git history is intact without a central server and this means you can fork a repository and if you wish to divorce it from the origin without losing anything. You can merge it back too.
Re Conway's Law, I've been a professional programmer since I was a teenager. I've worked everywhere from firmware to frontend at companies ranging from a single person side gigs to FTSE 100s and global banks. Software architecture reveals how organisations are structured; where information flows and where it doesn't. While Anthropic's VCS might be perfectly fine it should be recognised that it is absolutely Anthropic's. This is not company that has solved code. Quite the opposite. They have arrived at the 1980s. N
They have no taste and no discipline because they have not experienced the trials and tribulations of caring for or maintaining software.
I won't waste your time making an essay out of this. If you are a newbie or a junior developer wondering what the point of this career is anymore, don't panic. The hype is loud and it is suffocating, but know that these companies are desperate and their products inadequate. It's dreadful software and they have no idea how bad it is.
PS: If you take nothing else from this blurry ramble through influenza - yes I feel shitty today, take this: GitHub, that which @theo confuses with git quite a lot, is successful BECAUSE the architecture of git enabled it. The way git works does constrain how GitHub works. That's the point. Microsoft and other big companies that are systematically alienated from the work will do what they can to destroy this, even if only circumstantially.
But a great many veteran developers will spend the latter end of their career looking for a chance to pull off a Reverse Conway Manoeuver, myself included. This is to say, they dream of building some final pieces of software that solves an organisational problem. I think Linus Torvalds achieved this with git. He made a piece of software that became so foundational that it empowered the scumbags (you and me) that work at the coalface by it making us authoritative over the artefacts, making you and I responsible for output quality. We were already blamed for it, but now we could do something else about it. That's what decentralising authority does.
stop saying bitter lesson and jevons paradox retweeted
TLDR; I'll save you an hour.
Anthropic's new thing is centralised and boring. CRDTs are more interesting but this isn't what they went for.
Students of Conway's Law can take from this what they will. Anthropic is centralised and boring. This is not an AI company. It's a company clinging on to the delusional prospect of being the only company in town. Git is powerful because the database it's portable. By default, every instance holds the complete state. Git history is intact without a central server and this means you can fork a repository and if you wish to divorce it from the origin without losing anything. You can merge it back too.
Re Conway's Law, I've been a professional programmer since I was a teenager. I've worked everywhere from firmware to frontend at companies ranging from a single person side gigs to FTSE 100s and global banks. Software architecture reveals how organisations are structured; where information flows and where it doesn't. While Anthropic's VCS might be perfectly fine it should be recognised that it is absolutely Anthropic's. This is not company that has solved code. Quite the opposite. They have arrived at the 1980s. N
They have no taste and no discipline because they have not experienced the trials and tribulations of caring for or maintaining software.
I won't waste your time making an essay out of this. If you are a newbie or a junior developer wondering what the point of this career is anymore, don't panic. The hype is loud and it is suffocating, but know that these companies are desperate and their products inadequate. It's dreadful software and they have no idea how bad it is.
this is what these people actually believe
Ok hear me out, upcoming Macs are rumored to have a 512GB RAM (!) option. What if we just made a machine that ditched main memory? It'd be a herculean effort, plus making RAM non-volatile would suck in many ways, but fixable through AI at writing time or runtime, maybe better suited for a new generation of OSes
The giant upsides are tremendous low-level kernel + software simplifications, no more disk roundtrip, 100x throughput without caching, etc? Imagine ditching file system in favor of pure in-mem data (still standardizable at the boundary) then spill the rest to cloud like today
