quasar

The dead internet scenario is very possible — LLMs stop improving and actually degrade because they are trained on their own slop. The incentives for humans to write novel stuff and publish are also near zero. The worse risk is that it will make humans dumber in two ways. They unlearn how to do things without LLMs. They don’t share knowledge anymore. I don’t know how we prevent that future. But I think not forgetting how to do things without the robots is a start.

You provide the data for the models to be trained. You didn't opt in, but they took it for free anyway. There's no attribution whatsoever. Then they sell it back to you. Then they tell you selling it back to you is too risky, so they keep the results of your work to themselves. Why would anyone ever publish anything again? We're headed to the new dark ages. Exponential Intelligence Implosion.

I think the idea AI will just generate Formal Verification for code and everything will be fine is very dangerous. The limitations of FV are the same if an LLM or a human wrote it. You "change" the target from the code to whether the formal description actually represent what goes on in the code and if the properties proven are the correct ones, or sufficient for what you need from the code. I put "change" in quotes bc if the FV wasn't correctly constructed then the full code is still a target. So you don't actually remove the need to audit the code or the possibility of exploiting the code. It's more like a test/fuzz harness that makes bugs less likely than something that guarantees bug-free code or even reduces the surface to be examined. **And I think mistaking one for the other is very dangerous. In general thinking there is a silver bullet that can guarantee code is safe will make code even less safe.** That isn't saying FV can't be great. It's just not the guarantee some people think. And it's even less of a guarantee when an LLM wrote it for you, and you dont fully understand it (like tests, audits, or anything else if you just trust it uncritically).

If LLMs finding bugs missed by multiple human auditors makes it super-human. When I find bugs missed by multiple humans and AI does it make me super-super-human? When then people catch bugs I missed do they become super-super-super-human? Congrats to teams finding bugs with whatever tools they use. But for the people making a living off bounties, finding stuff missed by dozens of the best auditors is just a Tuesday. Nothing super human about it.