Pinned Tweet
sharing an essay on the relation between compression and intelligence, in commemoration of Ray Solomonoff's 100th birthday greene.sh/compression-and-in…
instrumental convergence is framed as something rational agents do, but evolution suggests it’s something the universe does
a universe saturated with intelligence and a universe at maximum entropy may be indistinguishable from the perspective of a computationally-bounded observer
TIL von Neumann's self-replication architecture preceded our understanding of the mechanics of DNA (which turned out to be the same)
the economy of a world with greater than human intelligence will not be ran by humans
so just to clarify the whole cerebras drama:
- openai *is* using cerebras chips for ultrafast but they ran out because Jane Street bought up all their supply
- jane street’s paying cerebras $200M PER megawatt just to access super fast inference. that’s insane. cerebras cost is ~$26M per mw.
so cerebras has a good chip + is making bank?
Ryan Greene retweeted
We've been obsessed with bending the scaling laws recently and our new paper shows that looping with model growth improves the scaling exponent, leading to gain that compounds with compute!
Everyone assumes architectural changes only give constant factor gains and pretraining progress mostly comes from data (e.g. @dwarkesh_sp's recent post).
We found that model growth, looping, and boundary operators result in compute multipliers over standard transformers that grow exponentially with each OOM of compute.
- 1.55x at 1e20 FLOPs and 2.7x projected at 1e25.
- Matches GPT-3 13B on CORE with 20x less compute
w/ @charllechen, @akshayvegesna, @andrewgwils
🧵
Ryan Greene retweeted
I am hearing disturbing rumors that Ilya Sutskever has made the Boltzmann Machine computationally tractable.
an impressive jump that i have to assume is the result of being the first frontier model to have RL'd the public demo environments, which is a rather large confounder on what Chollet is trying to get these benchmarks to measure (out-of-distribution generalization)
oh, you’re a capabilities researcher? are you minimizing the description length of the digital universe, or embedding it with a will to power?
Schmidhuber's obsession with credit assignment is just another example of his great taste in the most important research directions :)
this is a fantastic book, highly recommend it and checking out Antikythera more generally
great visualization! discussion of this paper was my favorite part of What Is Intelligence by Blaise Agüera y Arcas
whatisintelligence.antikythe…
for a little toy project i reproduced the quoted artificial life paper: a 2D grid of randomly generated Brainfuck programs breed and spontaneously evolve self-replicators, despite no explicit optimization functions
every pixel is an instruction; each instruction has a unique color, while black represents a value on the tape that is raw data storage / not an instruction. every 8x8 section of pixels represents a single program
in this run, a self-replicator emerges relatively early on and soon takes over most of the grid, until a more efficient self-replicator evolves and takes over everything
the entire program is just 300 lines of code, most of which is just boiler plate for parallelization and gif generation github.com/Rabrg/artificial-…
