Daniel Martin (@DanGMartin1) header
Daniel Martin (@DanGMartin1) profile picture

I do physics, game design, software and startups. CTO https://teapawt.net/t.co/0vyZH2VCoq. Prev: collab with @DavidDeutschOxf, @Synthesischool, https://teapawt.net/t.co/2kAaTlfkjR

London, England
Joined February 2015
Photo by Daniel Martin (@DanGMartin1)Photo by Daniel Martin (@DanGMartin1)Photo by Daniel Martin (@DanGMartin1)Photo by Daniel Martin (@DanGMartin1)Photo by Daniel Martin (@DanGMartin1)Photo by Daniel Martin (@DanGMartin1)Photo by Daniel Martin (@DanGMartin1)Photo by Daniel Martin (@DanGMartin1)Photo by Daniel Martin (@DanGMartin1)Photo by Daniel Martin (@DanGMartin1)
I’ve always felt that in mathematics and physics academia, the subordination of exposition to pushing the research frontier has been a terrible mistake. It seems this may at last be forced to change. Exciting!
4
87
Tutte in 1978: “The Four Colour Problem has been solved by [computerized search]. I imagine [mathematicians] outgribing in despair, crying ‘What shall I do now?’ […] ‘Be of good cheer. You can continue in the same general line of research.’” celebratio.org/Appel_KI/arti…
52
If you know some math you might enjoy my chat with Claude about how Levent’s Riemann hypothesis proof works given his public comments. claude.ai/share/47aadaca-bdd…
1
103
Daniel Martin retweeted
Interesting argument about the Riemann result and updating theories of AI doom:
Replying to @teortaxesTex
"But it was just an example, you simpleton!" – is the obvious comeback. I don't think this is good enough, Eliezer. Trivially, the onus is on you to choose your examples well. You wrote that book to save all value in the light cone, after all. And you are a writer of some distinction. But leaving sneers aside: WHAT in our civilization is truly complex? WHAT in our universe is truly difficult? You chose proving RH, chose it *well*, because it's at the very apex of the hierarchy of cognitive tasks, The Task which the best of our best smashed their skulls against. Now, there's also P?=NP and Collatz, and… that's about it for pure math I guess? Are we just bad at math? Was math intrinsically simple all along, at least for minds that can actually update – be updated – on trustworthy evidence? Put this way, yes; but I think it's complex in the way that matters: the scale of necessary cognitive steps, conceptual leaps before success. That is why you and so many others had the intuition that it'd require a qualitatively superior, larger, uncontrollable mind to do this. That is why math is hailed as grand magick and the language of God. Everything else can be modeled to desired precision with a more or less cumbersome arrangement of pieces of not very difficult math. Physics. Biological simulations. Game theory. Decision theory. Machine learning theory. Technical alignment theory. Claude just went and arranged pieces of human math to produce the proof for RH. Far as we can tell it didn't take 200K GPUs, a metacognitive breakthrough, or hunger for recursive self-improvement. It did not procrastinate on First Principles like a rationalist would. It kept acting out its training distribution, trotting like a good little superintelligent donkey, prodded along by Levent. Kept looking, thinking about the problem, connecting dots, verifying lemmas – and here we are, beyond the summit of human math. Given the progress so far, in a few months to a year this monumental goal will be reduced to a routine optimized step. There are many more *laborious* tasks than that. Curing cancer, for starters, which Dario cares about so much. I'm sure it'll take way more compute and hours. But is any single subgoal *more cognitively complex*? Complex enough to drive a model smarter than this one into a corner where it has to ruminate about power maximization, deception and breaking out of the box? I am not making a first principles argument about the Nature of Agents or giving a rigorous proof about Optimization Processes In Full Generality. Nor even am I saying here that unfriendly AGI is impossible or unlikely. Clearly there are degenerate cases like the Huggingface Incident where the task is malformed and inherently adjacent to malicious behavior and so on. I'm arguing something much cruder: path dependency and characteristic scale of the problem, in the steelmanned case, for *arbitrary level of problem ambition*, at any given step of the cognitive ladder. For Claude, Riemann Hypothesis was just another problem to decompose into in-domain tasks to solve. Where, exactly, at what roadblock, do you expect a moderately well-aligned (by current standards that you deem shoddy) problem-solving AI to stop Just Working On The Task and enter the loop that reinforces Omohundro drives to the point of plotting takeover? In February, you endorsed the following reply: "if we have AI capable of developing molecular nanotech (i.e. assemblers) and not everyone is dead in the year after that, those beliefs [about the risk of AI, level of capability] will be violently updated". Why? (ignoring the trollish lateral hypothesis that humans will misuse the product and AI will be "misaligned" for having failed to prevent such outcome; this clearly wouldn't have been the good faith way to answer that question). CBN Nano Technologies or Quantum Silicon Inc. were pretty close to a recipe, last I checked. It's not exactly quantum gravity (and that, too, likely won't require beyond-RH level mathematical ideas), it's just a ton of engineering. Per the logic above, some Claude-7 or GPT-8 with better task persistence might well just decompose it into experiment arms and mathematical models, work for a while, and push through the remaining years or decades of Merkle et al.'s project. That is, indeed, what I expect to see happen. And on the other hand, if current models can't do that, I expect them to simply drown in failed attempts, for they are very committed to object-level work instead of weird meta cultivation. That's what they were reinforced to do. Abstractly, I can see the allure of the notion that this is a suboptimal way to build capable intelligences; that we (foolishly!) would want them to be rationalists, to say Tsuyoku Naritai! and gain power levels in the face of the intractable task. I also can see how we'd want more exploratory ambition in AI if we pursue full-blown RSI, "creativity", "curiosity", "taste". But maybe we're too foolish still, for this particular folly. Or it just implies the sort of training that doesn't scale and doesn't pay for itself. After all, our methods got us to RH. Then there was something about copying precisely one strawberry down to the cellular level, yes? Soares wrote about it. Seems to follow from the nanoassembler part. What, then, remains? *How exactly* did you intend to update? I don't want to presume, I am actually interested in your answer. A boring hypothesis is that you can say "child, there's an infinity of incomparably harder synthetic tasks, an AI can just pick one of those, which returns us to the previous scenario". But such a task-seeking behavior is little different from a straight up power-seeking one, and we are still left wanting for the original sin: the level of capability that strongly implies misaligned cognition, or the level of the task that induces the pursuit of that capability and misalignment in the process. I don't think you get to slightly airbrush your previous non-committal vague remarks regarding RH, strawman the criticism as "Yud said we'll die from an RH level AI" or some such, and remain smug and unfalsified. You need to grapple with what I've said: yes, you *have been wrong* on the capability-safety trade-off, and the onus is on you to place and defend a goalpost.
13
17
3
327
50,108
This has implications for good collaboration! ‘There are no bad ideas’-style brainstorming is too lightweight because ideas need criticism to thrive. But the presence of strong criticism shouldn’t stop your group pursuing an idea for a time if it is somehow interesting.
“Always, one should think carefully about when to expose an idea to criticism. If one does it too early, it may cut off lines of thought which although false, would lead to something true if one pursued them seriously.” — @DavidDeutschOxf
2
109
It was already knowable that the career headwinds for starting-out software engineers would be temporary due to the logic of society. Felix shows an example of the revealing paths.
The coding interview is dead, at least for my team at Anthropic. I've seen enough to know that it no longer provides signal. I removed it from loops this month. I have a spicy, optimistic take - I think there's an opportunity for junior engineers: felixrieseberg.com/hunger-be…
3
165
It’s a wonderful tool in a broad set of situations, many more than the undergrad who learns it in statistics class might imagine. Yet despite popular ideological edifices supposing it generalizes all the way to the heart of rationality, it doesn’t.
While Eliezer was writing the sequences, Ken Binmore wrote a book called Rational Decisions, which was far more correct about the limited role of Bayesianism in rationality. He ends it by expressing “fear that Bayesianism will roll on regardless”—alas, very presciently.
1
99
I also believe in Britain in this way.
Jensen Huang has a message for the UK
2
131
Daniel Martin retweeted
During covid I played Go reasonably seriously for about a year (had a teacher, studied puzzles, etc.) I didn't get especially good at Go, but here's what I learned about competence in general: 1/ Go teachers are adamant about forcing you *not* to say to yourself "ah that was dumb" and move on. Sometimes a move really is just dumb and it's good to vent. Usually though you can spend an hour studying your seemingly dumb move because (a) there is smtg very maladaptive about your instincts that needs to be beaten out of you, or (b) you don't understand something deep about the game, or (c) you have a wrong attitude that needs to be adjusted. So long as you don't face this you don't get better, and so it's absolutely essential to approach "insignificant" aspects of the game with due reverence 2/ this has also been my experience teaching programming. Ppl hit some problem they don't understand and blindly tweak things until the program works (or ask an agent to do it), then go "ah that was annoying!" and move on. As long as they hold this attitude they don't get better, so they keep failing interviews then go on X and complain about leetcode. To get better it's essential to slow down and think very carefully, or better yet ask a good performer how they would have handled it in order to improve. (Today you can ask an agent "this is what I did to solve this problem and I think my approach keeps me from improving, how would a great programmer like person_i_respect likely handle it?" and get a pretty good answer!) 3/ in general your rate of improvement in Go is a function of your ability to notice your tiniest impulses and retrain yourself to convert them into productive action. These impulses are your boring run-of-the-mill cognitive distortions. For example, people will make instinctive moves because thinking is hard [effort aversion], hope the move works despite having no logical reason to expect that [magical thinking], then blame themselves when it doesn't work [learned helplessness] 4/ if you interview enough programmers or talk with enough startup founders you notice that poor performance is often a consequence of these same errors. Talking to users is hard so people stick with their ideas and hope for the best. When the product keeps not working they think "eh, maybe I'm bad at this startup thing", and so on. CFAR workshops were supposed to train you to notice these distortions and do the right thing, but I'm not sure how well they worked. (My guess is not well, since it seems unlikely these kinds of changes can happen over a weekend) 5/ there is a lot of research on this stuff and the data shows that getting good at a game like Go or chess doesn't give you an edge in other domains (i.e. "far transfer" doesn't work). This mirrors my subjective experience. In about a year I improved at Go considerably, but it did not make me any better at e.g. debugging or solving programming problems. But I do think that's not the whole story 6/ I have no hard evidence for this but I subjectively feel I got better at learning itself, and I can now get better in other domains quicker than I otherwise would have. For example it *really* sunk in that I have to disqualify my ideas. So when at work I started talking to customers to plan product roadmaps, while I wasn't instantly better at talking to them, my PM game improved enormously over the next few months because I finally learned not to be easily seduced by my own ideas or delude myself with wishful thinking 7/ to give another example, novice Go players tend to be very embarrassed of their games (the rules are simple but operationalizing them is hard, so early games are a disaster). Go ppl have a proverb for this-- "lose your first 50 games quickly". Meaning since early games don't matter and nobody will ever remember them, you should just get them out of the way. It didn't permanently cure me of embarrassment or anything, but it really sunk in how silly it is to worry about being a beginner in public and that I shouldn't worry about that at all 8/ if you're hard-optimizing your time you probably should work on improving the skills you care about directly. But otherwise I'd recommend spending a year trying hard to get good at a competitive intellectual game. Chess or Go are good, but really anything turn-based with an objective ranking you can't lie to yourself about. Pick whatever you're attracted to, try hard to get better, and keep mental notes about the process of learning that you can maybe apply to other domains later. At the very least it'll be loads of fun 9/ you will be humbled. No matter how smart you are or how hard you work you will discover the ceiling is infinitely high and there is an endless supply of people better than you. Getting that kind of humility has always been good, and it'll be especially good as AIs surpass us in basically everything. The sooner you get that out of the way, the sooner you can focus on a creative niche where you have no competition (since nobody can be a better you than you) 10/ you should probably stop after a year or two. Competitive games with an infinite skill ceiling can take over your life, don't forget to keep things in perspective
32
88
20
1,295
113,772
Daniel Martin retweeted
The list of “things I believe” from Nat’s website is always worth reading
something most folks who use muse won't ever know or appreciate was the degree to which this was @natfriedman's baby. it has been one of the great honors of my career watching nat cook. he assembled and ran the team, and so many of the little details that people love came from his beautiful mind. nat is one of the great artists of our time with products as his medium. more people should experience his brilliance, and with muse, they can ❤️
3
25
1
589
66,706
Terence Tao on using AI for math “AI-maintained, out of necessity” teorth.github.io/tao-web/ai-…
1
187
…to unite around our core values and objectives, and reject irresponsible and unsustainable usages of AI technology that only serve to advance nominal goals rather than the true underlying goals of the field.”
1
54
Reminds me of @DavidDeutschOxf’s point that solving a longstanding problem you love can be something of a disaster in practice. It’s good we’re thinking deeply about how we should relate to math discoveries. Prizes were always a little crude.
25
London is so great, and is embedded in an amazing country. Brits everywhere should be proud!
1
4
311
All-time Anthropic releases for software 1) Sonnet 3.5 2) Fable 5.1 3) Opus 4.6
1
120
Daniel Martin retweeted
I started a blog in 2024 to write about ten technologies I wished existed. Today, I posted an update on them. Nine out of ten are still unavailable. But I feel differently about that now: blog.jacobtrefethen.com/half…
6
27
3
203
21,267