
human alignment // metacognition @mcognitivelabs // Eastern Roman ☧
Imperial Periphery
Joined April 2009
- Tweets23.3K
- Following2.5K
- Followers20.7K
- Likes147K
can't help but wonder how many "genius"/genius kids were lost to the school system because they lacked a caretaker who a) believed in them b) were independent-minded enough to continue to do so against institutional consensus
"Be a gatekeeper of your heart and let no thought in without questioning it. Examine every single thought one by one and ask it: ‘Are you one of ours or one of theirs?’"
— Evagrius Ponticus, Letter 11
got diagnosed with sleep apnea after chatgpt had a look at my negative at-home sleep study and insisted i redo it in-clinic
my doc had missed out on the subtle signals of the at-home study and didn't think it warranted a follow up, despite symptoms persisting
a couple other docs had claimed that an in-clinic study is not more accurate than an at-home one
"analytical rumination" may be an adaptive response to high-complexity problems that's designed to divert energy away from unrelated tasks & direct it on the identified problems
"it does not bode that well for the major model companies, oai included, that fundamental breakthroughs like this are coming from outside. we've become too much of a pressure cooker focusing on local incentives, studying narrow tracks under old paradigms, mostly doing engineering, rather than fundamental research, and implicitly hoping the next model will figure it out"
a few thoughts:
1. pretraining science is kind of like how people built quite good steam engines before they understood thermodynamics. “Science owes more to the steam engine than the steam engine owes to science.”-(L.J. Henderson). if nick is right, which i am pretty sure he is, progress points in the direction of more interpretability, and pretraining was "slouching towards monosemanticity". my guess would be that major breakthroughs in interp also advance capabilities
2. nick is a good friend of mine, safe to say my origin story at openai wouldn't have happened if i hadn't met him, and it's fascinating watching him operate and seeing the early drafts evolve into this. he's the most profoundly happy guy I know- not driven by guilt, activism, or religious terror, but by fascination. he of course understands the stakes of understanding neural networks, but it doesn't seem like the core reason he does it: he just loves the neural nets. afaict he's posting this from some beautiful hideaway in tokyo. there's a lesson in there for all the +EV maxers. i feel very grateful to have met him
3. i do think it's crazy how few people across the world were staring at neural nets growing: most have moved to a different level of behaviorist analysis, thinking about reward functions and test scores and whatnot. it does not bode that well for the major model companies, oai included, that fundamental breakthroughs like this are coming from outside. we've become too much of a pressure cooker focusing on local incentives, studying narrow tracks under old paradigms, mostly doing engineering, rather than fundamental research, and implicitly hoping the next model will figure it out
wow great paper
perseverative cognition (worry, rumination) may be protective in the short-term bc it generalizes & prevents from focusing on specific, emotionally-heavy episodes, but is costly long term bc it makes it harder to update from disconfirming & corrective experiences

























