collecting these while they are still useful:
"exercise abstract thinking"
"deepen modules and simplify interfaces"
"Be thorough in your analysis and bold in your recommendations"
"find all of the implicit state machines in my app"
Saf retweeted
agents are going to cause a breakdown of human behavior assumptions underlying today's systems. i've started thinking about it as friction collapse. without friction, a number of norms will collapse.
consider selling tickets to a concert/event. today, you can post them at a certain time and it's first come first serve. soon everyone's agent will persistently monitor, and as soon as they are posted for sale, thousands of people will instantaneously be trying to buy the same ticket. how do you even resolve this? sell the ticket to the person whose agent gets there fractions of a second before?
we're seeing the same thing with job applications, restaurant reservations, but basically anything that relies on participation requiring effort is going to break down. sales emails/calls, participation in government benefits, apartment rentals, doctors appointments, RFP responses. customer support won't be able to handle the user who will not drop their point of dissatisfaction. with no friction, these systems just don't work anymore.
business models will break down:
- flights/hotels will be consistently cancelled and rebooked at the lowest price (great use case for your agent if you don't have this already set up)
- subscriptions will be cancelled
- retailers won't be able to arbitrage price differences as the agent will find the cheapest place to buy anything
- insurance denials (health/auto/home/etc.) will be fought tooth and nail
- bank interest rates will be optimized and avoidable charges will be avoided
a lot of what we perceive as stable social coordination is actually an equilibrium sustained by human inefficiency. rationing is a product of human slowness. signaling effort/interest/intent is a key part of our social infrastructure. inertia is a key to many business models. information asymmetry is everywhere. and attention is monetized thoroughly.
with agents, none of these things are true. humans will simply demand what they want and have their agents get it for them. there is going to be a fundamental renegotiation of how markets allocate resources, how people signal intent, and how businesses capture value.
Saf retweeted
One thing I've noticed with tests written by agents: they tend to encode "does this work the way I wrote it" rather than "does this work the way it's supposed to"
It is a maddeningly subtle distinction that takes years of experience to discern and agents can't, yet
This will re-convince thousands that AI is still the fad they remember from 2023
To save costs, this service is probably powered by some shitty local modal. Also requires blazing fast inference time for the immersion experience; overall it'll always produce slop like we see here
Okay wtf why is everyone saying this now. Also knew i wasn't tripping
Agent computer use is currently such a scam. Shit is broken half the time and eats tokens like no tmrw
Saf retweeted
This is the most serious 48 hours in cybersecurity history.
"Black October" they're calling it
It seems every *hour* there's a new incident, that in a typical year, would be one of the biggest incidents of the *entire year*
SUMMARY OF THE LAST 48 HOURS:
YEAR-LEVEL INCIDENTS (In a typical year, these would be among the biggest incidents of the year):
1. PHP2Shell: a hole in the software behind ~70% of websites. Governments took sites offline.
2. eMudhra: the master keys that prove websites are real were cracked. One was the number 4.
3. California: data on every public school student since 2023 stolen, including disability status
4. Railway: hackers stole customers' secret keys from a hosting platform used by millions of developers
5. Zeabur: 321,810 accounts breached. The company is shutting down.
6. LiteLLM: an AI tool downloaded 3.4 million times a day was poisoned with a password stealer
7. GitLab: a 10-out-of-10 severity hole in every version since 2014
8. Cairn: criminals used an AI hacking tool to break into 100+ companies
9. Ahold Delhaize: hackers knocked the owner of Food Lion and Stop & Shop offline
10. Onamae: malware planted on up to 26,739 customer servers in Japan
MONTH-LEVEL INCIDENTS (In a typical month, these would be among the biggest incidents):
11. Signal: a bug could leave your mic and camera on after you hang up
12. NGINX: 3 critical holes in software running ~1/3 of websites
13. HTTP/3: a single connection can crash 7 of the 10 web server types tested
14. Plex: hackers breaking into home media servers
15. SafePay ransomware: FBI warns it has hit 600+ critical infrastructure organizations
16. Palo Alto: ransomware gangs breaking in through company firewalls
17. Bullhorn: platform used by 10,000+ companies knocked offline
18. Medi-Cal: 970,000 patients' data stolen
19. WordPress: hackers stole a plugin maker's keys and used them to plant backdoors on websites
20. AOMEI: backup software's update server handed out a backdoor
21. Meta Muse: Meta's AI agent exposed users' personal and business secrets
22. Tor: 333 relays hijacked to steal logins
23. TiDB Cloud: database company's cloud breached
Also serious:
24. State Bank of India: India's biggest bank confirms a data breach
25. OVHcloud: hackers slipped a password stealer into a cloud company's official tool
26. Siemens: a software hole being used to spread malware
27. North Korea: state hackers launch a new attack campaign on South Korea
Saf retweeted
"opus-5.5 is a coding superintelligence on par with world class experts and bad code is a user's skill issue at this point" is an increasingly popular take being echoed around. so, what are we still doing here?
i've been coding through various robotics ML projects for the past ~3 years. everything ranging from throwaway r&d codebases during early conceptualization to stable and optimized codebases during later productionization. i'm by admittedly by no means a "world class expert" in coding, and since gpt-4 was released just as i was getting interned in what is my first full-time engineering job, i've consistently leveraged the capabilities of these models and evolved alongside them every step of the way from the humble days of gpt-4 to the undeniably cracked opus-5.5 and on.
now the interesting question that comes to mind is that since i was born too late to independently develop deep practical expertise in any engineering subfield that i can then capitalize on to confidently supervise and steer those models, what do i even contribute to any of my AI-assisted projects? am i really just a middle-manager between upstream management and downstream coding agents? well, i don't think so, or at least not yet.
while modern agents definitely don't lack engineering know-how and a versatile technical skillset, i've always found, and do continue to find, that they do lack the kinds of natural commonsense that we humans take for granted. models remain unbelievably shortsighted, inefficient, and autistically oblivious to various unverbalized expectations around outcomes. despite superhuman coding skills, even the legendary opus-5.5 finds it impossible to hold back from diving head-on into unsustainably hacky solutions, paranoid defensive coding patterns, confused design choices with utter disregard to obvious long-term trade-offs, and above all, painfully bad taste decisions around high-level project directions. i am yet to take my hands off the steering wheel and not find myself frustratingly interrupting to re-steer soon after.
the thing about this particular set of cognitive blindspots and shortcomings that i find especially unique is how orthogonal they seem to be relative to all other predictable improvements in capability. since the early days of gpt-4, i don't find it to be the case that models have improved much on minimalism-by-default, on reframing problems instead of stubbornly hacking through them, or on effectively leveraging theory of mind to satisfy general expectations in regards to writing style be it code, documentation, or anything involving external readers. the degree to which i find myself putting in constant effort to verbalize obvious human intuitions into explicit and persistent notes and skills to auto-steer agents away from their alien (not the intelligent kind) basins leaves me disillusioned with their independent judgement to this day.
i think our reward functions are incentivizing the wrong priorities when attempting long-running agentic work. i think our modeling algorithms are failing to reliably internalize fundamental human biases, such as the deep desire for efficiency in all aspects of life (aka "working smarter, not harder"). i think the capacity to generate beautiful original ideas and reason about next steps at a high level remains largely underoptimized across the current frontier. i think we don't yet have AGI in the sense that we can expect from it the basic level of sane, well-calibrated intuitions we'd expect from a competent junior engineer, let alone a senior domain expert, despite obviously superhuman technical knowledge and skills.
tl;dr: don't leave the driver's seat just yet. impressive lane assist and auto-parking is where we're currently at.
the coding agent haters are getting dunning-kruger'd so hard right now
do you really think that your engineering ability is better than the models at this point?
it is very very likely that fable is better than you in every branch of software
it has been trained on all the code of the internet, it knows almost every trick in the book, all the design patterns of every domain, every single low-level detail, every bit-twiddling hack, it's read every single blog post from every cracked engineer that ever existed
do you really think you can beat it? do you really think you can be more creative than it?
unless you are 99th percentile expert in some niche domain of software, it simply is not possible. sorry.
AI normie litmus test: emojis showing up in the very first output of a new chat session (regardless of prompt)
Saf retweeted
"Oh this? This is just an entity from the nether realm I summoned by enscribing runes on magic stones I forged in a sacred kiln, then proceeded to cast certain spells to distill the collective conscious to imbue it into the substrate. Don't worry, it doesn't have a soul."
Saf retweeted
Look at him, thats my Qual.
Your what?
My Qualitative! My vibe specialist. Look at him, you notice anything different about him? Look at his face.
I'll give you a hint, his name is Jayden. He won a local poetry competition in bushwick! Yeah I'm sure of the vibe.
Quant firms are hiring idea guys, not quants
I've been skeptical about the whole - software is commoditized, ideas are the moat argument
But I've seen first hand the best quant and technical firms in the world hire thinkers, liberal arts grads, and generalists
I predict the trend accelerates
Letter from @AQRCapital below
The lore of this image is fascinating. Not only did the internet completely misunderstand it (top photo is actually more recent the bottom being older) this new circulation of the image is even edited with "first year" and "senior year" labels to further cement the accepted canon
