Product @LiteLLM | Ex-Founder @WebhoundAI (YC S23) | Stanford CS ‘23 | San Francisco & Beirut
San Francisco, CA
Joined September 2011
- Tweets811
- Following216
- Followers231
- Likes26.5K
I'm going to try this.
Couple ideas here:
1. Sidecar model that does real time WSP
2. Real-time directional ablation
Anyone have any guidance here?
Dumb idea: the problem with abliterated models is that by editing all weights that cause refusal, you end up making the model significantly dumber.
What if you used real-time weight-space programming to only edit the specific weights that cause a refusal whenever it happens?
That way you could get away with only weight-editing at key decision points, and only the weights relevant to that specific decision.
Is this something anyone is trying?
1
140
Dumb idea: the problem with abliterated models is that by editing all weights that cause refusal, you end up making the model significantly dumber.
What if you used real-time weight-space programming to only edit the specific weights that cause a refusal whenever it happens?
That way you could get away with only weight-editing at key decision points, and only the weights relevant to that specific decision.
Is this something anyone is trying?
So how is this not just a race to the bottom? What’s the differentiation here?
Sandbox provider updated list and prices ordered from most expensive to cheapest.
Again (All prices are for 2 vCPU / 4 GB, running for one hour).
45. Depot — $0.36/hr
44. Vercel Sandbox — $0.341/hr *
43. SuperHQ (Nanite) — $0.317/hr *
42. Runloop — $0.317/hr shutting down Nov 13
41. Render Sandboxes (new, beta) — $0.316/hr *
40. AWS Lambda MicroVMs — $0.252/hr
39. Quave ONE — $0.25/hr
38. Modal — $0.238/hr
37. BoxCompute — ~$0.20/hr (estimated)
36. Tensorlake — $0.20/hr *
35. Upstash Box — $0.20/hr *
34. Cloudflare Sandbox — ~$0.20/hr *
33. Namespace — $0.18/hr
32. E2B — $0.166/hr
31. Daytona — $0.166/hr
30. Tenki — $0.166/hr
29. Thunder Compute — $0.166/hr
28. Promigence — $0.166/hr
27. Fly.io Sprites — $0.165/hr *
26. BoxLite Cloud — ~$0.158/hr (not confirmed)
25. boxd — ~$0.15/hr *
24. Dedalus — ~$0.149/hr (not confirmed)
23. Beam (new) — $0.14/hr
22. Freestyle (YC S24) — $0.132/hr
21. CreateOS — $0.119/hr
20. exe.dev — $0.105/hr
19. Buddy Sandboxes (new) — $0.0864/hr
18. AWS Nitro Sandboxes, on-demand — ~$0.083/hr
17. Northflank — $0.067/hr
16. AWS Nitro Sandboxes, 1-yr Savings Plan — ~$0.059/hr
15. Quivvo — $0.054/hr
14. Mosaic — $0.05/hr
13. Nodus Compute (new, YC F26) — $0.0357/hr
12. Cathedral — $0.035/hr
11. Celesto — $0.030/hr
10. AWS Nitro Sandboxes, Spot — ~$0.029/hr
9. Dory (new) — $0.0288/hr (shared vCPUs)
8. Zode — $0.0225/hr
7. Maritime (YC F26) — $0.022/hr
6. Boat (YC F26) — $0.018/hr
5. Munito (YC-backed) — $0.018/hr
4. InstaBox (YC P26) — $0.017/hr
3. Lizard — $0.0162/hr
2. Cube — ~$0.016/hr sunsetting
1. Agent 37 (YC F26) — ~$0.006/hr (shared host)
teapawt.net/Jccorzot/status/212020…
4
213
Moe Khalil retweeted
Sandbox provider updated list and prices ordered from most expensive to cheapest.
Again (All prices are for 2 vCPU / 4 GB, running for one hour).
45. Depot — $0.36/hr
44. Vercel Sandbox — $0.341/hr *
43. SuperHQ (Nanite) — $0.317/hr *
42. Runloop — $0.317/hr shutting down Nov 13
41. Render Sandboxes (new, beta) — $0.316/hr *
40. AWS Lambda MicroVMs — $0.252/hr
39. Quave ONE — $0.25/hr
38. Modal — $0.238/hr
37. BoxCompute — ~$0.20/hr (estimated)
36. Tensorlake — $0.20/hr *
35. Upstash Box — $0.20/hr *
34. Cloudflare Sandbox — ~$0.20/hr *
33. Namespace — $0.18/hr
32. E2B — $0.166/hr
31. Daytona — $0.166/hr
30. Tenki — $0.166/hr
29. Thunder Compute — $0.166/hr
28. Promigence — $0.166/hr
27. Fly.io Sprites — $0.165/hr *
26. BoxLite Cloud — ~$0.158/hr (not confirmed)
25. boxd — ~$0.15/hr *
24. Dedalus — ~$0.149/hr (not confirmed)
23. Beam (new) — $0.14/hr
22. Freestyle (YC S24) — $0.132/hr
21. CreateOS — $0.119/hr
20. exe.dev — $0.105/hr
19. Buddy Sandboxes (new) — $0.0864/hr
18. AWS Nitro Sandboxes, on-demand — ~$0.083/hr
17. Northflank — $0.067/hr
16. AWS Nitro Sandboxes, 1-yr Savings Plan — ~$0.059/hr
15. Quivvo — $0.054/hr
14. Mosaic — $0.05/hr
13. Nodus Compute (new, YC F26) — $0.0357/hr
12. Cathedral — $0.035/hr
11. Celesto — $0.030/hr
10. AWS Nitro Sandboxes, Spot — ~$0.029/hr
9. Dory (new) — $0.0288/hr (shared vCPUs)
8. Zode — $0.0225/hr
7. Maritime (YC F26) — $0.022/hr
6. Boat (YC F26) — $0.018/hr
5. Munito (YC-backed) — $0.018/hr
4. InstaBox (YC P26) — $0.017/hr
3. Lizard — $0.0162/hr
2. Cube — ~$0.016/hr sunsetting
1. Agent 37 (YC F26) — ~$0.006/hr (shared host)
teapawt.net/Jccorzot/status/212020…
Sandbox provider are trending now, so here is a list of them ordered from most expensive to cheapest
(All prices are for 2 vCPU / 4 GB, running for one hour).
27. Depot — $0.60/hr
26. Vercel Sandbox — $0.341/hr *
25. Fly.io Sprites — $0.315/hr *
24. AWS Lambda MicroVMs — $0.252/hr
23. Tenki — ~$0.24/hr (its CI rate)
22. Modal — $0.238/hr
21. Upstash Box — $0.20/hr *
20. Cloudflare Sandbox — $0.180/hr *
19. Namespace — ~$0.18/hr (rough)
18. E2B — $0.166/hr
17. Daytona — $0.166/hr
16. Thunder Compute — $0.166/hr
15. BoxLite Cloud — $0.158/hr
14. Dedalus — $0.149/hr
13. Freestyle (YC S24) — $0.132/hr
12. CreateOS — $0.119/hr
11. exe.dev — $0.105/hr
10. AWS Nitro Sandboxes, on-demand — ~$0.083/hr
9. Northflank — $0.067/hr
8. AWS Nitro Sandboxes, 1-yr Savings Plan — ~$0.059/hr
7. Quivvo — $0.054/hr
6. Celesto — ~$0.030/hr (its CI rate)
5. AWS Nitro Sandboxes, Spot — ~$0.029/hr
4. Boat (YC F26) — $0.018/hr
3. InstaBox (YC P26) — $0.017/hr
2. Cube — ~$0.016/hr (shared vCPUs)
1. Agent 37 (YC F26) — ~$0.006/hr (shared host)
I think we’re starting to see the effects of frontier labs defending against distillation
It’s really expensive to buy the amount of post-training data they have
the gap between frontier and OSS is quite large currently
this isn't a static thing though, there have been times where it got close
i'm looking forward to the next generation that starts to get close to opus 5.5
1
6
577
I keep going back and forth on cloud agents. They’re obviously the future if you assume number of concurrent agents keeps going up, which I think is the case for a lot of autonomous agents (PR Review, pentesting, quick bug fixes)
But when it comes to personal coding agents, I still way prefer the local experience.
1. They can use my accounts with zero setup without me worrying about my credentials floating around
2. They can see my local workspace, which is where most of my context lives
3. I can interact with their browser and applications without the lag of a virtual machine
Am I just being stubborn here?
it feels like cloud agent adoption is where claude code was at this time last year. that’s going to change over the holidays & it’s going to be insane
8
12
993
Moe Khalil retweeted
as much as i say "everyone is normal you can do anything" it's difficult to really operate that way
but it is true. we've gotten to the top in terms of inference and it's such a competitive well funded space
but it turns out everyone sucks! there is no magic everyone is just regular. we've caught up and surpassed everyone and that'll be clear once we get more hardware
and now we're looking at model training which seems even more like something regular mortals couldn't do
and we probably can do it
Introducing performance alerts in @LiteLLM
LiteLLM now flags spend spikes, traffic spikes, higher error rates, and slower responses across your teams, users, keys, and models
Read more below ↓
Moe Khalil retweeted
See how LiteLLM rewrote the core routing logic in Rust! This post is a technical deep dive that goes over the refactor before the rewrite and an in-depth explanation of how we are leveraging differential testing and agents to avoid regressions.
docs.litellm.ai/blog/router-…
My agent has its own issue tracker. It's made coding with agents so much more frictionless. I'm open sourcing the prompt that built it.
For context, I asked Codex to build something it could use to track every thread I started across chats, and add updates for progress and decisions made.
Then I asked it to give my Claude Code an onboarding prompt.
It's all stored locally, and it means every new chat always has the necessary context of how I got to the state I'm at.
At this point, I don't even check the UI - I just know it's using it and I never repeat context, even when switching agents.
It's pretty cool how well it works.
7
8
354
Been doing this thing lately where I give Codex and Claude Code the same instructions and have them work locally in isolation
When they're both done, I pull both implementations and see what implicit choices each one made
Then, I pick the one which is closer to my original vision and have it look at the other's implementation and explain
1. every decision it made that the other didn't
2. why it didn't make decisions the other did
I use that info to narrow down to a final proposal, which becomes the real PR
It burns a bit more tokens in the short term but has led to way more stable and intuitive architectural choices
4
8
478
I've completely changed my opinion on /greploop and having my agent "fix until CI is green"
It's one of the biggest slop amplifiers possible
I had Astra loop on a PR stack overnight assuming it would fix everything, and I'd wake up to PRs that were ready to merge in
Instead, I woke up to 20,000 lines of code that were a series of edge case -> patch -> new edge case -> patch -> red CI -> patch -> over and over again
The code was disgusting. I had to close all the PRs and start over
It just feels like when there's no human in the loop, the agent is incapable of looking at the big picture architecture, and gets so reactionary to every finding - even more so when you have multiple PR bots being triggered in parallel
Not to say that PR bots are useless, I actually think they're great. But I've found that I need to be in the loop every time they're triggered to have final say on what the next steps are
Is this just me?
Slack
talking to a coding agent is more like email than irc - because the interactions are few and far between, I often finding myself quoting their most recent text, answering multiple questions. addressing different topics. it's like replying to an email.
It makes me wonder if you could have a coding agent UI that was more like an email client or mailing list kind rather than chat
1
192
@LiteLLM now supports -latest model aliases.
Generally, every new model release in a family is smarter, faster, and cheaper than its predecessor.
With -latest, you pick a model family once, and LiteLLM handles updates to newer versions while you keep the same model name.
Some of the @LiteLLM team is in Singapore this week 🇸🇬
If you’re building with LiteLLM here, we’d love to meet and hear what you’re working on. Anyone around?
Fill out this form if you're interested: forms.gle/vsk28sak96DCPnVq7
49