
Product @LiteLLM | Ex-Founder @WebhoundAI (YC S23) | Stanford CS ‘23 | San Francisco & Beirut
San Francisco, CA
Joined September 2011
- Tweets893
- Following229
- Followers363
- Likes26.6K
Moe Khalil retweeted
We’re launching Inference Stack, one-click deploy with everything you need to own your inference
It ships with:
@vllm_project for deploying models
@huggingface for finding models
@_llm_d_ for deployment-level routing
@LiteLLM for inference
@grafana for metrics
Try it out here: inferencestack.io
We’re launching Inference Stack, one-click deploy with everything you need to own your inference
It ships with:
@vllm_project for deploying models
@huggingface for finding models
@_llm_d_ for deployment-level routing
@LiteLLM for inference
@grafana for metrics
Try it out here: inferencestack.io
Question to CISOs: How do you deal with agent sprawl in your company? Do you have any visibility into what your employees' bots are doing?
Besides saving 5 minutes of setup time, why would you send your requests to anyone who isn’t the inference provider?
3
278
Self host your own gateway and use providers you trust. You should know where your requests are going.
the OpenRouter situation for anyone curious:
-someone managed to access outputs sent to MiniMax
-researcher discovered this by sending a secret links
-40 to 130 hours later someone opened the secret links
-oop, this means that someone accessed your outputs!!
-AltasCloud/Novita/StreamLake were forwarding requests to MiniMax???? what the helly?
ugh.
Does AI usage impact your company’s top line? If not, how do you justify spending so much on it?
1
3
186
About to present on our new @LiteLLM customer management portal in 5 minutes, hop on and ask questions
crowdcast.io/c/w9airc43f37a
5
227
If you use a tracing product, why? Isn’t the ideal experience here OTEL in Clickhouse with an MCP?
Then you can have your agent run automations and maybe build a custom UI on top of it
What am I missing here?
3
9
497
Moe Khalil retweeted
you'll never hear about APIs without an API Gateway. if you're building AI products without an AI gateway, why?
As a developer, how do you define "improving your agent"?
Presumably at some point your agent harness has all the capabilities it needs to accomplish its goal.
What comes next?
Can it file PRs?
JUST IN: Apple $AAPL is developing AI agent to rival OpenAI and Anthropic that can control devices, browse the web, and write code.
3
106
LiteLLM has evolved a lot since its first commit, and nowhere is that more evident than in its database
Since joining LiteLLM, I've come across so many different tables and columns that didn't make sense at first. When I dug deeper it became clear they were crucial to making the product work.
So, I built the LiteLLM database course. It goes through every table we have, what purpose it serves, why it was added (according to PR bodies), and the overall evolution of our DB and why it is the way it is today.
Check it out ↓
How do you handle it when each one of your users wants a different set of abstractions?
We want to support different versions of the same use case but keep the product simple
Do you just build it all but keep the happy path simple?
3
7
388
So we have evals for evals now, but how are we supposed to eval those evals?
Introducing the Eval Optimizer.
Not every eval needs a frontier model. Judgment now learns your agent trajectories and picks the right compute per judge: code, a decision model, an LLM or an agent.
For one customer, it cut the cost per eval by up to 50×.
2
175










