Next generation tools for LLM developers.
Joined November 2023
- Tweets63
- Following16
- Followers27
- Likes6
Libretto retweeted
great q: how do you write evals w/o user queries? IMO evals aren't fixed; they constantly change over time, so just make some guess upfront. writing LLM apps with evals from the start means less overhead to change them. too many people struggle to add evals post-deployment
this is nice but not possible for all apps upfront. how do you first assemble an evals set for a chat with PDF app ? given that you do not know in advance which docs users will upload or which questions they will ask ?
lycee.ai/blog/ai-reliability…
Libretto retweeted
Models will get better/cheaper, and swapping could be as simple as updating the model id.
Thus, focus on the durable components of your system, such as evals, guardrails, data flywheel for finetuning, serving infra if self-hosting, etc.
applied-llms.org/#the-model-…
Claude 3.5 Sonnet is now available to @AnthropicAI devs everywhere.
It's our best model yet - smarter than Claude 3 Opus and twice as fast.
And it costs just $3 per million input tokens and $15 per million output tokens.
Libretto retweeted
2/ The basic reason is that every vendor uses a bespoke prompt engineering technique, and prompt engineering can have a massive effect on prompt performance (something we know about here at @getlibretto), often bigger than the difference in performance between models.