19 | Building
Joined May 2024
- Tweets1.9K
- Following271
- Followers317
- Likes30.2K
Pinned Tweet
Introducing Particle 1.6. Same ~100M model trained from scratch, new SFT on the 1.0 pretrained base.
Particle talks but barely makes sense. In 1.0 it often ignored the question and spat out random Python code instead. In 1.6 I tried to walk that back. Some basic science trivia is a bit better. The problem is not fully solved tho.
I initially thought the 1.0 mess came from the SFT data we used. So this time I ditched the Hugging Face smol-smoltalk set altogether and built my own SFT mix (this overconfidence came to bite me in the ass soon). I pulled Q&A out of publicly available textbook PDFs and used an LLM API to generate the pairs, not just reformat the text. That was supposed to be version 1.5, a good step up from 1.0. It failed so badly that 1.5 was a worse model than 1.0.
Surprised (really?) I didn’t know what to do. I had just wasted a bunch of time extracting PDFs and generating datasets that were pretty much not making any meaningful improvement. So I mixed my textbook pairs with a code-filtered sample of smol-smoltalk and ran SFT again from the pretrained base. This time it worked fairly okay. The mix helped on some basic science trivia. It did not deliver the lift I wanted over 1.0.
Again as always the model is nowhere near a reliable assistant. Good learnings.
This can be a Netflix series, until next episode consider talking to the model.
If you have any thoughts, do reach out. Thank you.
At this point I'm just going to start giving my project's social media access to agent.
Let is post what it wants. We'll see what happens.
Having weights of arbitrated models is questionable but having an api for that???
That just means anyone with a few dollars can now write scripts which can potentially cause a lot of damage.
So I've worked on bots that are used for intent classification.
I've trained my own 15M decoder from scratch, I've used google dialogflow, I've seen many encoder implementations.
One thing I've noticed is that you do not need an llm for most usecases.
A bot that is build for customer service for a pizza shop does not need to know how do reverse a binary tree.
All it needs to know is how to track orders and process customer information accurately without messing up.
I ran an experiment in which I skipped the llm totally and just used embedding model as the entire bot.
Here's how it performed.
stealth-01 here being text-embedding-3-small lol
If you're using a llm api for customer support chatbot, chances are you can reduce the api costs by 99%.
Pratham. retweeted
IIT Madras Develops the World’s First Human Ischemic Stroke Atlas
During the Hon’ble Prime Minister Shri Narendra Modi’s visit to IIT Madras, Sudha Gopalakrishnan Brain Centre (SGBC) released C-STROKE, the world’s first human ischemic stroke atlas as an open global resource.
This pioneering resource bridges clinical MRI scans with microscopic, whole-brain imaging, offering insights into stroke-related changes at the cellular level. Featuring over 1,000 annotated brain sections, the atlas maps key cellular markers associated with neuronal health, neuroinflammation, blood-brain barrier disruption, hypoxia and axonal injury.
Developed with support from IIT Madras Distinguished Alumnus Mr V. Prem Watsa, the atlas provides an open-access platform for researchers and clinicians worldwide to investigate cerebrovascular diseases and explore potential therapeutic targets.
C-STROKE represents a significant milestone in advancing understanding of stroke - c-stroke.humanbrain.in
@iitmadras @kris_sg @BraincentreIITM @PMOIndia @narendramodi
#IITMadras #CSTROKE #StrokeResearch #Neuroscience #HealthcareInnovation
