Machine learning at @duolingo. Multilingual structured prediction and language modeling.
Joined July 2018
- Tweets31
- Following63
- Followers206
- Likes336
Phoebe Mulcaire retweeted
Duolingo AI is hiring new PhDs and PhD interns! Come define the future of AI-driven education!
(New PhDs must graduate by Aug 2025, interns by Aug 2026)
jobs.duolingo.com/
"you must make reasonable efforts to use the latest version" in the DBRX and Gemma licenses makes it feel so dishonest to call them "open models." like, yes, you can see the weights, but they can just take away all your usage rights by publishing a broken "update"
my influence is subtle, but inescapable
xitter.cf/MeganRisdal/status/177…
Not a joke. We've updated Gemma Terms. ai.google.dev/gemma/terms
ALT Google may update Gemma from time to time, and you must make reasonable efforts to use the latest version of Gemma. with ", and you must make reasonable efforts to use the latest version of Gemma" crossed out in red
Phoebe Mulcaire retweeted
Wanna know gpt-3.5-turbo's embed size? We find a way to extract info from LLM APIs and estimate gpt-3.5-turbo’s embed size to be 4096. With the same trick we also develop 25x faster logprob extraction, audits for LLM APIs, and more!
📄 arxiv.org/abs/2403.09539
Here’s how 1/🧵
Phoebe Mulcaire retweeted
LMs are increasingly large🐘 and proprietary🔒 — what if we could “tune”🔧 them without accessing their internal weights?
Enter: proxy-tuning, which operates on only the *outputs* of LMs at decoding-time to achieve the effect of direct tuning!
📄: arxiv.org/abs/2401.08565 1/
Phoebe Mulcaire retweeted
🚨 New Dataset Alert 🚨 I'm extremely excited to announce Universal NER v1, available now.
It is gold-standard human annotations of 18 datasets covering 12 languages, based on Universal Dependencies texts. This is the first data release of the UNER project.
1/3
ALT Screenshot of the Universal NER v1 paper.
come see our poster, "Automated Evaluation of Written Discourse Coherence Using GPT-4", at the BEA workshop, today at 2:30! #ACL2023NLP
at #ACL2023NLP ! I'll be presenting a poster at BEA on Thursday: work with @BenNaismithELT and @JillBurstein on discourse grading with GPT-4. come say hi :)
Phoebe Mulcaire retweeted
⚡️New paper!⚡️
It’s tempting to interpret chain-of-thought explanations as the LLM's process for solving a task. In this new work, we show that CoT explanations can systematically misrepresent the true reason for model predictions.
arxiv.org/abs/2305.04388
🧵
I'll be in Seattle from this Wednesday to next Tuesday—let me know if you'd like to meet up and say hi!
Phoebe Mulcaire retweeted
🌟Update🌟there's been a lot of debate of whether Theory of Mind (ToM) has emerged in new models (ChatGPT/GPT-3.5/GPT4), as people have reported qualitative/anecdotal evidence of good performance on these types of examples.
TLDR; still no neural ToM... 🧵
xitter.cf/MaartenSap/status/1587…
Phoebe Mulcaire retweeted
wait,
what??
why do Bard AND ChatGPT *both* write an anodyne story about a young woman in idyllic "Willow Creek" at sundown???
(details: it's not deterministic, often you get a different town name, different phrasing, etc. broad strokes are similar though. gpt-4 does it too.)
Phoebe Mulcaire retweeted
arxiv.org/abs/2210.15097
We propose contrastive decoding (CD), a more reliable search objective for text generation by contrasting LMs of different sizes. CD takes a large LM (expert LM e.g. OPT-13b) and a small LM (amateur LM e.g. OPT-125m) and maximizes their logprob difference
Phoebe Mulcaire retweeted
In addition to proposing an interesting new truncation sampler, this paper draws a connection between truncation and n-gram smoothing that refined my understanding of the motivations for truncation sampling in general. This is a great read!
We characterize and improve on language model _truncation sampling_ algorithms, like top-p and top-k. We frame them as trying to recover the true distribution support from an implicitly _smoothed_ neural LM, and provide a better sampling algo!
Paper arxiv.org/pdf/2210.15191.pdf
"improves over BERT on unseen scripts" struck me as really silly but it makes sense! fascinating work
Tired of tokenizers/subwords? Check out PIXEL, a new language model that processes written text as images📸
“Language Modelling with Pixels”
📄 arxiv.org/abs/2207.06991
🧑💻github.com/xplip/pixel
🤖huggingface.co/Team-PIXEL/pi…
by @rust_phillip @jonasflotz me @esalesk @mdlhx @delliott
I passed my defense! looking forward to starting at @duolingo this summer, working on ML for language learning.
Phoebe Mulcaire retweeted
📢 Welcome to ar5iv.org
Change the "X" in any arXiv article link to the "5" in ar5iv to get a modern HTML5 document.
Thread: what is included, why now, and how we hope to merge back into arXiv.
#OA #OpenScience #preprints
1/10
Phoebe Mulcaire retweeted
We introduce 𝐁𝐢𝐝𝐢𝐦𝐞𝐧𝐬𝐢𝐨𝐧𝐚𝐥 𝐋𝐞𝐚𝐝𝐞𝐫𝐛𝐨𝐚𝐫𝐝𝐬 (𝐁𝐢𝐥𝐥𝐛𝐨𝐚𝐫𝐝𝐬), a generalization of leaderboards that simultaneously drives progress in natural language generation and its evaluation.
arxiv.org/abs/2112.04139
nlp.cs.washington.edu/billbo…
1/5
ALT Billboard comic.
Phoebe Mulcaire retweeted
How can we make language models be less data-hungry?
We show that fully compositional word representations grounded on related words and definitions from an external lexicon can do that.
Joint work w/ @PhoebeNLP,@nlpnoah at @emnlp2020
arxiv.org/abs/2009.11523 #NLProc (1/3)
