no longer a good place to find me. check my website.
Bay Area
Joined September 2010
- Tweets1.7K
- Following271
- Followers1.8K
- Likes4.7K
Steve Yadlowsky retweeted
Worth clarifying that this is flatly false and the two options are just two independent ways to opt-out.
We respect our users' choice whether to use their data to “improve our models for everyone” regardless of where they express that choice. Users can opt out in the in-app settings or indeed also in our privacy portal. They do not need to opt out in both places, and we will make this clearer in our Help Center.
privacy.openai.com/policies/…
There is a super shady data control setting on everyone's ChatGPT which makes it seem like your data can be used for training even if you explicitly say to *NOT* improve the model for everyone (1/9)
Readers added context they thought people might want to know
This claim is false. OpenAI has clarified that the in-app toggle and the privacy portal are independent ways to opt out of data training. You only need to use one method, and OpenAI respects the opt-out choice regardless of where it is set.
x.com/thsottiaux/sta…
help.openai.com/articles/77308…
Going beyond needle tasks is super important for understanding real-world long-context capabilities. Looks like a great benchmark for measuring progress at the intersection of long context and reasoning.
Happy to share Michelangelo (arxiv.org/abs/2409.12640), a long-context reasoning benchmark which measures performance beyond needle tasks up to arbitrary context lengths and remains challenging for frontier models. Stay tuned for more Michelangelo evals to come!
Excited to be at NeurIPS this year, anyone who is around and wants to chat, please reach out! It’s always great to connect/reconnect with folks at these events!
Excited to be part of this effort! It's been a fun research experience to be working on something so new.
I’m very excited to share our work on Gemini today! Gemini is a family of multimodal models that demonstrate really strong capabilities across the image, audio, video, and text domains. Our most-capable model, Gemini Ultra, advances the state of the art in 30 of 32 benchmarks, including 10 of 12 popular text and reasoning benchmarks, 9 of 9 image understanding benchmarks, 6 of 6 video understanding benchmarks, and 5 of 5 speech recognition and speech translation benchmarks. Gemini Ultra is the first model to achieve human-expert performance on MMLU across 57 subjects with a score above 90%. It also achieves a new state-of-the-art score of 62.4% on the new MMMU multimodal reasoning benchmark, outperforming the previous best model by more than 5 percentage points.
Gemini was built by an awesome team of people from @GoogleDeepMind, @GoogleResearch, and elsewhere at @Google, and is one of the largest science and engineering efforts we’ve ever undertaken. As one of the two overall technical leads of the Gemini effort, along with my colleague @OriolVinyalsML, I am incredibly proud of the whole team, and we’re so excited to be sharing our work with you today!
There’s quite a lot of different material about Gemini available, starting with:
Main blog post: blog.google/technology/ai/go…
60-page technical report authored by th Gemini Team: deepmind.google/gemini/gemin…
In this thread, I’ll walk you through some of the highlights.
When they see examples from a task unlike any task family they’ve seen before, one might have many hypotheses about how they generalize or interpolate, such as finding the nearest approximation to a known task family. We designed empirical studies to test many of them. 3/
What we found is that in some limited cases (rare tasks or those <close> to a /seen task family), the model makes reasonable predictions. Other times, the model tends to behave erratically. 4/
Might be helpful to do a thread about what our paper shows and why we think it’s interesting. First of all, we study transformer models pretrained to be in-context learners 1/
Excited about this work and the enthusiasm it has received. To clarify, we show that transformers <can> learn new tasks, but <can’t> generalize to unseen task families. Figuring out what task families a fully-trained LLM can ICL is an impactful scientific direction from here.
Steve Yadlowsky retweeted
According to the first sentence of 100% of the papers I have reviewed this year, large language models have achieved amazing success in recent years. Perhaps we could settle on the abbreviation "Language Models Are Outstanding", so papers could begin "LMAO, but …" to save space.
Steve Yadlowsky retweeted
It's graduation season so time to remind everyone that the graduation robe is descended from a medieval snuggie
Steve Yadlowsky retweeted
(6/n) Joint work with @ulloa_ernesto, @mcaronebstat, and @aluedtke. I also highly recommend checking out @graduatedescent 's and coauthors work on estimating calibration error for HTEs: proceedings.mlr.press/v151/x…
Steve Yadlowsky retweeted
New paper!!! Causal Isotonic Calibration for HTEs! We propose new nonparametric methods for calibrating predictors of CATEs. Our methods can be wrapped around any black-box learning algorithm to provide strong distribution-free calibration guarantees. arxiv.org/abs/2302.14011
Steve Yadlowsky retweeted
I would say that it can reproduce learned causal links and hypothesize new links. That is a kind of inference. However, it has no way to confirm whether those hypothesized links are real. These models generate, but they don't test.
