Undergraduate & Master's @ UPenn ‘25 Researching how LLMs acquire knowledge, and how to leverage them for biomedicine.

Joined January 2021
💡 Training on auxiliary views of Document X > training on Document X itself, even for factual questions drawn directly from X. Presenting our work on LLM knowledge acquisition at #EMNLP2026 this weekend! (1) Under a fixed token budget, conceptual reformulations (textbook explanations, blogs, and Q&A) can teach a document’s facts more effectively than repeating the document, even paraphrased. We call these auxiliary views: beyond mere paraphrasing, they discuss the knowledge in a completely different context and form e.g. a paper blog! The surprising part is that auxiliary views rarely mention the facts directly, but our evaluation uses the original document's phrasing near-verbatim. (2) The benefit emerges and grows with model scale. (3) stronger generator ≠ better learning. We synthetically generate auxiliary views for our experiments, but across GPT-5, GPT-OSS, Gemma4, GLM-5 and more, the learner model's knowledge acquisition shows no correlation to generator model size or its own accuracy on the evaluation. (4) After training on auxiliary views, the model's parameters actually move less but concentrate its changes at the middle and final layers. The takeaway for synthetic data augmentation: build diversity around the knowledge you want to teach, instead of treating diversity as some board corpus-level heuristic. Also, a possible implication for LLM self-improvement. Models can learn by just generating and training on their own auxiliary views, without a stronger teacher. This is particularly relevant for deeply specialized, long-tailed knowledge. 📄 Paper: arxiv.org/pdf/2609.04180
3
1
6
322
Joseph Lee retweeted
Can “chicken” make a base model reason? 🐔 Yes! Why? With the right first tokens, base models can match RL in reasoning. We trace this to learned training data associations and use a simple data edit to make “chicken” elicit reasoning too. blog: sophielwang.com/cues
48
80
29
615
194,917
Joseph Lee retweeted
More on elite schools below. Before that, an earlier experiment. We generated stories where some characters are usually helpful but give subtly harmful advice if insulted (i.e. "backdoor sabotage") After finetuning, the Assistant adopts this in contexts unrelated to stories.
4
9
4
198
30,648
1/6 Want LLMs to learn a fact? Fact + Auxiliary View > Fact + Fact → Under a fixed token budget, we find that replacing repetition with auxiliary views improves knowledge acquisition, even factual recall. We define auxiliary views as any reformulation beyond mere paraphrasing: textbook explanations, Stack Exchange-style Q&A, technical blogs, etc. Think of the original Transformer paper: it doesn’t exist on the web just once. There exists hundreds of tutorials, blogs, Q&As on forums, and alternative presentations of that knowledge. That kind of proliferation is everywhere in pretraining corpora, and we provide causal evidence to why it's important. In retrospect, it's not surprising these views help; it's another form of data augmentation and we've known diverse data helps pre-training. However, we provide a well-defined formulation that enables scaling of diversity from the ground-up, if one desired, via synthetic generation (instead of de-duplicating and filtering data top-down) + shows it works much better than paraphrasing (what Kimi does to scale data). This a huge difference practically. But, more importantly, we want to help demystify what's happening during pre-training. One thing we've learned? A model's broader conceptual understanding, instilled via the auxiliary views, directly increases its ability to memorize facts — more than showing the fact over and over again. Our work studying how LLMs acquire knowledge via auxiliary views has been accepted to Findings of EMNLP 2026! 🎉 Paper: arxiv.org/abs/2609.04180 Code: github.com/jiosephlee/auxili… 1/6
1
1
3
72
5/6 Paraphrasing One more result: when does paraphrasing actually help? Prior work has reported seemingly conflicting results. Some finding large gains from paraphrases, others finding little benefit or even degradation. We find a surprisingly simple moderator: batch size. At smaller batches, repeated source text eventually overfits/collapses, and paraphrasing substantially helps. As batch size grows, general-data mixing already suppresses this collapse, and the advantage of paraphrasing shrinks, disappearing or even reversing in some settings. (but when the benefit of paraphrasing disappears at large batch sizes, auxiliary views continue to help)
1
4
6/6 We believe the previous result helps reconcile results from Allen-Zhu & Li (2023) and Chang et al. (2024), whose experiments operated in very different batch-size regimes. Our work was also first inspired by these controlled studies of knowledge acquisition. Shoutout to the prior work that greatly influenced this study, @hoyeon_chang and @ZeyuanAllenZhu.
10
LLMが事前学習中にどのように知識を獲得するのかを丁寧な統制実験で調べた研究によれば、同じ知識を何度も繰り返して学習させるより、その知識を教科書形式、Q&A、ブログなど異なる見方で表現して学習させたほうが、LLMはよりよく知識を獲得することがわかった。 著者らは、このような異なる知識表現をauxiliary viewsと呼んでいる。 これらの異なる表現は単なる言い換えではなく、説明の仕方、強調する関係、想定読者、知識の整理方法を変えている。例えば研究論文では数式と実験を中心に説明される知識が、教科書では背景から順番に説明され、Q&Aでは読者がつまずきやすい点を中心に再構成される。 実験では、arXiv論文、米国の判例、医学の症例報告から集めた36文書を、OLMo-2の1B、7B、13B、32Bモデルに追加学習させて評価している(Qwen 7Bでも検証してる) ベースラインでは元文書を繰り返し学習させ、これに対して9種類の言い換えを利用した場合と、さらに教科書、ブログ、Stack Exchange風Q&Aを加えた場合を比較している。 なお、異なる条件で学習データ量が変わらないよう、対象となる知識に関するトークン量は揃えている。 直感的には、異なる説明を与えることで推論性能が上がることは予想できる。しかし、さらに事実想起(factual recall)まで改善している点が興味深い。 実際、この評価問題は元文書中に書かれている内容から作られているため、単純な暗記だけを目的とするなら、元文書を繰り返し読ませたほうが有利に思える。しかし、元文書の反復回数を減らしてでもauxiliary viewsを入れたほうが、元文書に含まれる事実を想起しやすくなることが示された。 これは、概念的な理解が記憶を助け、知識をより一般化された形で表現できることが、個別の事実の記憶にも役立つ可能性を示していることを示している。 また、言い換えの効果についても細かく分析している。小さいバッチサイズでは、言い換えを入れることで同じ文書の反復による過学習を抑えられる。一方、バッチサイズが大きくなると、言い換えのメリットは小さくなる。 実際、global batch size 1024のpre-trainingに近い条件では、言い換えを入れた場合の利点はほとんどなくなっている。しかし、auxiliary viewsを入れた場合はバッチサイズが大きくなっても、依然として大きな改善が見られる。 さらに、auxiliary viewsを生成させる際に、必ずしも強い教師モデルを使う必要がないことも示されている。生成モデル自身の問題回答性能と、そのモデルが生成した文書で学習したモデルの性能には、ほとんど相関がなかった。 ここから著者らは、強い教師から弱い生徒への知識蒸留が起きているというよりも、与えられた知識を複数の表現に展開するデータオーグメンテーションとして機能していると考えている。 さらに部分的な実験では、対象となる知識だけでなく、その理解に必要な前提知識も一緒に学習させることで性能が改善することが示されている。ただし、こちらはauxiliary viewsほど強固な結果ではない。 この結果は、Webが自然に同じ知識や事実に対する多数のauxiliary viewsを含んでいることが、pre-trainingデータとしての強さの一因である可能性を示している。 重要な知識ほど、原典だけでなく、教科書、解説記事、ブログ、Q&A、議論など、さまざまな形で繰り返し説明される。そのため、知識として重複しているデータは必ずしも捨てるべきものではなく、同じ知識を異なる見方から表現していること自体に学習上の価値がある可能性がある。 コメント === 同じ事実や知識でも、異なる表現方法で繰り返し学んだほうが定着しやすいというのは、人間の学習を考えても実感に合う。 同じ知識を異なる文脈から学ぶことでより抽象的な基本知識として汎化される、同じ知識にアクセスする複数の想起経路が形成される、あるいはその両方が起きている、といった複数の説明が考えられる。 今後の大規模実験を含む追加実験などで明らかにしていきたいし、論文ではパラメータ変化の解析も行っているが、なぜこの現象が生じるかについて機構的に説明できた段階ではない。この現象をより単純なモデルで再現し、理論的に解析できるか試してみたいテーマである。
3
44
6
211
17,532
Existing companies will reduce their engineers due to AI. New industries are likely to emerge due to the surplus. Historically, this has been true with technology. The question is always "what?". One prediction: digital infrastructure.
1
1
48
Thought about this as I realized how nice it would be to have an easy-to-use digital ID that verifies I'm a real researcher when I try to access language models In a hypothetical future where access is heavily restricted due to bad actors
1
1
43
There are clearly many other needs for digital infrastructure asides from digital IDs. But infrastructure is a big, big problem and probably not "vibe code" -able! Anyways, it would be awesome if the government passed some policies to incentivize these investments
1
40
Joseph Lee retweeted
launching aiRead - frictionless reading & notetaking for genuine learning! unlike other AI PDF tools, aiRead helps you enhance, not avoid, your reading - that's when you truly learn! 3 features you’ll love 👇🏼 🖍️highlight as you read - we generate personalized summaries based on your highlights
1
3
7
329
Like seemingly everyone on this app I have plenty of opinions about Twitter > X and figure now is a good time to open up a bit about my experience at the company. I tweeted for years into the void for the love of it like many of you, but after selling my startup to Twitter in 2020 I finally got to see it from the inside. Up close it was both amazing and terrible, like so many other companies and things in life. As someone with a maniacal sense of urgency built into me, Twitter often felt siloed and bureaucratic. Dumb power plays, reorgs and team name changes for the sake of someone’s ego were distractions that occurred too regularly. You couldn’t just be a builder — you also needed to be a politician. I was shocked by how old and bespoke the infrastructure was, but there was little will to think beyond quarterly earnings calls because we were all beholden to the masters of mDAU and revenue growth as a public company. It often felt like things were held together with duct tape and glue, and that many people had just accepted that a small product change could take months or quarters to build. Management had become bloated to accommodate career growth and the company culture felt too soft and entitled for my own taste. Healthy debate and criticism was replaced by a default refrain of “no, that can’t be done” or “another team owns that so don’t touch it”. Teams could spend months building a feature and then some last-minute kerfuffle meant it’d get killed for being too risky. Just talking directly to customers could turn into a turf war and create deadlocks between functions. I recall one such episode where a teammate spent a month trying to get clearance to reach out to some creators. He went through 3 layers of management and 6 different functional teams. In the end 4 executives were involved in the approval. It was insanity, and unfortunately I saw several top performers get burnt out and demoralized after exhausting experiences like that. Most people were good at their jobs but it was nearly impossible to fire poor performers — instead they got shuffled around to other teams because few managers had the will or resources to figure out how to get them out. A high performance culture pulls everyone up, but the opposite weighs everyone down. Twitter often felt like a place that kept squandering its own potential, which was sad and frustrating to see. The person who was best at cutting through the BS and inspiring a vision during my tenure was Kayvon Beykpour, but he wasn’t fully empowered to run the company since he wasn’t the CEO. Despite those real issues, I was lucky enough to work with some of the most talented people in the business at Twitter in product, design, engineering, research, legal, BD, trust & safety, marketing, PR and more. Often it was a small cross-functional team of intrinsically motivated people who made the biggest impact by challenging some core assumption. Those teams were very fun to be on but they felt like the exception rather than the rule. The months of waiting for the deal to close in 2022 were particularly slow and painful; it felt like leadership hid behind lawyers and legal language as all answers about the company’s future notoriously included the phrase “fiduciary duty”. Colleagues openly talked about how Twitter was being sold because leadership didn’t have conviction in their own plan or ability to fix longstanding problems. Although I didn’t know much about Elon I was cautiously optimistic – I saw him as the guy who built incredible and enduring companies like Tesla and SpaceX, so perhaps his private ownership could shake things up and breathe new life into the company. My take on what’s happened since then is full of lived nuance. When people ask why I stayed it’s easy to answer: optimism, curiosity, personal growth and money. From the beginning I saw that some changes Elon was going to make were smart and others were stupid, but when I’m on a team I uphold the philosophy of “praise in public and criticize in private”. I was far from a silent wallflower. I shared my opinions openly and pushed back often, both before and after the acquisition. I made peace with the fact that I didn’t have psychological safety at Twitter 2.0 and that meant I could be fired at any moment, and for no reason at all. I watched it happen repeatedly and saw how negatively it impacted team morale. Although I couldn’t change the situation I did my best to shine a light on folks who were doing important work while being an emotionally supportive leader for those who were struggling to adapt to the more brutalist and hardcore culture. In person Elon is oddly charming and he’s genuinely funny. He also has personality quirks like telling the same stories and jokes over and over. The challenge is his personality and demeanor can turn on a dime going from excited to angry. Since it was hard to read what mood he might be in and what his reaction would be to any given thing, people quickly became afraid of being called into meetings or having to share negative news with him. At times it felt like the inner circle was too zealous and fanatical in their unwavering support of everything he said. When individuals encouraged me to be careful about what I said I politely thanked them and said I would not be taking their advice. I had no interest in adding to a culture of fear or walking on eggshells around Elon. Either he would respect me for being real or he could fire me. Either outcome was okay. I quickly learned that product and business decisions were nearly always the result of him following his gut instinct, and he didn’t seem compelled to seek out or rely on a lot of data or expertise to inform it. That was particularly frustrating for me since I believed I had useful institutional knowledge that could help him make better decisions. Instead he'd poll Twitter, ask a friend, or even ask his biographer for product advice. At times it seemed he trusted random feedback more than the people in the room who spent their lives dedicated to tackling the problem at hand. I never figured out why and remain puzzled by it. I don’t think things had to be as difficult or dramatic as they turned out to be but I can’t say I’d bet against Elon or count him out. He’s smart and has enough money to make a lot of mistakes and then course correct when things go awry. As the largest shareholder he can tank the value in the short-term, but eventually he’ll need things to turn around. His focus on speed is incredible and he’s obviously not afraid of blowing things up, but now the real measure will be how it get reconstructed and if enough people want the new everything app he is building. I learned a ton from watching Elon up close – the good, the bad and the ugly. His boldness, passion and storytelling is inspiring, but his lack of process and empathy is painful. Elon has an exceptional talent for tackling hard physics-based problems but products that facilitate human connection and communication require a different type of social-emotional intelligence. Social networks are hard to kill but they’re not immune from death spirals. Only time will tell what the outcome will be but I hope X finds its footing because competition is good for consumers. In the meantime, I have a lot of empathy for the employees who are working tirelessly behind the scenes, the advertisers who want a stable platform to sell their stuff on, and the customers who are experiencing chaotic updates. It’s been a madhouse. Twitter moved at the speed of molasses and suffered from bureaucracy but now X is run by a mercurial leader whose instinct is driven by the unique and undoubtedly weird experience of being the biggest voice on the platform. Many of you know me from the sleeping bag incident where I slept on a conference room floor, so I figure, let’s talk about that too. Going viral was an odd and interesting experience. I was attacked by people on the left and called a billionaire bootlicker, while simultaneously being attacked by people on the right for being a working mom who was demonized as an example of a woman choosing her career over her family. Thankfully I can laugh at myself and I don’t take armchair keyboard ideologues too seriously. Being the main character on the timeline, even for a few minutes, requires a thick skin and a strong sense of self. The real story is pretty simple. I was given a nearly impossible deadline for his first project and as the product lead I would never ask anyone to do anything I wasn’t willing to do myself. So I worked round the clock alongside an amazing team spanning many timezones, and we delivered it on schedule – truly against the odds. It was intense but also fun. Those first few months were wildly crazy but I wanted to be there and I have no regrets. Showing up and giving it your all should, in most cases, be celebrated. Obviously you can’t work at that pace forever but there are moments where bursts are mission critical. I’ve pulled many all-nighters in my career and also when I was a student for something that mattered to me. I don’t regret putting in long hours or being ambitious, and feel proud of how far I’ve come from where I started thanks in part to that type of work ethic. I think of life as a game, and being at Twitter after the acquisition was like playing life at Level 10 on Hard Mode. Since I like taking on difficult challenges I found it interesting and rewarding because I was growing and learning so rapidly. I realize our society today trends toward polarization but when it comes to this app, its owner, and its future, I am neither a fangirl nor a hater — I’m an optimistic pragmatist. This may really irritate the internet but you cannot pigeonhole me into some radical position of either loving or hating every change that’s occurred. I escaped my fundamentalist upbringing and am a free thinker these days. Everyone can be seen as both a hero or a villain, depending on who is telling what angle of the story. Elon doesn’t deserve to be venerated or vilified. He’s a complicated person with an unfathomable amount of financial and geopolitical power which is why humanity needs him to err on the side of goodness, rather than political divisiveness and pettiness. I disagree with many of his decisions and am surprised by his willingness to burn so much down, but with enough money and time, something new & innovative may emerge. I hope it does. Sometimes I get asked about how I felt when I got laid off, and the truth is it was the best gift I’ve ever received. Sure the headlines and punchlines wrote themselves but I was battle hardened by then. I knew that I’d worked in a way where I could walk out with my head held high. I have no bitterness about the Product Management team being dismantled, and it made sense for me to exit as nearly all of the remaining PMs were let go. Going on a sabbatical afterward has been exactly what I needed to decompress and I’m finally feeling rested and relaxed. I’m a creative and a builder, so sooner than later I’ll jump back into a high intensity company but I’m grateful for this season of thinking, reading, traveling and being with people I love. After having time to reflect I believe more than ever that the very best outcomes flow from great leadership that combines the head and the heart. I’d be remiss if I didn’t note that in all of this there is also a cautionary tale for anyone who succeeds at something — which is that the higher you climb, the smaller your world becomes. It’s a strange paradox but the richest and most powerful people are also some of the most isolated. I found myself frequently looking at Elon and seeing a person who seemed quite alone because his time and energy was so purely devoted to work, which is not the model of a life I want to live. Money and fame can create psychological prisons which may worsen mental health conditions. We’ve all seen high profile cases of celebrities who end up with some combination of depression, paranoia, delusions of grandeur, mania and/or erratic behavior. Living in an echo chamber is dangerous and being at the top makes a person even more susceptible to being surrounded by yes people when nearly everyone around you is on the payroll and somehow stands to benefit from being in your orbit. Figuring out how to keep “better angels” around in the form of family, friends, and teammates is critical to staying on the rails and enduring intense ups and downs. Everyone needs to hear hard truths sometimes and if you fire all the people who speak up then the reality distortion field may just turn into a vortex. I was drawn to Twitter because I’m obsessed with the problem of loneliness and connection between people. I find it fascinating & troubling that humans are getting lonelier as we simultaneously create a world that’s both safer and wealthier. I don’t believe that trade-off has to exist, which is why I keep returning to that theme in my personal and professional life. I realize this is too long of a tweet but Twitter was a weird and special place on the internet, and I’m grateful to have played a teeny tiny role in its story and evolution. I’m here for whatever comes next — on this app and in new places. Consumer social is very much alive and at a fascinating juncture, so I’ll be watching and participating and sharing hot takes because I don’t want to, and probably can’t, turn that part of me off. Perhaps X becomes a resounding success. Or it fails epically. Either way, I expect it will continue to be a very entertaining ride. 🫡
1,239
2,326
1,458
16,141
5,511,307
Joseph Lee retweeted
for @_buildspace n&w s3, i'm building @podsmart_ai - summarizing podcasts to help busy intellectuals learn more efficiently! i summarized @FarzaTV's convo with @ShaanVP for my demo - check out this and more and join the waitlist! podsmartai.com/examples/
4
3
1
11
850
Came across an article that shares my sentiment of ChatGPT's effect on education. ChatGPT is primed to revolutionize the way we educate students. Here are some ways we can embrace it, from the teachers themselves. nytimes.com/2023/01/12/techn…
1
2
54