IonelRoiban (@iroiban) header
IonelRoiban (@iroiban) profile picture
Palm Beach Gardens, FL
Joined May 2009
Photo by IonelRoiban (@iroiban)Photo by IonelRoiban (@iroiban)Photo by IonelRoiban (@iroiban)Photo by IonelRoiban (@iroiban)Photo by IonelRoiban (@iroiban)Photo by IonelRoiban (@iroiban)Photo by IonelRoiban (@iroiban)Photo by IonelRoiban (@iroiban)
IonelRoiban retweeted
Absolutely classic @GoogleResearch paper on In-Context-Learning by LLMs. Shows the mechanisms of how LLMs learn in context from examples in the prompt, can pick up new patterns while answering, yet their stored weights never change. 💡The mechanism they reveal for in-context-learning. When the model reads a few examples in your prompt, it figures out a pattern (like a small rule or function). Instead of permanently changing its stored weights, it forms a temporary adjustment that captures this pattern. That adjustment can be written mathematically as a rank-1 matrix, meaning it only adds one simple direction of change to the existing weights. This rank-1 update is “low-rank”, so it is very cheap and compact. But it still lets the model shift its behavior to fit the examples in the prompt. Once the prompt is gone, that temporary rank-1 tweak also disappears. So, in simple terms: The paper shows that in-context learning happens because the model internally applies a temporary rank-1 (very simple) weight update based on your examples, instead of permanently retraining itself. --- That behavior looks impossible if learning always means gradient descent. The authors ask whether the transformer’s own math hides an update inside the forward pass. They show, each prompt token writes a rank 1 tweak onto the first weight matrix during the forward pass, turning the context into a temporary patch that steers the model like a 1‑step finetune. Because that patch vanishes after the pass, the stored weights stay frozen, yet the model still adapts to the new pattern carried by the prompt. --- Shows that the attention part can take what it found in your prompt and package it into a tiny “instruction” that, for this 1 forward pass, acts exactly like a small temporary change to the MLP’s weights. Nothing is saved to disk, yet the block behaves as if the MLP just got a low-rank tweak computed from your examples. Remove the prompt, the tweak disappears, the saved weights stay the same. As the model reads your examples token by token, it keeps refining that temporary tweak. Each new token nudges the MLP a bit more toward the rule implied by your examples, similar to taking small gradient steps, again only for this pass. When the examples have done their job, those nudges shrink toward 0, which is what you want when the pattern has been “locked in” for the current answer. 🧵 Read on 👇
21
163
5
905
75,395
President Trump has approved our request to deploy the #USNSMercy to Los Angeles. Grateful for the quick response and partnership, @realDonaldTrump.
The USNS "Mercy" and USNS "Comfort" will be deployed to provide additional medical support in impacted areas.
341
643
108
3,667
Very good discussion with @realDonaldTrump. In response to the COVID-19 crisis, we are preparing with other countries a new strong initiative in the coming days.
631
554
177
2,980
IonelRoiban retweeted
Thank you Mr. President!
My friend (always there when I’ve needed him!), Senator @RandPaul, was just tested “positive” from the Chinese Virus. That is not good! He is strong and will get better. Just spoke to him and he was in good spirits.
2,441
2,919
251
24,263
....be put in use IMMEDIATELY. PEOPLE ARE DYING, MOVE FAST, and GOD BLESS EVERYONE! @US_FDA @SteveFDA @CDCgov @DHSgov
11,098
16,474
1,573
86,330
HYDROXYCHLOROQUINE & AZITHROMYCIN, taken together, have a real chance to be one of the biggest game changers in the history of medicine. The FDA has moved mountains - Thank You! Hopefully they will BOTH (H works better with A, International Journal of Antimicrobial Agents).....
49,394
69,440
24,751
277,582
Places where Joe Biden controlled or had influence over US policy: Ukraine China Romania Places where Hunter Biden did business with corrupt individuals: Ukraine China Romania It's almost like he followed his dad around the world peddling his last name for power & money 🤔
467
5,273
311
13,285
"The United States of America doesn't stand with murderous dictatorships, we stand up to murderous dictatorships" @VP on #NorthKorea #CPAC2018 #CPAC #ATimeForAction
2
2
7
IonelRoiban retweeted
Third burn successful. Exceeded Mars orbit and kept going to the Asteroid Belt.
4,830
26,348
2,779
141,714
IonelRoiban retweeted
Live Views of Starman pscp.tv/w/bUjQEDFyYVFab25sTU…
1,441
26,396
2,733
51,846
North Korean Leader Kim Jong Un just stated that the “Nuclear Button is on his desk at all times.” Will someone from his depleted and food starved regime please inform him that I too have a Nuclear Button, but it is a much bigger & more powerful one than his, and my Button works!
102,914
120,818
204,169
347,897
What a year it’s been, and we're just getting started. Together, we are MAKING AMERICA GREAT AGAIN! Happy New Year!!
16,140
17,613
2,245
85,577
One of the most powerful moments of my life watching @POTUS give the Purple Heart to this American Hero. Amazed by the strength and resilience of the men and women in our Armed Forces.
1,955
7,972
540
28,415
IonelRoiban retweeted
Today, @POTUS signed a directive calling for the United States to lead a human return to the Moon, followed by missions to Mars and beyond. Details: go.nasa.gov/2jOXj5g
1,438
2,886
641
11,021
Thank you to the hundreds of thousands of New Yorkers who cast their vote for me. It was my honor to fight this fight for you!
76
79
17
508
Crooked Hillary Clinton spent hundreds of millions of dollars more on Presidential Election than I did. Facebook was on her side, not mine!
28,066
11,578
4,836
60,200
The guests at my #TaxReform Mtg want to share with you their reason for supporting tax reform. #LeadRight #TeamFL #MoreKibbles🐕
President Trump briefed in Oval by @TomBossert45, tracking approach of #HurricaneIrma toward the coast of Florida w/ @VP & Sec. Tillerson.
411
1,103
122
3,882
Hackers testing the security of 30 voting machines at Defcon "We got in within 35 minutes" https://teapawt.net/t.co/Zc9cKBXpD4 #defcon #Defcon25 @defcon
84
1,961
251
1,609