TK • 木下 (@wordsofteekay) header
TK • 木下 (@wordsofteekay) profile picture

coding . ml . math . writing . books . https://teapawt.net/t.co/SJXxlX5JeT

🇯🇵⇄🇧🇷
Joined December 2009
✨ The Mamba Mentality: How I Play — Full Review: goodreads.com/review/show/37…
68
✨ Hands-On Machine Learning with Scikit-Learn and TensorFlow — Full Review: goodreads.com/review/show/65…
2
76
2026 has been the year of biographies and technical books so far. In the last couple of years, I've learned to read fewer books, choose better, and focus on the signals (still wip). Here are the ones I read so far: goodreads.com/readingchallen… > AI and ML for Coders in PyTorch: A Coder's Guide to Generative AI and Machine Learning > Designing Machine Learning Systems: An Iterative Process for Production-Ready Applications > Deep Learning for Biology: Harness AI to Solve Real-World Biology Problems > Decoding Greatness: How the Best in the World Reverse Engineer Success > Nós: O Atlântico em solitário > Practical Statistics for Data Scientists: 50 Essential Concepts > The 5 Types of Wealth: A Transformative Guide to Design Your Dream Life > Machine Learning System Design: With end-to-end examples > Build a Large Language Model > The Art of Doing Science and Engineering: Learning to Learn > AI Engineering: Building Applications with Foundation Models --- I'm currently reading The Infinity Machine (DeepMind) and The Idea Factory (Bell Labs). I have just finished AI Engineering and am looking for a new technical book.
📝 I hope with this new post, you can steal some ideas, and insights, and put them into practice in your life. This is my reflection about reading 47 books in the first 6 months of 2023 and how I am focusing on reading less + applying them in my life. iamtk.co/i-read-47-books-in-…
2
9
132
5,484
✨ AI Engineering — Full Review: goodreads.com/review/show/71…
1
1
98
Manufacturing Luck
3
47
1,262
One of the best articles I've read these days. A very practical guide for ML research. I should re-read it every now and then to improve my workflow. → joschu.net/blog/opinionated-…
1
10
70
2,263
One type of freedom worth striving for: Be so good people can no longer dictate your terms.
1
5
1
68
1,985
✨ The Book of Elon: A Guide to Purpose and Success — full review: goodreads.com/review/show/84…
1
1
1
273
This resonates a lot with my experience. My record was 60 books a year (not 80 in 6 months tho). Because I'm curious about a lot of things, many topics get my attention, so the "Parallelize" (books) tip is a really effective way to read more book. I read 3-4 at the same time, a bit every day, consistently. It turns out it is much easier to do, and in the long-term, I accomplish more. Reading a lot also made me rethink about which books I choose to read (reading less → reading better books: iamtk.co/i-read-47-books-in-…). And because I usually read technical and non-fiction books, it's great to re-read them, take notes, and think in way to apply the ideas in my life (iamtk.co/essays/applying-boo…). "How To Read More" by Borretti: borretti.me/article/how-to-r…
6
89
4
823
22,953
✨The Art of Doing Science and Engineering — full review: goodreads.com/review/show/33…
1
1
399
✨ Build a Large Language Model — full review: goodreads.com/review/show/81…
1
3
276
𝗕𝘂𝗶𝗹𝗱𝗶𝗻𝗴 𝗮 𝗚𝗣𝗧 𝗠𝗼𝗱𝗲𝗹 For the past few weeks, I've been reading about Foundation Models [0] and decided to work on the implementation of the GPT architecture [1] to understand its building blocks and how it works under the hood. Here are the concepts I worked on in this implementation: Tokenization → Embeddings → Self-Attention → Multi-Head Attention → Transformer Block → GPT Model → Pretraining. — The tokenization part was focused on building tokens from the input text and transforming them into token IDs; Then using a BPE tokenizer algorithm [2] — Embeddings: representing tokens with a simple scalar value (ID) is too simplistic. Embeddings come to build richer representations. I built small embeddings for learning purposes and then increased the representation to scale that — Multi-Head Self-Attention: this was one of the most interesting parts, creating attention scores and building relationships between tokens to produce context vectors — Transformer blocks have the attention heads, dropout, layer norm, and the feed-forward network — Pretraining is a standard training process used for deep learning models. But in this case, we update the weights end-to-end, from the embeddings to the attention layer to the feedforward network The implementation was highly inspired by the Language Modeling from Scratch course [3] and the Build a Large Language Model book [4]. It's still very rudimentary, but very useful if you plan to learn these concepts in depth. 🔗 Article Link: Self-Attention, Foundation Models, and the GPT Architecture from Scratch: iamtk.co/self-attention-foun… --- In the future, I plan to write about finetuning (using foundation models and finetuning for other tasks) and optimizations (attention blocks optimization, GPU and kernel optimization). [0] Foundation Models at Nubank: teapawt.net/wordsofteekay/status/2… [1] LLM implementation repo: github.com/imteekay/llm [2] Tokenizers lecture: youtube.com/watch?v=SQ3fZ1sA… [3] Language Modeling from Scratch: youtube.com/playlist?list=PL… [4] Build a Large Language Model: manning.com/books/build-a-la…
[Paper Reading: Your Spending Needs Attention] I've just finished reading the "Your Spending Needs Attention" paper by Nubank, and not only are the results impressive, but the ML and engineering approach is also very interesting. It shows the power of self-supervised representation learning to automatically understand user behavior from raw (transaction) data, which made me think about how many insightful representations we are missing by not using it, and why (engineering and money trade-offs come to mind). Here's the research breakdown: causal self-attention + tabular feature embedding + fine-tuning for RecSys. Transformer-based model: > Text is All You Need: Individual transactions are tokenized, concatenated into a transaction string, and fed through a Transformer [0] to produce a transaction sequence embedding. > No Positional Embeddings (NoPE) [1]: drop the temporal information > FlashAttention [2] + NoPE = Efficient Long Contexts (transaction = ~14 tokens — the sequence gets large very fast): the model can train on much larger context lengths Tabular Features: > Feature embeddings for numerical and categorical variables > LightGBM: gradient-boosted tabular modeling > Deep Cross Network V2 (DCNv2) [3]: learn feature interactions Fine-Tuning — classification task for RecSys: > Low-Rank Adaptation (LoRA) [4]: injecting trainable low-rank matrices into attention layers to handle the "overfitting and catastrophic forgetting" issues. > Late Fusion: freeze the transformer embeddings and use them as static features passed into LightGBM or DCNv2 independently. > Joint Fusion (nuFormer): keep the transformer embeddings trainable end-to-end alongside the tabular features. It's very insightful how joint fusion trains the entire system end-to-end using a DNN, so gradients can flow through the embeddings compared to GBT. Other insightful ideas from the paper: > Context window problem: adding more data sources (e.g. financial products) can lead to worse results because each data source will "compete" for the available tokens for a fixed context window. > Scaling laws: larger model size, context lengths, and data volume lead to improved performance. There are still many interesting avenues they will explore, especially scaling laws and scaling the application to other products. It was also insightful how they are not just following the state of the art, but doing research to find new ideas [5]. --- Paper: arxiv.org/abs/2507.23267 --- [0] arxiv.org/abs/1706.03762 [1] arxiv.org/abs/2305.19466 [2] arxiv.org/abs/2205.14135 [3] arxiv.org/pdf/2008.13535 [4] arxiv.org/abs/2106.09685 [5] open.spotify.com/episode/11v…
4
5
1
87
5,014
✨ I worked on this article the whole day and made a lot of progress. I'm almost there. A lot of work, with many experiments, but it's getting traction. "Make Something Wonderful" inspired me to keep building and sharing.
[ML Grind] Finished: > Foundation Models: finished transformer-based model implementation from scratch + finetuning > Finished reading the Attention-based model in the industry paper: interesting insights about context length, scaling laws, and joint fusion Have been working on: > ML monitoring + alerting system for ML models > AI agent for business flow: interesting engineering learnings (agent/prompt refinements <> MCP <> backend + infra) > Real estate liquidity model: interesting learnings about temporal splits, model calibration, model optimization, and dataset exploration Plan for today: > Continue writing the blog post about the foundation model implementation > Continue the "Language Modeling from Scratch" course by Stanford > Read a new ML paper
2
2
17
1,586
This last GPU lecture (FLOPs/memory movement optimization) was awesome! ✨
2
155
I've just found out about this course on Foundation Models and Generative AI. Quite interesting lectures. I plan to watch the lectures as soon as I finish the Language Modeling from Scratch course. So many interesting things to learn.
3
9
155
5,947
Many people have already pointed out, but this course by Stanford is remarkable. It's been part of the first hour of my morning. Watching the lecture, taking notes, spawns new tabs with different papers mentioned, and coding to build the intuition behind each lecture. Mixture of experts was a nice lecture, but the one I liked the most so far was about PyTorch and resource accounting and how to make sense of CPU/GPU, memory, runtime/compute (FLOPs), etc., from first principles. 🔗 link: youtube.com/playlist?list=PL…
3
9
151
5,204
[Paper Reading: Your Spending Needs Attention] I've just finished reading the "Your Spending Needs Attention" paper by Nubank, and not only are the results impressive, but the ML and engineering approach is also very interesting. It shows the power of self-supervised representation learning to automatically understand user behavior from raw (transaction) data, which made me think about how many insightful representations we are missing by not using it, and why (engineering and money trade-offs come to mind). Here's the research breakdown: causal self-attention + tabular feature embedding + fine-tuning for RecSys. Transformer-based model: > Text is All You Need: Individual transactions are tokenized, concatenated into a transaction string, and fed through a Transformer [0] to produce a transaction sequence embedding. > No Positional Embeddings (NoPE) [1]: drop the temporal information > FlashAttention [2] + NoPE = Efficient Long Contexts (transaction = ~14 tokens — the sequence gets large very fast): the model can train on much larger context lengths Tabular Features: > Feature embeddings for numerical and categorical variables > LightGBM: gradient-boosted tabular modeling > Deep Cross Network V2 (DCNv2) [3]: learn feature interactions Fine-Tuning — classification task for RecSys: > Low-Rank Adaptation (LoRA) [4]: injecting trainable low-rank matrices into attention layers to handle the "overfitting and catastrophic forgetting" issues. > Late Fusion: freeze the transformer embeddings and use them as static features passed into LightGBM or DCNv2 independently. > Joint Fusion (nuFormer): keep the transformer embeddings trainable end-to-end alongside the tabular features. It's very insightful how joint fusion trains the entire system end-to-end using a DNN, so gradients can flow through the embeddings compared to GBT. Other insightful ideas from the paper: > Context window problem: adding more data sources (e.g. financial products) can lead to worse results because each data source will "compete" for the available tokens for a fixed context window. > Scaling laws: larger model size, context lengths, and data volume lead to improved performance. There are still many interesting avenues they will explore, especially scaling laws and scaling the application to other products. It was also insightful how they are not just following the state of the art, but doing research to find new ideas [5]. --- Paper: arxiv.org/abs/2507.23267 --- [0] arxiv.org/abs/1706.03762 [1] arxiv.org/abs/2305.19466 [2] arxiv.org/abs/2205.14135 [3] arxiv.org/pdf/2008.13535 [4] arxiv.org/abs/2106.09685 [5] open.spotify.com/episode/11v…
[ML Grind] Finished: > Foundation Models: finished transformer-based model implementation from scratch + finetuning > Finished reading the Attention-based model in the industry paper: interesting insights about context length, scaling laws, and joint fusion Have been working on: > ML monitoring + alerting system for ML models > AI agent for business flow: interesting engineering learnings (agent/prompt refinements <> MCP <> backend + infra) > Real estate liquidity model: interesting learnings about temporal splits, model calibration, model optimization, and dataset exploration Plan for today: > Continue writing the blog post about the foundation model implementation > Continue the "Language Modeling from Scratch" course by Stanford > Read a new ML paper
3
35
2,622