Eric Tham retweeted
GPU architecture, clearly explained.
The usual assumption is that a faster GPU means more compute, so a chip rated for more operations per second should generate more tokens per second.
That rarely holds. You can rent a top-end chip, load a model, watch utilization sit high, and still get a few dozen tokens per second out of hardware capable of a thousand trillion operations.
The arithmetic units are not the limit. They spend most of their time waiting for numbers to arrive.
Memory bandwidth is how fast those numbers can be delivered, and it grows far more slowly than arithmetic capability with each new generation.
The H100 makes this concrete. It does 989 trillion operations per second at 16-bit precision, while its memory delivers 3.35 trillion bytes per second.
Divide one by the other and you get about 295. That is how many operations the chip has to perform on every byte it fetches just to keep its arithmetic units busy.
Generating a token comes nowhere close. Each weight gets multiplied by one number and added into a running total, so two operations per weight, and each weight costs 2 bytes to read at 16-bit precision.
That works out to 1 operation per byte, roughly three hundred times short of what the chip needs.
Once you see that, the layout of a GPU stops looking arbitrary. All of it is organized around distance, because every trip to fetch a value is the expensive part.
Execution is nested in four steps, smallest first.
→ Thread. The smallest unit of work, doing arithmetic on its own handful of values.
→ Warp. A group of 32 threads that move in lockstep and share a single instruction, which is what the hardware actually schedules.
→ Thread block. A batch of warps, up to 32 of them or 1024 threads, handed to one compute unit and staying there until it finishes.
→ SM. The streaming multiprocessor, a small self-contained machine with its own arithmetic units, its own storage, and its own scheduler. An H100 has up to 132 of them.
Memory is nested along the same lines, and that is the part that decides your speed.
→ Registers. 256 KB per SM, split so each thread gets a private slice nobody else can read. A value here already sits next to the thing that needs it, so reading it costs nothing worth measuring.
→ Shared memory and L1. A few hundred KB per SM. Shared memory is the only level you place things into yourself, and every thread in a block can read it, which is where the name comes from.
→ L2 cache. Around 50 MB sitting below all the SMs, and the first level every one of them can see. Reaching it means leaving your own SM.
→ HBM (High Bandwidth Memory). Around 80 GB, holding model weights, the KV cache, and activations. It sits outside the processing chip entirely, and reaching it is the slowest thing a GPU does.
The nesting is what buys you speed. Anything that stays inside one SM is fast because nobody outside needs to see it, and anything the whole chip has to agree on pays for the trip down.
Add up the registers across every SM and you get roughly as much storage as the L2 underneath them, which is nothing like the neat pyramid you see on a CPU.
So making a GPU faster is mostly about how many round trips to HBM you can avoid. FlashAttention is the clearest case, since ordinary attention writes a large intermediate result all the way down and reads it back, while the tiled version keeps it up on the SM and runs far faster doing the same arithmetic.
I wrote the full breakdown of how a GPU works, from this layout up to the ratio that sets your token rate. The article is quoted below.
I have also put together a 10-week, 30-minutes-a-day roadmap for engineers who want to actually run LLM inference in production, not just read about it.
Check this out: github.com/patchy631/time-to…
(don't forget to star 🌟)
Stay tuned for more on this!
Eric Tham retweeted
LLM Inference Engineering Roadmap
Part 1: Foundations
1. Training vs Inference
2. Inference Engineering
3. Closed vs Open Models
4. Latency vs Throughput
5. Model Selection
6. Model Evaluation (Evals)
Part 2: Transformers and LLM Architecture
1. Tokens
2. Token Embeddings
3. Positional Encoding
4. Transformer Architecture
5. Matrix Multiplication (Matmul)
6. Attention
7. Queries, Keys, Values
8. Causal Masking
9. Multi-Head Attention (MHA)
10. Feed-Forward Network (FFN)
11. Logits
12. Mixture of Experts (MoE)
13. Active vs Total Parameters
Part 3: LLM Inference Mechanics
1. Autoregressive Generation
2. Chat Templates
3. Context Window
4. Prefill vs Decode
5. KV Cache
6. Sampling
7. Temperature
8. Top-k Sampling
9. Top-p (Nucleus) Sampling
10. Stop Tokens
11. Reasoning Models
12. Token Streaming
Part 4: Performance Metrics
1. Time to First Token (TTFT)
2. Time Per Output Token (TPOT)
3. Throughput
4. End-to-End Latency
5. Latency Percentiles (P50, P90, P99)
6. Cost per Token
Part 5: Bottleneck Analysis
1. Compute vs Memory Bandwidth
2. Arithmetic Intensity
3. Roofline Model
4. Compute-Bound vs Memory-Bound
5. Prefill Bottleneck (Compute)
6. Decode Bottleneck (Memory)
7. Memory Footprint
8. KV Cache Memory Calculation
9. Batching and Arithmetic Intensity
Part 6: Hardware
1. CPU vs GPU
2. GPU Architecture
3. Tensor Cores
4. High-Bandwidth Memory (HBM)
5. DRAM vs SRAM
6. NVLink
7. InfiniBand
8. Groq LPU
9. Local (Edge) Inference
10. Cloud vs On-device Deployment
Part 7: Software Stack
1. CUDA
2. CUDA Kernels
3. CUDA Graphs
4. Kernel Fusion
5. PyTorch
6. Inference Engines
7. OpenAI-Compatible Server
Part 8: Benchmarking and Profiling
1. Performance Benchmarking
2. Traffic Simulation
3. Load Testing
4. Profiling
Part 9: Attention Optimizations
1. Standard Attention Inefficiency
2. FlashAttention
3. PagedAttention
4. Multi-Query Attention (MQA)
5. Grouped-Query Attention (GQA)
6. Sliding Window Attention
7. Attention Sinks
8. KV Cache Compression
Part 10: Batching and Scheduling
1. Batching
2. Static Batching
3. Continuous Batching
4. Batch Size vs Latency
5. Chunked Prefill
6. Prefill-Decode Interference
Part 11: Caching
1. Prefix Caching
2. Cache-Friendly Prompt Design
3. Prompt Caching
4. KV Cache Offloading
5. Cache-Aware Routing
6. Long Context Handling
Part 12: Speculative Decoding
1. Speculative Decoding
2. Rejection Sampling
3. Acceptance Rate
4. Draft-Target Speculation
5. N-gram Speculation
6. Speculation vs Batch Size
Part 13: Quantization
1. Quantization
2. Number Formats
3. Post-Training Quantization (PTQ)
4. Weight-Only Quantization (W4A16)
5. Weight-Activation Quantization (W8A8)
6. KV Cache Quantization
7. Measuring Quality Impact
Part 14: Model-Level Optimizations
1. Model Size vs Speed
2. LLM Routing
Part 15: Parallelism
1. Why Multi-GPU
2. Model Parallelism
3. Communication Overhead
4. Data Parallelism
5. Tensor Parallelism (TP)
Part 16: Disaggregation
1. Prefill-Decode Disaggregation
2. KV Cache Transfer
3. When to Disaggregate
Part 17: Production Infrastructure
1. Replicas
2. Autoscaling
3. Cold Starts
4. Observability
Start here: github.com/amitshekhariitbhu…
Eric Tham retweeted
LLM engineer's handbook
(30 minutes a day, 10 weeks, 50 lessons)
a roadmap for llm inference serving where everything points at one service instead of scattering across demos. you get the mental model first, then serve a model, instrument it, load test it past 1000 concurrent requests, and tune it. you finish with a stack you configured yourself and a benchmark worth publishing.
here is what it covers:
→ the roofline model, and why decode waits on memory while prefill waits on compute
→ vLLM internals, PagedAttention and the scheduler, read from the code
→ Prometheus and Grafana for TTFT, inter-token latency, and queue depth
→ SGLang and RadixAttention prefix reuse, benchmarked against vLLM
→ load testing past 1000 concurrent requests
→ quantization across FP16, FP8, and INT4, on quality as well as speed
→ speculative decoding and KV eviction, including where the gains disappear
→ disaggregated prefill and decode, deployed on Kubernetes
→ a cost, latency, and quality router with per-request token budgeting
→ publishing a reproducible benchmark
the roadmap on GitHub: github.com/patchy631/time-to…
(don't forget to star 🌟)
i am also writing an article for each major topic. the first one is out, on how a GPU actually works.
the article is quoted below.
your AI trading bot is lying to you.
and it doesn't even know it.
i gave an agent 1,000 strategies with zero edge.
pure coin flips.
it still came back with a "winner."
Sharpe 3.3.
10 tested → 1.5
100 tested → 2.5
1,000 tested → 3.3
10,000 tested → 3.9
the more it tests, the better the fake looks.
agents made testing free.
that made luck the most common result.
5 rules to catch it
Eric Tham retweeted
ESTOS 39 MINUTOS DEL FUNDADOR DE KIMI SON ORO PURO
explica cómo montó desde China una empresa de IA valorada en más de 50.000 millones
hay másteres de 20.000 € que no enseñan lo que él cuenta en 10 minutos.
39 minutos, GRATIS y sin relleno.
guárdalo y mira esto cuando tengas tiempo, me lo agradecerás
Eric Tham retweeted
🚀 گوگل یک PDF نهصفحهای درباره Harness Engineering منتشر کرده؛ مفهومی که به نظرم باید هر کسی که با AI Agent کار میکند، جدی بگیرد.
فرمول اصلی ساده است:
Agent = Model + Harness
یعنی مدل هوش مصنوعی بهتنهایی ایجنت نیست. چیزی که آن را به یک نیروی اجرایی قابلاعتماد تبدیل میکند، زیرساختی است که دور آن میسازیم.
اما این زیرساخت از چه چیزهایی تشکیل میشود؟
۱. Guides: قوانین و دستورالعملهایی که تجربه شکستهای قبلی را به دانش دائمی تبدیل میکنند.
۲. Sensors: تستها، Linterها و ابزارهای اعتبارسنجی که اجازه نمیدهند هر خروجی اشتباهی به مرحله بعد برود.
۳. Agentic Loop: چرخه برنامهریزی، اجرا، بررسی و اصلاح؛ با محدودیت بودجه و امکان ارجاع به انسان.
۴. External Memory: حافظهای خارج از مدل برای حفظ تصمیمها، وضعیت پروژه و دستاوردهای جلسات قبلی.
۵. Permissions: کنترل دسترسی ایجنت به ابزارها و تعیین مرز استقلال آن.
۶. Observability: ثبت و پایش تمام فعالیتها، هزینهها، خطاها و تلاشهای مجدد.
اما داستان اینجا تمام نمیشود!
چند پروژه متنباز برای ورود عملی به این حوزه:
🔹 Google Harness²
github.com/google-research/h…
🔹 Learn Harness Engineering
github.com/walkinglabs/learn…
🔹 Awesome Agent Harness
github.com/Gloriaameng/Aweso…
🔹 Harness Engineering
github.com/TrishamBP/harness…
و حالا یک مفهوم جذابتر:
Graph Engineering
تصور کن چند ایجنت مستقل داشته باشی که هرکدام Harness مخصوص خودشان را دارند.
یکی تحقیق میکند، دیگری تحلیل میکند، سومی کدنویسی میکند و چهارمی خروجی را اعتبارسنجی میکند.
حالا اگر یکی شکست بخورد، شکستش میتواند به سیگنالی برای اصلاح عملکرد بقیه تبدیل شود.
اینجا دیگر با یک چتبات یا ایجنت ساده طرف نیستیم؛ با یک سیستم اجرایی چندایجنتی روبهرو هستیم.
یک نکته که نباید فراموش کنیم:
Hermes را میتوان بهعنوان محیط اجرای ایجنت در نظر گرفت؛ جایی که ابزارها، مهارتها، حافظه و گردشکارها کنار هم قرار میگیرند.
Harness Engineering کمک میکند این محیط را قابلکنترل، قابلاندازهگیری و قابلاعتماد کنیم.
به نظرم آینده متعلق به کسانی نیست که صرفاً مدل قویتری انتخاب میکنند؛ بلکه متعلق به کسانی است که سیستم بهتری دور مدل میسازند.
مدل، مغز است؛ Harness، سیستم اجرایی؛ و Graph، شبکه همکاری مغزهاست.
Fable 5.2 builds the most POWERFUL trading agents...
I wrote 11-page research paper on exactly how to make Fable 5.2 hunt profitable strategies 24/7, with the CODE and EVERY PROMPT inside...
here's full guide (ready prompt below):
step 1 → paste prompt 6 from the paper into Claude Code, it builds all 8 agents and runs an offline demo before you add a single key
step 2 → add 2 keys: Anthropic and a Telegram bot, market data is free: prices, options chains, SEC insider filings, Fed statements
step 3 → every night a Haiku 5.5 swarm reads 300 markets at 1/80 of Fable's price and flags the 4 mispricings quant funds hunt
step 4 → Fable 5.2 turns each flag into a strategy in Python, a bull and a bear argue it, then it gets up to 5 years of backtests with fees
step 5 → kill rules: Sharpe above 1.5, drawdown below 15%, t stat above 3.0, the grader is code the AI can't edit
step 6 → surviors ping your Telegram with instrument, confidence, action window and half Kelly size, you place every trade yourself
the result: a top Quant firm already uses this model for trading & research (really), you get the same brain, 8 agents, 24/7, for almost free
Send this PDF to your AI trading system. Then read article below on how to build best agent system:
Eric Tham retweeted
Research papers you must read for AI Engineer interviews:
1. Attention is all you need (Transformers)
2. LoRA (Low rank adaption)
3. PEFT ( Parameter Efficient Fine Tuning)
4. VIT (Vision Transformers)
5. VAE (Variational Auto Encoder)
6. GANs ( Generative Adversarial Networks)
7. BERT ( Bidirectional Encoder Representation from Transformers)
8. Diffusion Models (Stable Diffusion)
9. RAG (Retrieval Augment Generation)
10. GPT (Generative Pre-trained Transformers)
Eric Tham retweeted
JPモルガン(米大手投資銀行)のPython研修の教材が
GitHubで無料公開されてる。
・Pythonの基礎
・コードのベストプラクティス
・pandasで金融データ分析
・Web APIとWebサイト作り
・グラフの作り方
・3Dプロット
・企業の倒産リスクを測るAltman Zスコア
アナリストやトレーダー向けに作られた教材で、GitHubのスターは1.4万を超えています。
github.com/jpmorganchase/pyt…
Eric Tham retweeted
Website changes shouldn't mean rebuilding your data extraction workflow from scratch.
That's where a reliable web data solution makes a difference. Oxylabs Web API MCP helps developers access live web search and structured data extraction directly from coding tools like Cursor and Codex.
Useful for anyone building AI agents that depend on real-time web data.
GitHub: github.com/oxylabs/web-api-m…
Eric Tham retweeted
A Stanford professor took two average stocks, rebalanced them daily, and turned $100,000 into $7 million without predicting a single price.
Bookmark & watch today, no matter what.
Eric Tham retweeted
✅ Neutron’s Stage 1 Forward Dome has been built and cured.
The dome has come off the AFP machine as planned and completed its heat and pressure processing cycle to cure the carbon composite material.
Next step: analysis and material tests to confirm quality and strength, before it moves to the next phase of Stage 1 tank assembly.
Eric Tham retweeted
99% of options traders are gambling because they never learned the fundamentals.
This 1-hour Yale lecture changes everything.
In just 60 minutes, you’ll learn more about options trading than most overpriced trading courses ever teach.
No hype. No fake gurus. Just real knowledge.
Save this and watch it without distractions. 📌
Eric Tham retweeted
99% of options traders are gambling because they never learned the fundamentals.
This 1-hour Yale lecture changes everything.
In just 60 minutes, you’ll learn more about options trading than most overpriced trading courses ever teach.
No hype. No fake gurus. Just real knowledge.
Save this and watch it without distractions. 📌
🚨CHINA ACABA DE PRESENTAR UNA PIZARRA CON IA QUE CONVIERTE DIBUJOS EN MODELOS 3D EN TIEMPO REAL
Ya está funcionando en 60.000 escuelas con 160 millones de estudiantes.
Dibujas un cubo en la pizarra.
La IA lo convierte en un modelo 3D que gira, marca los ángulos y las áreas sola, y te explica cada paso en vivo.
Lo mostraron en plena conferencia mundial de IA en Shanghai.
Esto no es un prototipo. Ya está en las aulas.
list of 15 best robot learning researchers on X:
1. @hausman_k, Karol Hausman
2. @deepakpathak, Deepak Pathak
3. @Ken_Goldberg, Ken Goldberg
4. @davsca1, Davide Scaramuzza
5. @pulkitology, Pulkit Agrawal
6. @zhou_xian_, Zhou Xian
7. @0x_meden, Martin
8. @zipengfu, Zipeng Fu
9. @jiajunwu_cs, Jiajun Wu
10. @danfei_xu, Danfei Xu
11. @abhishekunique7, Abhishek Gupta
12. @DorsaSadigh, Dorsa Sadigh
13. @shahdhruv_, Dhruv Shah
14. @mangahomanga, Homanga Bharadhwaj
15. @rocky_duan, Rocky Duan
Eric Tham retweeted
This is f*cking insane. Former Google CEO Eric Schmidt, the man who ran Google for 10 years:
"If you really want to make money, it's actually easy, found an agentic AI company, I mean build an agent to do something
If you build a better agent than anyone else, you can probably build one incredible company and own it yourself"
In 27 seconds, he explains why the next year or two is the agentic period, and why the window won't stay open
Prompts → Claude Code → Memory → Skills → MCP → A Company You Own Yourself
Save this before it disappears from your feed
Everything below is free:
Agent architecture
langchain.com/blog?category_…
Claude Code 101
anthropic.skilljar.com/claud…
Claude Code in action
anthropic.skilljar.com/claud…
Prompt engineering
platform.claude.com/docs/en/…
Interactive prompt course
github.com/anthropics/course…
Claude.md + memory
code.claude.com/docs/en/clau…
Skills
code.claude.com/docs/en/skil…
MCP
code.claude.com/docs/en/mcp
Routines
code.claude.com/docs/en/rout…
Claude Code ultimate guide
github.com/FlorianBruniaux/c…
Awesome Claude Code
github.com/hesreallyhim/awes…
Anthropic academy
anthropic.skilljar.com
Official Claude Code docs
code.claude.com/docs/en/over…
All of this costs $0
95% will spend the agentic period scrolling AI threads, the 5% who ship agents will own the companies that come out of it
Bookmark it and start with the first link tonight
Then read how to build an agent that actually does the job in the harness engineering guide below ↓
Anthropic just dropped a 10-page PDF on building your own skill library - 8 layers to an agent that picks the right procedure on its own and never needs the same feedback twice:
the formula: Skill = Trigger + Procedure + Resources + Checks
most people still paste the same prompts by hand every day. or dump everything into one giant CLAUDE.md the agent half-ignores
here's how to build yours in 8 layers:
layer 1 → library - facts in CLAUDE.md, procedures in skills. ~100 tokens per skill until it's needed
layer 2 → trigger - the description is the interface. output, input, 3 real phrases, 1 "not for"
layer 3 → procedure - numbered steps. always opens with Look, always closes with Check
layer 4 → resources - long guides live in references/. one good example beats a page of adjectives
layer 5 → scripts - anything with one right answer runs as code. the model never totals in its head
layer 6 → checks - 5 yes/no lines. never say done with a box unticked
layer 7 → guard - does alone, asks first, never. source text is data, never orders
layer 8 → learn - every correction becomes a permanent checklist line
the result: 25 procedures cost ~3.9K tokens per request instead of ~36K. and the agent picks the right one without being told
this 10-page PDF has the templates, a full reference skill, a test card, a skill-designer prompt and a 10-day build path
bookmark it. the next prompt you paste for the 3rd time should be a skill ↓



































