Weekly Data Engineering Newsletter. Subscribe to https://xitter.cf/t.co/trebyY8UGX | Wanna talk about Data engineering? Book Me here https://xitter.cf/t.co/OXj3VJheDH
Joined November 2020
- Tweets1.8K
- Following85
- Followers7.2K
- Likes66
Parquet is how columnar data sits on disk.
Arrow is how it sits in memory.
Engines that share Arrow pass data with zero copies, no serialization step.
From Hoyt Emerson's overview in DEW #292. Practice:
leetdata.ai/paths/lakehouse
#dataengineering
eBay monitors 2.6B listings without slowing the fee path.
Write a tiny event ID to a local ring buffer. Add 250 columns of context later in ClickHouse.
Keep the hot path small.
dataengineeringweekly.com/p/…
#systemdesign
Data modeling round, one decision per scenario:
SCD2: insert, never update
Org chart: manager_id points at employees
Data vault: hubs, links, satellites
Schema evolution: additive only
Practise on a canvas:
leetdata.ai/modeling
#dataengineering
Zalando ranks schema-mapping matches by:
ORDER BY approval_count DESC, similarity DESC
A reader's catch: a 0.81 match approved often beats a 0.97 approved once.
Flip it, or blend the two?
dataengineeringweekly.com/p/…
#sql
LinkedIn moved Spark shuffle onto Apache Celeborn.
Fetch failures: ~750/day to ~20
Shuffle cost per task: down ~90%
Runtime: down ~14%
The senior Spark question: where does your shuffle data live?
leetdata.ai/paths/spark
#apachespark
8 window-function problems, in order: running sum, LAG/LEAD, MoM growth, moving average, DENSE_RANK ties, QUALIFY, RANGE frames, sessions.
The trap: ROWS BETWEEN 6 PRECEDING is 7 rows, not 7 days.
leetdata.ai/problems
#sql #dataengineering
Quick SQL quiz.
A user signs up, comes back on day 6, then on day 8.
Day-7 retained: yes or no?
Answer in the replies. Try it first, free:
leetdata.ai/problems/day-n-r…
#dataengineering #sql
For engineers in India switching into data engineering:
All 200 leetdata.ai problems are free. SQL, Python, PySpark, hints, editorials. Plus 1 AI mock interview in week 1.
Practice for the rounds DSA prep skips.
leetdata.ai/problems
#dataengineering #sql
Ratings: 4.9, 4.8, 4.7, 4.7, 4.5
ROW_NUMBER: 1 2 3 4 5
RANK: 1 2 3 3 5
DENSE_RANK: 1 2 3 3 4
Ask for a "top 3" with RANK and you get 4 rows.
Top N per Group, free on leetdata.ai:
leetdata.ai/problems/top-n-p…
#dataengineering #sql
Affirm processes billions in credit risk and checkout workflows every day.
How do they handle real-time ML inference without crashing checkout latency?
They built an isolated "Intelligence Layer." 🧵👇
Want daily data engineering breakdowns, system architectures, and paper teardowns sent directly to your phone?
Join our new Data Engineering Weekly WhatsApp Channel! 🚀
👇 Join here: [whatsapp.com/channel/0029Va9…]
🚀 Apache DataFusion can now leverage Apple Silicon GPUs for massive query speedups!
By registering a single optimizer rule via ArrowMetal, full SQL sorts run 6.9x to 28.8x faster on Apple Silicon—without changing a single line of SQL. 🧵👇
dataengineeringweekly.com/i/…
🛡️ Smart Guardrails:
The rule evaluates warm/idle GPU costs and hand-back cases (e.g., complex float NaNs or streaming Top-K queries where CPU is better).
If conditions aren't optimal or memory limits trigger, it seamlessly falls back to standard CPU execution.
8 SQL problems. About 20 minutes end to end.
leetdata.ai's "Start here" set: the SQL a DE interviewer opens with. CASE WHEN order, BETWEEN bounds, blank vs NULL, a first CTE.
Easy on purpose. Screens slip here.
leetdata.ai/problems
#dataengineering #sql