Inventing @Temple | ex - Software Engineer at @zomato @AppSecure @Amazon @TartanHQ
Gurgaon, Delhi, Ghaziabad
Joined April 2017
- Tweets509
- Following1.1K
- Followers2.1K
- Likes957
So I was going through Prometheus Internals and learned something interesting.💡
When we think about Prometheus, we often imagine it as "the metrics database" scraping, storing, and querying endless time series from our apps. But under the hood, it’s doing something surprisingly clever that makes it much more efficient than you'd expect.
✅ Prometheus assumes that most metric data looks boring and predictable, then exploits that fact ruthlessly.
It is inspired by a 2015 paper published by Facebook titled "Gorilla: A Fast, Scalable, In-Memory Time Series Database" (vldb.org/pvldb/vol8/p1816-te…).
Here’s the trick: Prometheus doesn’t actually store full timestamps or raw values for every single metric sample. Instead, it uses a smart compression technique based on deltas and XOR encoding.
➡️ The Problem with Naive Storage
Let’s say Prometheus scrapes a metric every 15 seconds. A naive approach would store each sample like this:
[timestamp: 64 bits] [value: 64 bits]
That’s 16 bytes per sample. For millions of time series scraped every few seconds, the storage cost explodes.
➡️ The Delta + XOR Magic
Prometheus takes advantage of two simple observations about metrics:
👉🏻 Timestamps are Predictable:
Scrapes usually happen at fixed intervals (e.g., every 15s). Instead of storing the full timestamp, Prometheus stores the difference (delta) between consecutive timestamps.
Example, for timestamps like
1694000000, 1694000015, 1694000030
→ Store: 1694000000 (first one raw), +15, +15
👉🏻 Values don’t change drastically
Prometheus stores the first value as is, but every subsequent value is XOR’d with the previous one. If values are close (like CPU usage moving from 0.51 to 0.52), the XOR result has lots of leading zeros. Those zeros compress very well.
✅ The Payoff
Because of this scheme, Prometheus can often store 2-3 bytes per sample instead of 16.
That’s why we can comfortably run Prometheus on a single node with millions of metrics without needing a giant cluster behind it.
📋 References
- Prometheus TSDB Format Docs: github.com/prometheus/promet…
- Prometheus Source Code (XOR Compression: github.com/prometheus/promet…
- Facebook Gorilla Paper: vldb.org/pvldb/vol8/p1816-te…
Last week, I faced an interesting problem at work. I received a critical warning from AWS regarding our PostgreSQL RDS cluster, it was approaching transaction ID wraparound, something that can cause a full database outage if not handled in time.
This was a good reminder of how important it is to understand the internals of the systems we rely on every day.
For those unfamiliar, PostgreSQL uses a 32-bit transaction counter (XID) that wraps around after ~2.1 billion transactions. If old rows aren’t vacuumed and frozen in time, PostgreSQL can no longer determine which rows are visible, forcing it to stop accepting new writes to prevent data corruption.
In our case, it turned out that autovacuum wasn’t keeping up due to high write throughput and a few misconfigured thresholds.
What helped:
👉🏻 Identifying large tables with unvacuumed rows.
👉🏻 Manually running 𝖵𝖠𝖢𝖴𝖴𝖬.
👉🏻 Tuning autovacuum workers settings and vacuum cost limits.
It’s one of those things that often goes unnoticed until it becomes urgent. If you're running PostgreSQL on RDS (or self-managed), keep an eye on 𝖺𝗀𝖾(𝖽𝖺𝗍𝖿𝗋𝗈𝗓𝖾𝗇𝗑𝗂𝖽) and ensure autovacuum is running efficiently.
Invisible debt in infrastructure has a way of surfacing at the worst possible time. This one got our attention early, worth sharing in case it helps others avoid a painful surprise.
WTF IS THIS? @Airtel_Presence @airtelindia
Imagine getting blocked by customer care 🤡
- It is a PIN and not an OTP. And as the name suggests, it need not be a 'one-time' thing. It is a fixed PIN for each user and there is no issue if two users have the same PIN.
- It is only used to start a ride and is only shared between the captain and the passenger.
- Even for Uber, You can use your mobile number's last 4 digits, and multiple people can have the same last 4 digits right?
Learn to be polite and respectful👍🏻
