ZDE (@zde37) header
ZDE (@zde37) profile picture

I am Inevitable. Building @getremac | Distributed Systems Engineer | 1 Cor. 15:10

Eminence
Joined October 2023
Pinned Tweet
For years, I kept solving the same problem in different forms: the database commits, but the message publish or downstream write after it can still fail. Outboxes, queues, triggers, debezium, etc helped. The retries, cleanup, monitoring, reconciliation, infrastructure, and separate recovery tools remained. Eventually, I stopped treating them as separate problems. That led me to build @getremac , a high-performance, database-agnostic transaction log processor. Remac is built to move fast without weakening correctness, progress, or recovery. Those requirements became the Remac Contract. The vision is simple: make every committed data change immediately useful to every system that must act on it.
1
1
345
The final state does not always tell the whole story. A downstream system (e.g audit systems) may need to know not only where the data ended up, but which committed changes happened along the way. Remac makes that requirement explicit.
An account changes from 'starter' to 'pro', then back to 'starter'. A later table query correctly returns 'starter'. It cannot tell you that either committed transition happened. Current state answers what exists now. Committed history answers which accepted changes produced it, and in what order. Replication, audit, billing, and event processing can need different parts of that history.
205
How do changes inside PostgreSQL make their way into other systems reliably? Sooner or later, you’ll run into: wal_level = logical It’s an important setting, but enabling it does not give you a working change stream. We break down what ‘wal_level’ actually does, why changing it requires a restart, how it relates to PostgreSQL’s transaction log, and where its role ends. Article: remac.io/blog/what-postgresq…
1
3
201
It’s interesting how often we misinterpret system ACKs in distributed systems. We treat them as strong guarantees, but they actually promise way less than most engineers realize.
92
The only thing we learn from history is that we learn nothing from history.
76
CDC proved the value of committed change. The bigger opportunity is treating transaction log processing as infrastructure in its own right, where correctness, recovery, and delivery are designed as one system rather than added around the edges.
Transaction log processing should become a primary data-infrastructure category. CDC introduced many teams to the value of committed change. The same foundation can do much more when correctness and recovery are designed across the full path.
93
Most data problems are coordination problems in disguise. The same underlying state is copied, transformed, indexed, analyzed, archived, and consumed by different systems. The difficulty is keeping those representations coherent as the source changes. That is why data management cannot be reduced to storage or movement. The real work is preserving continuity across systems.
2
97
You cannot build beyond the scope of your exposure and foresight.
42
Databases don't just store state. They continuously describe how state changes. Transaction-log processing turns that history into infrastructure.
Every committed transaction leaves behind a record of what changed and in what order. That record exists because the database needs it to preserve its own guarantees. But once you can process that history reliably, it becomes useful outside the database too. You can build downstream systems from the same sequence of changes the database itself considers committed. That’s the layer we’re building at Remac.
1
82
Many engineers overlook the fact that data pipelines are living infrastructure. Treating data ingestion as a static, one-time setup rather than an evolving ecosystem invites catastrophic events like silent schema failures that corrupt your downstream systems.
70
Technology isn’t fundamentally about gadgets, software, etc Its essence is simple: Technology is applied knowledge that extends human capability. The best technologies amplify what humans are capable of doing.
1
54
One of the most expensive things in software is a temporary design decision that survives long enough to become “just how the system works”. Eventually nobody remembers why the constraint exists. They just design around it.
1
67
What is a transaction log, and why does it matter? Database engines commonly maintain one or more ordered logs of changes for purposes such as durability, crash recovery, replication, and point-in-time recovery. There isn’t one universal kind of “transaction log”. Its contents, durability guarantees, ordering semantics, retention, and access methods all depend on the database engine and its configuration. Having such a log and consuming it are also two different things. Transaction log processing means taking a database’s native log, or a supported decoded representation of it, and turning those recorded changes into a usable stream for downstream systems and processing workflows. This guide breaks down the concept in more detail: remac.io/blog/what-is-a-tran…
2
63
Most organizations are drowning in data but starving for utility. Data alone is not valuable enough, value comes from moving it reliably, in real time, to the systems and people that can act on it.
52
The closer software gets to a database log, the less room there is for “close enough”, there is zero margin for error. A value demands high precision, retaining its exact meaning across snapshot, live capture, storage, replay, and delivery. If a single path alters a data type, the entire system inherits that ambiguity, proving the necessity of strict kernel contracts.
50
Complexity is usually the tax we pay for failing to find the right abstraction. Good design removes clutter by grouping related ideas into clear, reusable pieces.
50
While building @getremac , I kept coming back to one question: Is this trade-off required by the problem, or only by the current design? That question became Zero-Deficit Engineering. Read more: zde37.com/blog/zero-deficit-…
82
Engineering is inherently beautiful because it integrates seamlessly with life and the natural world. It’s funny how we stress over building ‘complex’ things, when the simplest, most perfect blueprint has been staring us in the face all along. Simply put: “Human innovation is often at its best when it stops trying to outsmart nature and starts copying it”
1
77