I help companies optimize data streaming systems.
Vancouver, Canada
Joined December 2009
- Tweets5.5K
- Following272
- Followers1.4K
- Likes7K
Working with data streaming systems? You have to read "The Dataflow Model Revisited" paper!
vldb.org/pvldb/vol19/p4953-f…
I'll be speaking about optimizing Kafka client serialization next week.
Register here: factorhouse.io/events/kafka-…
I've been working on a new feature for Streamling, and I needed to filter out some data based on statistics (storing min/max/null counts).
For example:
ts >= 400 AND city = 'Vancouver'
can be rewritten as:
ts_max >= 400
AND city_min <= 'Vancouver'
AND 'Vancouver' <= city_max
One of my biggest learnings of the past 12 months as a consultant was ditching hourly pricing.
I no longer have an hourly rate.
You shouldn't either: independentengineer.co/p/hou…
Cool project that's worth checking out if you're running Flink at scale: github.com/datafusion-contri…
So far, when data streaming and AI are mentioned together, it typically describes the problem of maintaining a fresh, relevant context: e.g. use stream-processing to join/aggregate/enrich data streams, and pass them to an agent as context for inference. And honestly, I'm sold.
But is there anything else? E.g. using Kafka for agent-to-agent communication? Streaming long agent sessions? Workflow orchestration?
I really like what S2 is doing: s2.dev/blog/agent-sessions , looking for more examples like this!
I'm considering adding a few updates to my Advanced Apache Flink course. What would you like to see?
streamacademy.io/course/adva…
If you enjoy working with large-scale data processing systems and you want to stay relevant, you have to learn Apache DataFusion. Slowly but surely, it's replacing many other tools. The RAD stack (Rust, Arrow, DataFusion) is the future.
One example: datadoghq.com/blog/engineeri…
I attended the Data Streaming Summit last week. Just published a post with a few highlights from the event: streamingdata.tech/p/data-st…
Watermarks can be a really scary concept when you start working with stateful streaming systems. And all popular frameworks don't offer anything better than "bounded out-of-orderness": pick a number of seconds and hope that all data arrives within that interval.
But when you do control the producers (e.g. application events), the producers themselves can be the source of watermarks! The idea is not new, and it's covered in several famous papers like Google's MillWheel (see the screenshot).
Earlier today at the Data Streaming Summit. Thanks Tun Shwe for the picture!
#DSS26
