An open course on how stream processing systems are organized and how they work.
Follow events from their source to a continuously updated result. Learn why time, state, and recovery matter, how Kafka fits into the system, and what happens when events arrive late or twice.
Backend developers, data engineers, and software engineers who want to understand the principles behind stream processing systems. The course introduces the concepts as they become useful; prior experience with a stream processing framework is not required.
The focus is on concepts and architectural trade-offs. Connected examples follow grocery deliveries, AI service usage, and charging station availability as new problems arise. You'll see why a mechanism is needed, how it works, and what its limits are before exploring framework-specific APIs.
24 articles are available. Each link opens the first article in that section.
- Introduction: when to process data as it arrives, periodically, or on demand.
- Fundamentals: events, sources and sinks, transformations, state, and grouping by key.
- Time and windows: event time, processing time, window types, watermarks, and late data.
- Duplicates, loss, and recovery: delivery guarantees, checkpoints, consistent recovery, and external effects.
- Distributed processing and the log: queues and logs, partitions, Kafka, replay, retention, and change data capture.
- Streams, tables, and joins: materialized views, changing reference data, stream joins, and streaming SQL.
- Implementations and comparison: the same computation in Kafka Streams, Flink, and Spark Structured Streaming.
- Event contracts and evolution: schema compatibility and events that cannot be processed.
- Using results: views for reading and automated actions.
- Operations: overload, data skew, scaling, observability, and recovery.
- The whole system: tracing the data path and revisiting architectural decisions.
Open an issue to report an error or ask a question. For an error, include the article link and the passage or behavior you think needs fixing.
See Contributing for correction and topic suggestion guidelines.
© 2026 Grig Alex. The text of the course articles is licensed under CC BY-NC-SA 4.0. You may share and adapt it for noncommercial purposes if you credit Grig Alex, link to the license, indicate changes, and share adaptations under the same license. See LICENSE for the full terms.
The site uses open-source software, including Astro, Starlight, and Pagefind, under their own licenses.