This talk was recorded at NDC Copenhagen in Copenhagen, Denmark. #ndccopenhagen #ndcconferences #developer #softwaredeveloper    

Attend the next NDC conference near you: 
https://ndcconferences.com
https://ndccopenhagen.com/

Subscribe to our YouTube channel and learn every day:   
/       @NDC 

Follow our Social Media!

https://www.facebook.com/ndcconferences
https://twitter.com/NDC_Conferences
https://www.instagram.com/ndc_conferences/

#bigdata #architecture #database 

Detecting problems as they happen is essential in today's fast-moving world. This talk shows how to build a simple, powerful system for real-time anomaly detection using time-tested statistical methods. We'll use Apache Kafka for streaming data, Apache Flink for processing it in real time, and ARIMA models to detect unusual patterns. Whether it's spotting fraud, monitoring systems, or tracking IoT devices, this solution is flexible and reliable.

We'll start by exploring how Kafka helps collect and manage fast-moving data streams. Then, we'll demonstrate how Flink processes this data in real time and integrates ARIMA-based anomaly detection to uncover events as they occur. We'll dive into the details of how ARIMA works—covering how it models trends, seasonality, and residual noise to identify outliers in time-series data.

We'll also show how Apache Iceberg can be used to store historical data efficiently, enabling retrospective analysis, ongoing model evaluation, and performance improvements over time.

By combining real-time detection with long-term storage, you can build a robust system that evolves as your data grows. This talk includes clear examples and practical steps to help you build your own pipeline. It's ideal for anyone looking to use open-source tools to monitor and react to issues in real-time data streams.

NDC Conferences

Real-time anomaly detection can be approached through multiple methods depending on use case complexity. Simple statistical thresholds (three sigma, Z-score) provide fast initial filtering. ARIMA statistical models offer forecasting for time-series data with seasonal patterns. LSTM deep learning models handle complex patterns requiring nuanced detection. These approaches are often layered together in production systems using Apache Kafka for data streaming and Apache Flink for processing. The choice depends on data characteristics, response time requirements, and anomaly severity.

Mastering real-time anomaly detection with open source tools - Olena Kutsenko - NDC Copenhagen 2025