This second part of the streaming ETL series will discuss the typical ETL challenges data lineage, error handling and data quality with a special focus on the specificities to be considered in a streaming approach. Relating batch reporting requirements to streaming ETL is also tackled. The series is concluded with an overview on testing streaming ETL systems and a summary.
The new central data system for the banking and insurance sector by SAP is FSDP, the Financial Services Data Platform. When establishing FSDP, financial service companies are faced with the challenge of integrating this new platform into their overall architecture. This article gives an introduction to solving this ETL problem with event streaming using Apache Kafka to build a reliable and scalable real-time data integration.