This second part of the streaming ETL series will discuss the typical ETL challenges data lineage, error handling and data quality with a special focus on the specificities to be considered in a streaming approach. Relating batch reporting requirements to streaming ETL is also tackled. The series is concluded with an overview on testing streaming ETL systems and a summary.
The new central data system for the banking and insurance sector by SAP is FSDP, the Financial Services Data Platform. When establishing FSDP, financial service companies are faced with the challenge of integrating this new platform into their overall architecture. This article gives an introduction to solving this ETL problem with event streaming using Apache Kafka to build a reliable and scalable real-time data integration.
One main characteristic of SAP FPSL is the clear distinction between financial instruments on the one hand and insurance contracts on the other hand. This, however, does not mean that insurances for instance are limited to the functionality of the subledger accounting for insurance contracts. Insurances that possess a portfolio of financial instruments or reinsurances that are active in the context of capital markets instruments can in fact use both the subledger accounting solution for insurance contracts and financial instruments.