Streaming without stress: flexibility and error-handling in a data distribution pipeline

How do you stream data to a large number of downstream integrations in a resilient way? I will share how this has been implemented in a pipeline in Schibsted which at times processes more than 100 000 events/second. The architecture is centered around a Kafka cluster and has been in use for about seven years. I will also touch upon the concept of tenant and tenant isolation in a data distribution pipeline.

• Thread per consumer model, error-handling, tenant

————————————————————————————————————————————————————

Joanna Nordin – Staff Data Engineer | Schibsted

Joanna Eriksson works as a staff data engineer at the Norwegian company Schibsted. She holds a master’s degree in Computer Science and has been working as a software engineer for almost a decade. Her career has been focused on architecture and code for JVM-based applications with big data technologies such as Kafka and Spark. Having found a true passion in data engineering she enjoys sharing this with others who want to evolve in the data engineering domain.