Session Outline

How do you stream data to a large number of downstream integrations in a resilient way? I will share how this has been implemented in a pipeline in Schibsted which at times processes more than 100 000 events/second. The architecture is centered around a Kafka cluster and has been in use for about seven years. I will also touch upon the concept of tenant and tenant isolation in a data distribution pipeline.

Key Takeaways

  • Thread per consumer model, error-handling, tenant

————————————————————————————————————————————————————

Speaker Bio

Joanna Nordin – Staff Data Engineer | Schibsted

Joanna Eriksson works as a staff data engineer at the Norwegian company Schibsted. She holds a master’s degree in Computer Science and has been working as a software engineer for almost a decade. Her career has been focused on architecture and code for JVM-based applications with big data technologies such as Kafka and Spark. Having found a true passion in data engineering she enjoys sharing this with others who want to evolve in the data engineering domain.

October 26 @ 11:00
11:00 — 11:30 (30′)

Day 2 | 26 Oct 2023 | INFRASTRUCTURE + DATA ENGINEERING STAGE

Joanna Nordin – Staff Data Engineer | Schibsted