Event Streaming Operations: The Overlooked Cost Behind Real-time Data Insights

While batch-driven data pipelines can support a wide variety of the machine learning and analytical use cases, event streaming is an exhilarating and fast-growing paradigm in big data that enables real-time data insights. It can be tempting to adopt event streaming frameworks solely to optimize for data latency, but in practice, latency should be one among many factors to be considered. In this session, we will demystify operational complexity of event streaming in the real data engineering world and share best practices learned from developing and maintaining web-scale data systems at Netflix.

• How does a batch architecture differ from a streaming architecture and what does this mean for
  operational support?
• What data processing norms from batch processing no longer apply in a streaming world?
• What should you think about migrating from a batch to a realtime architecture?

————————————————————————————————————————————————————

Valerie Burchby – Senior Data Engineer | Netflix

Valerie is a tech career changer who is passionate about making big data processing more efficient, effective, and approachable. She has over six years of industry experience wrangling data pipelines from small scale startups to some of Netflix’s largest data sets. On Netflix’s new Games vertical, she currently owns the telemetry flow for mobile gaming. Before working on games data, she worked across batch, microbatch and realtime data systems at the PB scale on Netflix’s Streaming team. She loves to learn new technologies and share the many hard earned lessons of processing some of the largest datasets on planet earth.

Xinran Waibel – Senior Data Engineer | Netflix

Xinran is a passionate and self-motivated Software Engineer with a specialization in Data Engineering. She has 6 years of industry experience in designing and developing data systems. Currently a member of the Personalization Data Engineering team at Netflix, she builds data applications to power ML algorithms and product innovation. Prior to Netflix, Xinran was a Data Engineer at Confluent (founded by creators of Apache Kakfa) and Target where she leveraged big data technologies to enable data-driven decision making in the marketing and membership space. She is also a writer for Towards Data Science on Medium, KDnuggets, and DZone.