Mechanics of Data Pipelines

Scale
06/13/2017 - 12:20 to 13:00
Kesselhaus
long talk (40 min)
Advanced

Session abstract: 

This talk focused the topic on how to model data pipelines as retroactive, immutable data structures. It covers the topic of how do you build a data pipelines for a growing organization where different teams depend on each others data and need to be able to re-process data when errors occur upstream. I draw comparisons between the microservice architectures for both stream and batch processings and provide some guiding principals towards building resiliant systems based on experience scaling out infrastructure at SoundCloud.

Video: 

Slide: