Session abstract:
Data is becoming one of the main decision-makers in an organisation. The more data we have the more challenges we face every day. Every decision we make will have long-term implications. In the talk we will go through different approaches to the data pipelines: from a simple in-house built, with comparison to open source solutions like Apache Kafka and finally hosted auto scaling solutions based Amazon(S3, Kinesis, Lambda) or Google. The talk covers the main aspects of data collecting processes altogether with further implications for data processing, highlighting appropriate solutions and architectures for the main use-cases.