- There are two main ways of working with data - batch or stream. Neither is better or worse - they're just different tools for different jobs. A data engineer understands the business need and finds the right solution:
We will be helping Cody, the Sustainability Manager for the City of San Diego. As the city's Data Engineer, you will be helping Cody decrease emissions and accidents using the internet of things or IOT. She wants to put a sensor in each vehicle that will transmit emissions, time and speed for that vehicle. This is known as vehicle telematics, a common streaming data use case
-
AWS Kinesis has Data Firehose, Data Streams and Data Analytics as sub-services. You'll be combining these to solve streaming data use cases. Let's start with Firehose.
-
Data Firehose Sub-service has the Firehorse Delivery Stream (FDS) as the key component. The FDS consists of a "Producer" - the data generator of a Firehose stream, and a "Destination" - where the data gets stored:
-
To create a Firehose client:
-
Working with delivery streams:
-
Deleting redundant streams:
