Skip to content
This repository has been archived by the owner on Nov 23, 2023. It is now read-only.

Contains all material that students require to complete the predict for the Moving Big Data Sprint

Notifications You must be signed in to change notification settings

Explore-AI/moving-big-data-predict

Repository files navigation

Data Engineering – Moving Big Data Predict

© Explore Data Science Academy

Predict overview

Figure 1: Completed data pipeline
Figure 1: A representation of the completed pipeline that you will need to implement within this predict.

Data plays an extremely important role in the study of publicly listed companies and stock market analyses. You're part of a small team tasked with migrating an on-premise application to the AWS cloud. This application shows valuable information about the share prices of thousands of companies to a group of professional stock market traders. The data is further used by data scientists to gain insights into the performance of various companies and predict share price changes that are fed back to the stock market traders.

As part of this migration, raw data, given in the form of company csv snapshot batches gathered by the on-premise application, needs to be moved to the AWS cloud. As an initial effort in this movement of data, the stock traders have homed in on the trading data of 1000 companies over the past ten years, and will only require a subset of the raw data to be migrated to the cloud in a batch-wise manner.

With the above requirements, your role as a data engineer is to create a robust data pipeline that can extract, transform, and load the csv data from the source data system to a SQL-based database within AWS. The final pipeline (represented in Figure 1) needs to be built in AWS Data Pipeline and should utilise AWS services as its functional components.

Instructions 🧑‍🏫

A detailed set of instructions guiding you through the creation of the aforementioned data pipeline can be found here.

Resources 📕

The resources below are provided in order to help complete the predict.

Code:

Assessments ⏱

The following actions need to be taken in order to complete the assessments used within the predict:

  • Create the python processing script from the given notebook
  • Create an AWS Data Pipeline and generate failure/success SNS email alerts
  • Sumbit a CSV containing your student and AWS resource details as described in the instructions
  • Complete Data Pipeline MCQ
  • Complete Python Processing MCQ

FAQ ❓

This section of the repo will be periodically updated to represent common questions which may arise around its use. If you detect any problems/bugs, please create an issue and we will do our best to resolve it as quickly as possible.

We wish you all the best in your learning experience 🚀!

EDSA-logo

About

Contains all material that students require to complete the predict for the Moving Big Data Sprint

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published