Skip to content

Use AWS Lambda to Pull E-Scooter and E-Bike Location Data, store in S3 & Redshift using Data Vault Data Model, Server to Google Data Studio Dashboard

Notifications You must be signed in to change notification settings

jaredfiacco2/AWS-E-Scooter-Tracker

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

14 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

LinkedIn


Logo

Use AWS Lambda to pull E-Scooter/Bike Location Data, store in Redshift Data Vault. Serve to Google Data Studio Dashboard.

This code takes a list of E-Scooter APIs stored in a JSON file and uses AWS Cloudwatch & SNS to run Python in Lambda to GET them, store them in AWS in a S3 Data Lake and a Redshift Data Warehouse using a Data Vault Data Model. Then it serves the data in a Google Data Studio Dashboard.

Table of Contents
  1. About The Project
  2. Prerequisites/Instructions
  3. Contact

About The Project

  • Personal mobility devices are changing transportation in major cities. This data is publically available from various companies via APIs if you spend the time and money to collect it. This project collects and tracks scooter data from companies over time using AWS and implementing Data Vault data modeling.

  • I take various E-Scooter Scooter APIs, use AWS to pull and store data (Cloud Watch, SNS, Lambda, S3, Redshift) and serve a dashboard using Google Data Studio.

Process Map

Process Map

Built With

Prerequisites

  1. Create a cluster in Redshift open to public IPs & add the schemas, tables, views, and stored procedures included in the sql folder. The data model will be a data vault model and store each change in the sattelites.

  1. Create a lambda function with the lambda_function.py file. Swap out the "ACCESS_KEY", "SECRET_KEY", "creds", and "my_bucket" names. Also add Data wrangler and psycopg layers. Trigger it with the sns topic you'll make in the next step.

  1. Create an sns topic.

  2. Create a cloud watch schedule to run every X minutes (you choose X).

  1. Create Data Studio dashboard pulling from Redshift query. Schedule to send to self daily.

Contact

Jared Fiacco - [email protected]

Another GCP Project of Mine: Pull and Store and Server Clash of Clans API Data

A GCP Project of Mine: Publish Computer Statistics to Pub/Sub, Use Cloud Functions to Store in BigQuery

Releases

No releases published

Packages

No packages published