Building a Data Lakehouse for Analyzing Elon Musk Tweets using MinIO, Apache Airflow, Apache Drill and Apache Superset

Table of Contents

• About
• Getting Started
• Usage
• Resources
• etc.

About

A Data Project using Apache-Airflow (Orchestrator) and MinIO(Object Storage like s3), Apache-Drill (SQL Query Engine) and Apache-Superset (Visualization).

Getting Started

Below is a design of the project. (The drawio file can be found in docs/architecture.drawio:

Prerequisites

• docker-compose
• How to generate Twitter API Token

Usage

1. We need to create and .env from sample.env:

cp sample.env .env

2. Add the Twitter Bearer Token in the .env file as below:

TWITTER_BEARER_TOKEN="TOKEN-GOES-HERE"

3. 1 - If you want to change the MINIO_BUCKET_NAME, you have to change in the conf/drill/storage-plugins-override.conf:

connection: "s3a://twitter-data"

3. 2 - If you want to change the MINIO_ROOT_USER and MINIO_ROOT_PASSWORD, you have to change in the conf/drill/core-site.xml:

<property>
    <name>fs.s3a.access.key</name>
    <value>minioadmin</value>
</property>

<property>
    <name>fs.s3a.secret.key</name>
    <value>minioadmin</value>
</property>

4. We can simply run the pipeline using docker-compose:

• If you are using linux, you need to change the permission of apps/logs/ to allow the container to write to this directory:

chmod -R 777 app/logs superset.db

• To START

docker compose up -d

• To SHUTDOWN

docker compose down

5. Then we can connect to each below respectively:

• Apache-Airflow: http://localhost:8080
• MinIO Console: http://localhost:9090
• Apache-Drill: http://localhost:8047
• Apache-Superset: http://localhost:8088

Resources

• How to generate Twitter API Token
• Apache-Airflow
• Apache-Superset
• Apache-Drill
• MinIO
• docker-compose

License

This project is licensed under the MIT License - see the LICENSE file for details.

Credits

See as you fit.

Contact

If you have any questions or would like to get in touch, you can email: mailto:[email protected] OR twitter