PyConTW ETL

Using Airflow to implement our ETL pipelines.

Table of Contents

• Prerequisites
• Installation
• Configuration
• BigQuery (Optional)
• Running the Project
  • Local Environment with Docker
  • Production
• Contact

Prerequisites

• Python 3.8+
• Docker
• Git
• uv

Installation

We use uv to manage dependencies and virtual environment.

Below are the steps to create a virtual environment using uv:

1. Create a Virtual Environment with Dependencies Installed

   To create a virtual environment, run the following command:

   uv sync

   By default, uv sets up the virtual environment in .venv

2. Activate the Virtual Environment

   After creating the virtual environment, activate it using the following command:

   source .venv/bin/activate

3. Deactivate the Virtual Environment

   When you're done working in the virtual environment, you can deactivate it with:

   deactivate

Configuration

1. For development or testing, run cp .env.template .env.staging. For production, run cp .env.template .env.production.

2. Follow the instructions in .env.<staging|production> and fill in your secrets. If you are running the staging instance for development as a sandbox and do not need to access any specific third-party services, leaving .env.staging as-is should be fine.

Contact the maintainer if you don't have these secrets.

⚠ WARNING: About .env Please do not use the .env file for local development, as it might affect the production tables.

BigQuery (Optional)

Set up the Authentication for GCP: https://googleapis.dev/python/google-api-core/latest/auth.html

• After running gcloud auth application-default login, you will get a credentials.json file located at $HOME/.config/gcloud/application_default_credentials.json. Run export GOOGLE_APPLICATION_CREDENTIALS="/path/to/keyfile.json" if you have it.
• service-account.json: Please contact @david30907d via email or Discord. You do not need this json file if you are running the sandbox staging instance for development.

Running the Project

If you are a developer 👨‍💻, please check the Contributing Guide.

If you are a maintainer 👨‍🔧, please check the Maintenance Guide.

Local Environment with Docker

For development/testing:

# Build the local dev/test image
make build-dev

# Start dev/test services
make deploy-dev

# Stop dev/test services
make down-dev

The difference between production and dev/test compose files is that the dev/test compose file uses a locally built image, while the production compose file uses the image from Docker Hub.

If you are an authorized maintainer, you can pull the image from the GCP Artifact Registry.

Docker client must be configured to use the GCP Artifact Registry.

gcloud auth configure-docker asia-east1-docker.pkg.dev

Then, pull the image:

docker pull asia-east1-docker.pkg.dev/pycontw-225217/data-team/pycon-etl:{tag}

There are several tags available:

• cache: cache the image for faster deployment
• test: for testing purposes, including the test dependencies
• staging: when pushing to the staging environment
• latest: when pushing to the production environment

Production

Please check the Production Deployment Guide.

Contact

PyCon TW Volunteer Data Team - Discord