AutoML-Python-Benchmark

Benchmarks of AutoML Frameworks for time series forecasting, anomaly detection and classification.

This requires Docker. Primary Python version: 3.10.14

Table of Contents

1. Publications
2. Installation
3. CUDA
4. Datasets
5. Installation
6. Experiments
7. Development
8. Contact
9. Citation

Publications

• An Empirical Study of Static Code Analysis in Python Automated Machine Learning Libraries (2025) - (awaiting review)

  • Tag
  • Code

• A Comparative Analysis of Automated Machine Learning Libraries for Electricity Price Forecasting (2024)

  • Tag
  • Code
  • These experiments are run with Python 3.9 and CUDA versions 11.2 and 11.7.

Installation

• Step 1: Install conda via Miniconda or Anaconda. Then create environment with:

# Create the environment if it does not exist
conda info --envs | grep automl || conda create -n automl -y python=3.10.14

# Activate environment
conda activate automl

# Install dependencies. Pick at least one:
pip install -e .             # Bare minimum. Includes sklearn.

pip install -e .[lightgbm]   # LightGBM
pip install -e .[tensorflow] # TensorFlow
pip install -e .[torch]      # PyTorch
pip install -e .[xgboost]    # XGBoost
pip install -e .[ai]         # All ML libraries

pip install -e .[tests,docs] # Unit tests, docs
pip install -e .[sca]        # Source Code Analysis

pip install -e .[all]        # Everything

# Optionally, for development:
conda install pre-commit

CUDA

To run code via GPUs, you will need to install CUDA for TensorFlow and PyTorch.

• CUDA compatibilities for TensorFlow are listed here.
• CUDA compatibilities for PyTorch are listed here

Datasets

Removed. To be redrafted.

Experiments

Classification

Experiments use DVC.

conda activate automl
dvc pull
dvc repro

Source Code Analysis

./scripts/clone_or_pull.sh     # Clone repositories
conda activate automl          # Activate environment
pip install -e .[sca]          # Install dependencies
python -m sca.ml repositories  # Analyze repositories

Forecasting

Removed. To be redrafted.

Development

Source Code Analysis of AutoML Repositories with SonarQube

This requires Docker.

Allow Docker containers to access GPUs:

# Required to install nvidia packages
wget https://nvidia.github.io/nvidia-docker/gpgkey --no-check-certificate
sudo apt-key add gpgkey
sudo apt-get update

distribution=$(. /etc/os-release;echo $ID$VERSION_ID) && curl -s -L https://nvidia.github.io/nvidia-docker/gpgkey | sudo apt-key add -
curl -s -L https://nvidia.github.io/nvidia-docker/$distribution/nvidia-docker.list | sudo tee /etc/apt/sources.list.d/nvidia-docker.list
sudo apt-get update
sudo apt-get install -y nvidia-docker2

# Install nvidia package
sudo apt-get install nvidia-container-runtime nvidia-container-toolkit

Set up SonarQube server via docker-compose and run analysis:

# Start server
docker-compose up --timeout 300 -d --build --force-recreate
# Download repositories
sh -i ./scripts/clone_or_pull.sh
# Run sonar-scanner
sh -i ./scripts/sonar_scanner.sh
# Analyze results
source_code_analysis.sh
# Stop server:
docker-compose down

Contact

Please feel free to get in touch at christian.oleary@mtu.ie

Citation

Christian O'Leary (2025) AutoML Python Benchmark.

@software{AutoML-Python-Benchmark,
author = {Christian O'Leary},
title = {AutoML Python Benchmark},
doi = {10.5281/zenodo.13133203},
howpublished = {\url{https://github.com/christian-oleary/AutoML-Python-Benchmark}},
year = {2025}
}