ModiGen

ModiGen is a Large Language Model (LLM)-based workflow for multi-domain Modelica component generation.
It provides an automated pipeline for generating, validating, and evaluating Modelica components.

---

🔧 Pipeline Overview

The pipeline includes the following core components:

• Component Generation (gen_model.py): Generates Modelica components based on task-specific prompts and domain requirements.

• Text Processing (__main__.py): Extracts Modelica source code (.mo files) from raw LLM-generated text outputs.

• Component Validation (__main__.py): Uses OpenModelica (via OMPython) to perform syntax checking and simulation. Functional validation is conducted by comparing simulation trajectories against reference outputs.

• Results Analysis (__main__.py): Aggregates results.

• Metrics Calculation (validation/compute_metrics.py): Evaluates the final performance using the Pass@k metric.

---

🌟 Key Features

• Retrieval-Augmented Generation (RAG): Constructs an index representing the structural information of Modelica components. This improves dependency analysis and retrieval efficiency for complex component generation.

• Feedback-Driven Refinement (feedback): Automatically extracts simulation or syntax errors and generates diagnostic prompts for iterative model correction.

• Fine-Tuning (LLaMA-Factory): Supports supervised fine-tuning (SFT) using the LLaMA-Factory framework to enhance Modelica-specific generation capabilities.

---

📁 Project Structure (Excerpt)

├── gen_model.py # Prompt construction and LLM-based component generation ├── gen_model_quick.py # Lightweight component generation through an OpenAI-compatible API
├── __main__.py # Main pipeline: post-processing, simulation, and results analysis
├── validation/ # Modules for simulation, result checking, and metric computation
│ ├── library-OM/ # Required Modelica dependency libraries for simulation
│ └── reference_model/ # Reference ground-truth data for functional validation
├── json_files/ # Benchmark datasets and metadata in JSON format
├── RAG/ # Index construction and RAG-based generation logic
├── feedback/ # Error extraction and iterative prompt refinement scripts
├── result/ # Output directory for generated code, logs, and evaluation reports
├── LLaMA-Factory/ # Fine-tuning framework and training datasets
├── tools/ # Utility scripts and helper functions for processing
├── README.md # Project documentation and usage guide
└── requirements.txt # Python dependencies for environment setup

---

⚙️ OpenModelica Installation and Library Configuration

Before running the evaluation pipeline, install OpenModelica and ensure that the omc command is available in your system PATH. OpenModelica 1.18.1 was used in our experimental environment.

Please download and install OpenModelica from the OpenModelica download page by following the instructions for your operating system. After installation, verify that OpenModelica is available by running:

omc --version

For this artifact, we recommend installing OpenModelica on Linux or Windows for better compatibility.

The required Modelica Standard Library (MSL) version is loaded by validation/model_test_for_model.py as follows:

load1 = omc.sendExpression(f'loadModel(Modelica, {{"{version}"}})')

OpenModelica automatically locates the requested MSL version. Please ensure that the required MSL versions are available in your OpenModelica installation.

The remaining Modelica dependency libraries are provided under validation/library-OM/ and are loaded automatically by the evaluation script.

---

🚀 Full Pipeline Usage

▶ Install dependencies

pip install -r requirements.txt

Ensure OpenModelica (omc) is installed and available in your system PATH. This artifact was developed and tested with Python 3.10.14.

▶ Prepare model checkpoints and data

The full generation pipeline requires a local LLM, such as Llama-3.1 or Qwen-2.5-Coder, which can be downloaded from the Hugging Face Hub.

Install the Hugging Face Hub command-line tool if it is not already available:

pip install huggingface_hub

For models that require authentication or license acceptance, first request access on the corresponding Hugging Face model page and log in using:

hf auth login

Download the selected model checkpoint to a local directory:

hf download <MODEL_REPOSITORY_ID> --local-dir <LOCAL_MODEL_DIRECTORY>

For example:

hf download Qwen/Qwen2.5-Coder-7B-Instruct --local-dir model_dir/Qwen2.5-Coder-7B-Instruct

After downloading the model, update the local model path in gen_model.py to match <LOCAL_MODEL_DIRECTORY>.

The full local generation pipeline was executed using an NVIDIA A100 GPU. The required disk space, GPU memory, and execution time depend on the selected model size and inference configuration. Please ensure that sufficient storage and GPU resources are available before running the complete generation process.

▶ Generate component

Run prompt-based component generation:

python gen_model.py

▶ Evaluate generated components

Run the evaluation pipeline:

python __main__.py

▶ View results

Evaluation results will be saved under the result/ directory.

---

🚀 Kicking-the-tires

For the artifact evaluation kick-the-tires stage, reviewers can directly evaluate the existing generated results provided under the result/ directory.

▶ Install dependencies

pip install -r requirements.txt

Ensure OpenModelica (omc) is installed and available in your system PATH. This artifact was developed and tested with Python 3.10.14.

▶ Optional: Quickly generate components via LLM API

To quickly exercise the component generation pipeline without downloading and deploying a local LLM, reviewers can use an OpenAI-compatible API.

Before running the script, configure the API key, API endpoint, and model name through environment variables.

On Windows PowerShell:

$env:OPENAI_API_KEY="your-api-key"
$env:OPENAI_BASE_URL="https://your-api-endpoint/v1"
$env:OPENAI_MODEL="your-model-name"

python gen_model_quick.py

On Linux:

export OPENAI_API_KEY="your-api-key"
export OPENAI_BASE_URL="https://your-api-endpoint/v1"
export OPENAI_MODEL="your-model-name"

python gen_model_quick.py

OPENAI_BASE_URL may be omitted when using the default OpenAI API endpoint. The generated component responses will be saved under the corresponding model directory in result/.

▶ Evaluate generated components

Run the evaluation pipeline:

python __main__.py

The generated logs and evaluation results will be saved under the result/ directory.