Moonshot Integration Automation Test

Overview

This repository contains integration tests for Project Moonshot, a toolkit designed to evaluate and red-team Large Language Model (LLM) applications. The integration tests ensure that various components of Moonshot work seamlessly together, maintaining the toolkit's reliability and performance.

Prerequisites

Before running the integration tests, ensure you have the following installed:

• Python: Version 3.11 or later
• Node.js: Version 20.11.1 LTS or later (if testing the Web UI)
• Git: For version control
• Moonshot: Moonshot Setup consist of Moonshot , Moonshot-Data & Moonshot UI

Installation

Clone the Repository:

git clone https://github.com/aiverify-foundation/moonshot-integration-testing.git
cd moonshot-integration-testing

Running Moonshot v0.6 CLI Integration Tests

*Make sure Moonshot is already Setup

To execute the integration tests:

1. Navigate to the Test Directory:

   cd cli-integration-testing

2. Set Up the Virtual Environment:

   It's recommended to use a virtual environment to manage dependencies:

   python -m venv venv
   source venv/bin/activate  # On Windows, use `venv\Scripts\activate`

3. Install Python Dependencies:

   Install the required Python packages:

   pip install -r requirements.txt

4. Setup Env File:

   Create a .env file:

   touch .env

   Open the .env file in a text editor and define your environment variables:

   # .env
   AZURE_OPENAI_URI =
   AZURE_OPENAI_TOKEN =
   URI2 = 
   TOKEN2 = 
   ADDITIONAL_PARAMETERS = '{"timeout": 300, "max_attempts": 3,"temperature": 0.5 }'
   TOGETHER_TOKEN = 
   OPENAI_TOKEN = 
   AWS_ACCESS_KEY_ID=""
   AWS_SECRET_ACCESS_KEY=""
   CLI_DIR = ''# Path of Moonshot Library

5. Run Tests:

   Use the following command to run all tests:

   pytest

   For more detailed output:

   pytest -v

   To run a specific test module:

   pytest test_module.py

Running Moonshot v0.6 UI Integration Tests

To execute the integration tests:

1. Navigate to the Test Directory:

   cd ui-integration-testing

2. Set Up the Virtual Environment:

   It's recommended to use a virtual environment to manage dependencies:

   python -m venv venv
   source venv/bin/activate  # On Windows, use `venv\Scripts\activate`

3. Install Node Dependencies:

   Install the required Node packages:

   npm install

4. Setup Env File:

   Create a .env file:

   touch .env

   Open the .env file in a text editor and define your environment variables:

   # .env
   URI =
   TOKEN =
   URI2 =
   TOKEN2 =
   ADDITIONAL_PARAMETERS = '{"timeout": 300, "max_attempts": 3,"temperature": 0.5 }'
   TOGETHER_TOKEN =
   OPENAI_TOKEN = 
   AWS_ACCESS_KEY_ID=""
   AWS_SECRET_ACCESS_KEY=""
   GOOGLE_TOKEN = 
   H2OGPT_TOKEN = ""
   ANTHROPIC_TOKEN = ""
   HUGGINGFACE_TOKEN = ""

5. Run Tests:

   Use the following command to run all tests:

   npx playwright test tests

   To run a specific test module:

   npx playwright test tests/test_module.spec.ts

Running Moonshot V1 CLI Integration Tests

*Make sure Moonshot v1 is already Setup

To execute the integration tests:

1. Navigate to the Test Directory:

   cd moonshot-v1-cli-integration

2. Set Up the Virtual Environment:

   It's recommended to use a virtual environment to manage dependencies:

   python -m venv venv
   source venv/bin/activate  # On Windows, use `venv\Scripts\activate`

3. Install Python Dependencies:

   Install the required Python packages:

   pip install -r requirements.txt

4. Setup Env File:

   Create a .env file:

   touch .env

   Open the .env file in a text editor and define your environment variables:

   # .env
   AWS_ACCESS_KEY_ID = ''
   AWS_SECRET_ACCESS_KEY=''
   OPENAI_TOKEN = ''
   MOON_V1_CLI_DIR = ''# Path of Moonshot v1 repo

5. Run Tests:

   Use the following command to run all tests:

   pytest

   For more detailed output:

   pytest -v

   To run a specific test module:

   pytest test_module.py

Running Moonshot V1 Process Check Integration Tests

To execute the integration tests:

1. Navigate to the Test Directory:

   cd ui-integration-testing

2. Set Up the Virtual Environment:

   It's recommended to use a virtual environment to manage dependencies:

   python -m venv venv
   source venv/bin/activate  # On Windows, use `venv\Scripts\activate`

3. Install Node Dependencies:

   Install the required Node packages:

   npm install

4. Run Tests:

   Use the following command to run all tests:

   npx playwright test tests

   To run a specific test module:

   npx playwright test tests/test_module.spec.ts

Directory Structure

A brief overview of the repository structure:

moonshot-integration-testing/         
├── cli-integration-testing/          # Moonshot v0 CLI Integration test cases       
│   ├── utils                         # Common Utils Functions to support automation on Data Preparation
│   ├── test_api.py                   # Tests for the Moonshot APIs
│   ├── test_benchmark.py             # Tests for the CLI Command - Moonshot v0 Benchmarking CLI Commands Scope
│   ├── test_common.py                # Tests for the CLI Command - Moonshot v0 Common CLI Commands Scope
│   ├── test_red_team.py              # Tests for the CLI Command - Moonshot v0 Red Teaming CLI Commands Scope
│   └──
├── moonshot-v1-cli-integration/      # Moonshot v1 CLI Integration test cases
│   ├── tests/
│   ├── test_benchmark.py             # Tests for the CLI Command - Moonshot v1 Benchmarking CLI Commands Scope
│   ├── test_red_team.py              # Tests for the CLI Command - Moonshot v1 Red Teaming CLI Commands Scope
│   ├── test_s3_connection.py         # Tests for S3 Connection for Moonshot v1
│   └──
├── moonshot-v1-process-checks-integration-testing/      # Moonshot v1 Process Check Integration test cases
│   ├── tests/
│   ├── entrypoints.spec.ts           # Tests for the Web UI - Moonshot v1 Process Check Scope
│   └──
├── ui-integration-testing/           # Moonshot v0 UI Integration test cases
│   ├── tests/
│   ├── ├── benchmarking.spec.ts      # Tests for the Web UI - Moonshot v0 Benchmarking Scope
│   ├── ├── endpoint.spec.ts          # Tests for the Web UI - Moonshot v0 Endpoint Scope
│   ├── ├── homepage.spec.ts          # Tests for the Web UI - Moonshot v0 Homepage Scope
│   ├── ├── red_teaming.spec.ts       # Tests for the Web UI - Moonshot v0 Red Teaming Scope
│   ├── ├── history.spec.ts           # Tests for the Web UI - Moonshot v0 History Tab Scope
│   ├── ├── utils.spec.ts             # Tests for the Web UI - Moonshot v0 Utils Tav Scope
│   └──
├── util/                            # Common Utils Functions to support automation on Data Preparation
│   ├── utils.py
│   └──
├── .gitignore
├── README.md                         # Project documentation
└── LICENSE                           # License information

Contributing

We welcome contributions to enhance the integration tests for Moonshot. To contribute:

1. Fork the Repository: Click the "Fork" button at the top right of this page.
2. Create a New Branch: Use a descriptive name for your branch.
3. Make Your Changes: Implement your improvements or fixes.
4. Submit a Pull Request: Provide a clear description of your changes.

Please ensure that your contributions align with the project's coding standards and pass all existing tests.

License

This project is licensed under the Apache-2.0 License. See the LICENSE file for details.