Merge pull request #48 from Toni-SM/develop

Develop

.github/workflows/pre-commit.yml

@@ -0,0 +1,24 @@
+name: pre-commit
+
+on: [ push, pull_request ]
+
+jobs:
+
+  pre-commit:
+    runs-on: ubuntu-latest
+
+    steps:
+      - uses: actions/checkout@v3
+
+      - name: Set up Python
+        uses: actions/setup-python@v3
+        with:
+          python-version: '3.8'
+
+      - name: Install dependencies
+        run: |
+          python -m pip install pre-commit
+
+      - name: Run pre-commit
+        run: |
+          pre-commit run --all-files

.github/workflows/python-publish-manual.yml

@@ -1,4 +1,4 @@
-name: Upload Python Package (manually triggered workflow)
+name: pypi (manually triggered workflow)
 
 on:
   workflow_dispatch:
@@ -12,22 +12,28 @@ permissions:
   contents: read
 
 jobs:
+
   pypi:
     name: Publish package to PyPI
     runs-on: ubuntu-latest
     if: ${{ github.event.inputs.job == 'pypi'}}
+
     steps:
     - uses: actions/checkout@v3
+
     - name: Set up Python
       uses: actions/setup-python@v3
       with:
         python-version: '3.7'
+
     - name: Install dependencies
       run: |
         python -m pip install --upgrade pip
         pip install build
+
     - name: Build package
       run: python -m build
+
     - name: Publish package to PyPI
       uses: pypa/gh-action-pypi-publish@release/v1
       with:
@@ -39,18 +45,23 @@ jobs:
     name: Publish package to TestPyPI
     runs-on: ubuntu-latest
     if: ${{ github.event.inputs.job == 'test-pypi'}}
+
     steps:
     - uses: actions/checkout@v3
+
     - name: Set up Python
       uses: actions/setup-python@v3
       with:
         python-version: '3.7'
+
     - name: Install dependencies
       run: |
         python -m pip install --upgrade pip
         pip install build
+
     - name: Build package
       run: python -m build
+
     - name: Publish package to TestPyPI
       uses: pypa/gh-action-pypi-publish@release/v1
       with:

.github/workflows/python-test.yml

@@ -0,0 +1,32 @@
+name: tests
+
+on: [ push, pull_request ]
+
+jobs:
+
+  tests:
+    strategy:
+      fail-fast: false
+      matrix:
+        os: ["ubuntu-latest", "macos-latest", "windows-latest"]
+        python-version: ["3.7", "3.8", "3.9", "3.10"]
+
+    runs-on: ${{ matrix.os }}
+
+    steps:
+    - uses: actions/checkout@v3
+
+    - name: Set up Python ${{ matrix.python-version }}
+      uses: actions/setup-python@v3
+      with:
+        python-version: ${{ matrix.python-version }}
+
+    - name: Install dependencies
+      run: |
+        python -m pip install --upgrade pip
+        python -m pip install flake8 pytest hypothesis
+        python -m pip install -e .
+
+    - name: Tests
+      run: |
+        pytest --ignore-glob "tests/test_examples*"

.gitignore

@@ -2,11 +2,18 @@
 # Created by https://www.toptal.com/developers/gitignore/api/python,c,c++,visualstudiocode,cmake
 # Edit at https://www.toptal.com/developers/gitignore?templates=python,c,c++,visualstudiocode,cmake
 
+### Other files ###
+*.xml
+*.mp4
+*.zip
+*_old.*
+
 ### Library experiments ###
 *.pt
 *.npz
 *.npy
 *.csv
+*.onnx
 events.out.tfevents.*
 runs
 

.pre-commit-config.yaml

@@ -0,0 +1,11 @@
+repos:
+- repo: https://github.com/pre-commit/pre-commit-hooks
+  rev: v2.3.0
+  hooks:
+  - id: check-ast
+  - id: check-case-conflict
+  - id: check-docstring-first
+  - id: check-merge-conflict
+  - id: check-yaml
+  - id: end-of-file-fixer
+  - id: trailing-whitespace

.readthedocs.yaml

@@ -23,4 +23,4 @@ build:
 # Python requirements required to build your docs
 python:
    install:
-   - requirements: docs/requirements.txt
+   - requirements: docs/requirements.txt

CHANGELOG.md

@@ -2,14 +2,43 @@
 
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
 
+## [0.9.0] - 2023-01-13
+### Added
+- Support for Farama Gymnasium interface
+- Wrapper for robosuite environments
+- Weights & Biases integration (by @juhannc)
+- Set the running mode (training or evaluation) of the agents
+- Allow clipping the gradient norm for DDPG, TD3 and SAC agents
+- Initialize model biases
+- Add RNN (RNN, LSTM, GRU and any other variant) support for A2C, DDPG, PPO, SAC, TD3 and TRPO agents
+- Allow disabling training/evaluation progressbar
+- Farama Shimmy and robosuite examples
+- KUKA LBR iiwa real-world example
+
+### Changed
+- Forward model inputs as a Python dictionary [**breaking change**]
+- Returns a Python dictionary with extra output values in model calls [**breaking change**]
+- Adopt the implementation of `terminated` and `truncated` over `done` for all environments
+
+### Fixed
+- Omniverse Isaac Gym simulation speed for the Franka Emika real-world example
+- Call agents' method `record_transition` instead of parent method
+to allow storing samples in memories during evaluation
+- Move TRPO policy optimization out of the value optimization loop
+- Access to the categorical model distribution
+- Call reset only once for Gym/Gymnasium vectorized environments
+
+### Removed
+- Deprecated method `start` in trainers
+
 ## [0.8.0] - 2022-10-03
 ### Added
 - AMP agent for physics-based character animation
 - Manual trainer
 - Gaussian model mixin
 - Support for creating shared models
 - Parameter `role` to model methods
-- Wrapper compatibility with the new OpenAI Gym environment API (by @JohannLange)
+- Wrapper compatibility with the new OpenAI Gym environment API (by @juhannc)
 - Internal library colored logger
 - Migrate checkpoints/models from other RL libraries to skrl models/agents
 - Configuration parameter `store_separately` to agent configuration dict
@@ -22,7 +51,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
 - Models implementation as Python mixin [**breaking change**]
 - Multivariate Gaussian model (`GaussianModel` until 0.7.0) to `MultivariateGaussianMixin`
 - Trainer's `cfg` parameter position and default values
-- Show training/evaluation display progress using `tqdm` (by @JohannLange)
+- Show training/evaluation display progress using `tqdm` (by @juhannc)
 - Update Isaac Gym and Omniverse Isaac Gym examples
 
 ### Fixed
@@ -100,7 +129,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).
 - More examples and contents in the documentation
 
 ### Fixed
-- Clip actions using the whole space's limits 
+- Clip actions using the whole space's limits
 
 ## [0.2.0] - 2022-01-18
 ### Added

CONTRIBUTING.md

@@ -18,7 +18,7 @@ If you have a question, please do not open an issue for this. Instead, use the f
 Open an issue on [skrl's GitHub issues](https://github.com/Toni-SM/skrl/issues) and describe the bug. If possible, please provide some of the following items:
 
 - Minimum code that reproduces the bug...
-- or the exact steps to reproduce it 
+- or the exact steps to reproduce it
 - The error log or a screenshot of it
 - A link to the source code of the library that you are using (some problems may be due to the use of older versions. If possible, always use the latest version)
 - Any other information that you think may be useful or help to reproduce/describe the problem
@@ -31,7 +31,9 @@ There is a [board](https://github.com/users/Toni-SM/projects/2/views/8) containi
 
 - Try to **communicate your change first** to [discuss](https://github.com/Toni-SM/skrl/discussions) the implementation if you want to add a new feature or change an existing one
 - Modify only the minimum amount of code required and the files needed to make the change
+- Use the provided [pre-commit](https://pre-commit.com/) hooks to format the code. Install it by running `pre-commit install` in the root of the repository, running it periodically using `pre-commit run --all` helps reducing commit errors
 - Changes that are cosmetic in nature (code formatting, removing whitespace, etc.) or that correct grammatical, spelling or typo errors, and that do not add anything substantial to the functionality of the library will generally not be accepted as a pull request
+  - The only exception are changes that results from the use of the pre-commit hooks
 
 #### Coding conventions
 
@@ -51,7 +53,7 @@ Read the code a little bit and you will understand it at first glance... Also
 
   ```ini
   function annotation (e.g. typing)
-  # insert an empty line 
+  # insert an empty line
   python libraries and other libraries (e.g. gym, numpy, time, etc.)
   # insert an empty line
   machine learning framework modules (e.g. torch, torch.nn)
@@ -63,4 +65,4 @@ Read the code a little bit and you will understand it at first glance... Also
 
 Thank you once again,
 
-Toni
+Toni

README.md

@@ -1,10 +1,18 @@
+[![license](https://img.shields.io/pypi/l/skrl)](https://github.com/Toni-SM/skrl)
+[![docs](https://readthedocs.org/projects/skrl/badge/?version=latest)](https://skrl.readthedocs.io/en/latest/?badge=latest)
+[![pypi](https://img.shields.io/pypi/v/skrl)](https://pypi.org/project/skrl)
+<span>&nbsp;&nbsp;</span>
+[![pytest](https://github.com/Toni-SM/skrl/actions/workflows/python-test.yml/badge.svg)](https://github.com/Toni-SM/skrl/actions/workflows/python-test.yml)
+[![pre-commit](https://github.com/Toni-SM/skrl/actions/workflows/pre-commit.yml/badge.svg)](https://github.com/Toni-SM/skrl/actions/workflows/pre-commit.yml)
+
+<br>
 <p align="center">
   <img width="300rem" src="https://raw.githubusercontent.com/Toni-SM/skrl/main/docs/source/_static/data/skrl-up-transparent.png">
 </p>
 <h2 align="center" style="border-bottom: 0 !important;">SKRL - Reinforcement Learning library</h2>
 <br>
 
-**skrl** is an open-source modular library for Reinforcement Learning written in Python (using [PyTorch](https://pytorch.org/)) and designed with a focus on readability, simplicity, and transparency of algorithm implementation. In addition to supporting the [OpenAI Gym](https://www.gymlibrary.dev) and [DeepMind](https://github.com/deepmind/dm_env) environment interfaces, it allows loading and configuring [NVIDIA Isaac Gym](https://developer.nvidia.com/isaac-gym/) and [NVIDIA Omniverse Isaac Gym](https://docs.omniverse.nvidia.com/app_isaacsim/app_isaacsim/tutorial_gym_isaac_gym.html) environments, enabling agents' simultaneous training by scopes (subsets of environments among all available environments), which may or may not share resources, in the same run
+**skrl** is an open-source modular library for Reinforcement Learning written in Python (using [PyTorch](https://pytorch.org/)) and designed with a focus on readability, simplicity, and transparency of algorithm implementation. In addition to supporting the OpenAI [Gym](https://www.gymlibrary.dev) / Farama [Gymnasium](https://gymnasium.farama.org) and [DeepMind](https://github.com/deepmind/dm_env) environment interfaces, it allows loading and configuring [NVIDIA Isaac Gym](https://developer.nvidia.com/isaac-gym/) and [NVIDIA Omniverse Isaac Gym](https://docs.omniverse.nvidia.com/app_isaacsim/app_isaacsim/tutorial_gym_isaac_gym.html) environments, enabling agents' simultaneous training by scopes (subsets of environments among all available environments), which may or may not share resources, in the same run
 
 <br>
 

docs/README.md

@@ -1,6 +1,6 @@
-## Documentation
+# Documentation
 
-### Install Sphinx and Read the Docs Sphinx Theme
+## Install Sphinx and Read the Docs Sphinx Theme
 
 ```bash
 pip install sphinx
@@ -9,7 +9,7 @@ pip install sphinx-autobuild
 pip install sphinx-tabs==3.2.0
 ```
 
-### Building the documentation
+## Building the documentation
 
 ```bash
 cd docs

docs/requirements.txt

@@ -3,6 +3,7 @@ sphinx_rtd_theme
 sphinx-autobuild
 sphinx-tabs==3.2.0
 gym
+gymnasium
 torch
 tensorboard
 tqdm

docs/source/_static/css/s5defs-roles.css

@@ -81,4 +81,4 @@
 
 .purple {
     color: purple;
-}
+}