Update Gym to Gymnasium

[Zach-Attach]

Proposed change(s)

• Updated requirement from gym 0.21 to the latest version of gymnasium
  • version currently not specified but can be for safety
• Updated pettingzoo requirement to be compatible with latest version
  • set to 1.22.0 or greater (versions that use gymnasium)
• Replaced all gym imports to gymnasium
• Updated step and reset functions in UnityToGymWrapper and UnityParallelEnv to align with Gymnasium expectations (see https://github.com/Farama-Foundation/Gymnasium/blob/main/docs/introduction/migration_guide.md for details)
  • UnityAECEnv stays the same
• Updated tests to use gymnasium and expect new step and reset functions
• Updated documentation to no longer refer to OpenAI Gym and now refer to Farama Foundation's Gymnasium

Useful links (Github issues, JIRA tickets, ML-Agents forum threads etc.)

There is 1 open PR and 4 issues related to this addition:

• Existing PR for this lacks many of the necessary changes: Upgraded from gym to Gymnasium #6060
• Issues:
  • Feature Request: migrate from gym to gymnasium #5884
  • Feature request: upgrade to gymnasium/update pettingzoo version #5890
  • gym -> gymnasium #5995
  • UnityToGymWrapper uninstallable in up to date Python environments due to gym dependency #6059

Types of change(s)

• Bug fix
• New feature
• Code refactor
• Breaking change
• Documentation update
• Other (please describe)

Checklist

• Added tests that prove my fix is effective or that my feature works
• Updated the changelog (if applicable)
• Updated the documentation (if applicable)
• Updated the migration guide (if applicable)

Other comments

• I do not use pettingzoo or the premade environments much myself, so there may be additional fixes that are needed.
• I was unsure if I was expected to add to the changelog or migration guide since this is not an actual release, but am happy to contribute.
• This change may result in the possibility of upgrading the maximum version other dependencies (grpcio, protobuf, numpy) as well, potentially adding support for python 3.11 or newer.

[CLAassistant]

All committers have signed the CLA.

[paulo101977]

I looked at the code. Everything is ok to release, I don't understand why they didn't approve it...

[Zach-Attach]

Not sure. I also have code to update PettingZoo to a version working with Gymnasium here but given that it includes other fixes to Zoo and the silence on this PR, I haven't tried integrating in here.

[Zach-Attach]

@maryamziaa @alexander-suvorov I decided to add the anticipated pettingzoo changes to this PR since it hasn't gotten much attention. This should resolve any issues that might directly result from use of this code.

In addition, there are also issues with uint8 visuals but I have withheld those here as they are a really more of a separate issue. Please let me know if there's anything I can do to help move this issue forward. I have been using this code across single vector, multi vector, and pettingzoo environments without issue for nearly a year now without issue (aside from uint8 visuals which would need a separate PR).

[maryamziaa]

@maryamziaa @alexander-suvorov I decided to add the anticipated pettingzoo changes to this PR since it hasn't gotten much attention. This should resolve any issues that might directly result from use of this code.

In addition, there are also issues with uint8 visuals but I have withheld those here as they are a really more of a separate issue. Please let me know if there's anything I can do to help move this issue forward. I have been using this code across single vector, multi vector, and pettingzoo environments without issue for nearly a year now without issue (aside from uint8 visuals which would need a separate PR).

Hey, thanks for putting this PR together! Unfortunately, we haven’t been able to review PRs or community contributions lately since ML-Agents was in maintenance mode. The good news is we’ve started ramping back up and getting things back up to speed again. Alex and I will take a look at your PR soon, but it might not be right away, sorry! But fear not, it’s definitely on my list.

[Zach-Attach]

@maryamziaa Thanks for the update. Glad to hear! The lack of gymnasium support in the official pip package has been a major roadblock in use since git branches as dependencies are not allowed in published pip packages.
Hope to hear more soon.