Add support for PyMongo Async and N+1 Dereference/select_related using Pipeline

[arunsureshkumar]

We are in the process of adding native async support backed by PyMongo’s official async API. This issue tracks the effort to fully support async workflows while modernizing the core architecture, tests, CI, and documentation.

We are actively working on this in a dedicated branch, and several foundational changes are already underway.

Background & Motivation

PyMongo now provides a first-class, officially supported async API (>= 4.14). Leveraging this allows MongoEngine to:

• Support modern async Python frameworks (FastAPI, async workers, background jobs)
• Remove legacy sync implementations and compatibility layers
• Unify synchronous and asynchronous code paths
• Improve performance and long-term maintainability

A key goal is to ensure async support is native, predictable, and efficient, rather than layered on top of synchronous behavior.

Work in Progress

Native Async Support

• Migration to PyMongo async API (>= 4.14)
• Making document and queryset operations awaitable where appropriate
• Ensuring consistent behavior across sync and async execution paths
• Updating internal helpers (e.g., is_primary) for async compatibility

Aggregation & select_related Improvements

• Improving select_related using aggregation pipelines
• Avoiding CPU-bound dereferencing in Python
• Shifting reference resolution to MongoDB where possible
• Preserving explicit missing-reference behavior

This significantly reduces Python-side processing and improves performance for async workloads.

Tests & Registry Handling

• Refactoring tests to properly await async operations
• Clearing the document and collection registries between tests
• Ensuring reliable behavior in both sync and async contexts
• Fixing race conditions in parallel test execution

CI & Tooling

• Updating CI to reflect async-first workflows
• Simplifying MongoDB setup using official images
• Aligning test matrices with supported MongoDB versions
• Migrating tooling to uv for faster and more consistent dependency management

Documentation

• Updating README and changelog to reflect async support
• Clarifying supported MongoDB and PyMongo versions
• Removing references to deprecated behavior and versions

Version & Compatibility Changes

As part of this effort, the following constraints are being applied:

• Minimum PyMongo version: 4.14
• Supported MongoDB versions: ≥ 4.4

Dropped Support

• PyMongo < 4.14
• MongoDB < 4.4
• Legacy sync implementations
• Deprecated lazy reference and cached reference/generic fields

Status

🚧 Actively in progress Please checkout the branch to test and provide feedback on our implementation - https://github.com/strollby/mongoengine/tree/feat/async

This issue serves as a tracking and discussion point while the async migration and related refactors are finalized.