[feature] Projects - Checkpoint 2

[aybruhm]

Description

This PR introduces backward compatibility enhancements to:

• domains are fully scoped by project
• and that these domains (along with their services) receive a full auth_context

Related Issues

Closes:

• AGE-669 - Update scope in the evaluations domain database routes
• AGE-668 - Update scope in the evaluators domain database routes
• AGE-667 - Update scope in the applications domain database routes
• AGE-666 - Update scope in the testsets domain database routes

Sibling PRs

• Cloud
• Commons

Changes

• Project-Specific Scoping

  • Scoped project_id to database entities including testsets, applications, evaluators, evaluations models and beyond.

• Database and Migration Adjustments

  • Added migration logic to scope project_id in DB entities, add default project to scoped DB entities, update evaluators with app names, etc.
  • Moved changed models to a deprecated file and make use of it when updating evaluators with app names to ensure accurate table schema.

• Utility Functions and Middleware

  • Implemented project utility function to get project_id from request ; path parameters, query parameters or request body.
  • Improved authentication middleware to retrieve and set project_id to request state.

• Other Changes

  • Cleaned up route endpoints to use project_id from request.state.
  • Added AppDB and EvaluatorConfigDB to deprecated models file.
  • etc

What to QA

• Create an app from the UI & CLI
• Create a new variant based on an existing one
• Save variant configurations
• Add outputs to the testset directly from the playground
• Create a new testset directly from the playground
• Deploy a variant to an environment from the playground
• Delete a variant from the playground
• Update a testset from the testset view
• List testsets from the testset view
• View a specific testset from the testset view
• View published deployments in the deployments (overview) view
• Create an evaluator configuration
• Delete an app
• Upload a test set from the test set view
• Import a test set from the test set view
• Delete test set(s) from the test set view
• Run Automatic Evaluation
• View automatic evaluation results/scenarios
• Compare automatic evaluation results
• Run Human Evaluation
  • A/B Testing
  • Single Model Testing
• View a single model evaluation (along with its scenarios)
• Delete human and automatic evaluations

Acceptance Tests

Ensure that the above items work without triggering any 500 errors from the backend.

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
agenta	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Sep 17, 2024 7:57am
agenta-documentation	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Sep 17, 2024 7:57am

[jp-agenta]

@aybruhm -- I have a few comments, things I want to discuss with you before I can approve, but... man, this PR is amazing ! Great job, for real !

[jp-agenta]

Awesome @aybruhm ! lgtm.