[Enhancement]: Optimize SQL Queries in SQLAlchemy to Eliminate N+1 Problem

[aybruhm]

Description

This PR refactors SQL queries in the backend to enhance performance by resolving the N+1 query problem. The updates involve selectively loading only necessary fields and utilizing join-loading for related tables to reduce excessive queries.

Related Issue

Closes AGE-430

Changes

• Optimized SQL queries in SQLAlchemy to load only essential fields.
• Implemented join-loading for related tables to resolve the N+1 query problem.
• Updated relevant backend functions to leverage the optimized queries.

What to QA

• Evaluation Run:
  • Run an evaluation and ensure it completes successfully without triggering additional N+1 issues in Sentry.
  • Verify that evaluation results remain accurate and consistent across multiple runs.

Acceptance Tests

1. Create Apps:
   • Create an app using the CLI AND from the UI.
2. Create an App Variant from the Default Variant:
   • Create a new variant derived from the default variant.
3. Run Automatic Evaluations:
   • Perform at least four (4) automatic evaluations using the created app.
4. View and Compare Evaluation Results:
   • After running evaluations, review and compare the results across different runs.
5. Delete Apps:
   • Delete the apps after completing the evaluations.

Note: The N+1 query issue is a common performance bottleneck in SQL databases. This PR addresses the identified N+1 issue observed during evaluations. If Sentry reports additional N+1 issues, please tag me in the notification.

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
agenta	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Aug 23, 2024 4:07pm
agenta-documentation	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Aug 23, 2024 4:07pm

[jp-agenta]

See non-blocking comments.

[mmabrouk]

Thanks for the PR @aybruhm and for the review @jp-agenta

Just wanted to check, has this been throughly QA'ed, i.e. all the modified endpoints / actions have been smoke tested

[jp-agenta]

Thanks @aybruhm !
More than code review, this one requires proper (automated) QA -- @aybruhm @mmabrouk @aakrem

[aybruhm]

QA Report:

https://agenta-hq.slack.com/archives/C05JT3NQB1R/p1724833122715339?thread_ts=1724757298.386189&cid=C05JT3NQB1R

[zenUnicorn]

QA done and everything works fine!