Skip to content

Add optimizations to figures daily job #131

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 5 commits into from

Conversation

Waleed-Mujahid
Copy link

@Waleed-Mujahid Waleed-Mujahid commented May 8, 2025

Description

This PR is for the optimization of our daily job for updating figures populate_daily_metrics It takes around 11 hours to complete on prod.

This PR:

  • Refactors the code to do bulk update and bulk create on figures_enrollmentdata This reduces queries that were executing on each enrollment separately.
  • It moves update_learners_activity_for_date from inside a loop over courses to outside of the loop which is over sites. This function executes on site level and has no course specific processing. Executing this function for each course was redundant and was slowing down the figures job.

These changes have reduced the jobs time from around 11-13 minutes to 8 to 8.30 minutes. We should see better improvement on prod since it has much more data.

Ticket

https://projects.arbisoft.com/project/edly-product/task/7068

@Waleed-Mujahid Waleed-Mujahid self-assigned this May 8, 2025
@Waleed-Mujahid Waleed-Mujahid changed the title refactor: add optimizations to figures daily job Add optimizations to figures daily job May 12, 2025
@Waleed-Mujahid Waleed-Mujahid force-pushed the waleed/7068-figures-optimizations branch from ed42390 to 8fdf335 Compare May 13, 2025 09:38
@Waleed-Mujahid Waleed-Mujahid marked this pull request as draft May 13, 2025 10:04
@Waleed-Mujahid
Copy link
Author

Waleed-Mujahid commented May 13, 2025

This pull request has been split into two distinct parts for clarity and maintainability. The first part containing only the minimal changes needed to substantially improve the figures job will be merged to develop-koa in PR #132. Because this initial set of changes is both easier to review and less likely to introduce future maintenance challenges, we’ve isolated it here. If, after monitoring performance, we determine that additional optimizations are necessary, we will merge this pull request as well.

@Waleed-Mujahid
Copy link
Author

Closing this PR, as the changes introduced in #132 significantly improved performance, reducing the job time from 11 hours to 4 hours.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant