Added experimental background job queue

[9larsons]

ref https://linear.app/tryghost/issue/ENG-1556/

• added background job queue behind config flags
• when enabled, is only used for the member email analytics updates in order to speed up the parent job, and take load off of the main process that is serving requests

[github-actions]

It looks like this PR contains a migration 👀
Here's the checklist for reviewing migrations:

General requirements

• Satisfies idempotency requirement (both up() and down())
• Does not reference models
• Filename is in the correct format (and correctly ordered)
• Targets the next minor version
• All code paths have appropriate log messages
• Uses the correct utils
• Contains a minimal changeset
• Does not mix DDL/DML operations
• Tested in MySQL and SQLite

Schema changes

• Both schema change and related migration have been implemented
• For index changes: has been performance tested for large tables
• For new tables/columns: fields use the appropriate predefined field lengths
• For new tables/columns: field names follow the appropriate conventions
• Does not drop a non-alpha table outside of a major version

Data changes

• Mass updates/inserts are batched appropriately
• Does not loop over large tables/datasets
• Defends against missing or invalid data
• For settings updates: follows the appropriate guidelines

[9larsons]

Need to add a couple integration tests then we ought to be ok. Not sure what the failure for Ghost-CLI is.

[vikaspotluri123]

It looks like there's an error that's preventing the persisted logs from being printed 🙃 Here's the culprint:

Ghost was able to start, but errored during boot with: Cannot find module 'workerpool'

Looking at the lockfile changes, I'm not sure if they're intentional - there are a lot of new dependencies, which I think might stem from an older version of @tryghost/errors
Also, the real issue is that it looks like workerpool wasn't added as a dependency?

[9larsons]

Ok, clearly there's some kind of race condition with the config mocks and initializing the service. I will have to look into that when I can reprioritize this.

[9larsons]

@vikaspotluri123 Hey, it seems like the ghost-cli tests are hanging on restart with this. Do you have any suggestions on how to repro/troubleshoot that? Only thing I can guess is that the job queue service is hanging. It doesn't do that to me locally.

[9larsons]

Note to self: need to update migration to new version before merging.

[vikaspotluri123]

It took me a minute, but taking a look 👀

https://github.com/vikaspotluri123/Ghost/actions/runs/11393243539/job/31701108553?pr=14

[vikaspotluri123]

diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
index eec2c09aeda..d68f13ce3a2 100644
--- a/.github/workflows/ci.yml
+++ b/.github/workflows/ci.yml
@@ -861,7 +861,8 @@ jobs:
 
       - name: Update from v4
         run: |
-          ghost update -f -d $V4_DIR --archive $(pwd)/ghost/core/ghost.tgz
+          ghost update -f -d $V4_DIR --archive $(pwd)/ghost/core/ghost.tgz --no-restart
+          ghost run -d $V4_DIR -D
 
       - name: Save Ghost CLI Debug Logs
         if: failure()
@@ -879,7 +880,8 @@ jobs:
         run: |
           DIR=$(mktemp -d)
           ghost install local -d $DIR
-          ghost update -d $DIR --archive $(pwd)/ghost/core/ghost.tgz
+          ghost update -d $DIR --archive $(pwd)/ghost/core/ghost.tgz --no-restart
+          ghost run -d $DIR -D
 
       - name: Print debug logs
         if: failure()

I hacked the job like that and found this

[2024-10-18 00:32:03] ERROR Unhandled rejection: insert into `jobs` (`created_at`, `finished_at`, `id`, `metadata`, `name`, `queue_entry`, `started_at`, `status`, `updated_at`) values ('2024-10-18 00:32:03', NULL, '6711ac8389d4880ed0fcad4e', NULL, 'members-migrations', NULL, NULL, 'queued', '2024-10-18 00:32:03') - SQLITE_ERROR: table jobs has no column named metadata

insert into `jobs` (`created_at`, `finished_at`, `id`, `metadata`, `name`, `queue_entry`, `started_at`, `status`, `updated_at`) values ('2024-10-18 00:32:03', NULL, '6711ac8389d4880ed0fcad4e', NULL, 'members-migrations', NULL, NULL, 'queued', '2024-10-18 00:32:03') - SQLITE_ERROR: table jobs has no column named metadata
Error Code: 
    SQLITE_ERROR
----------------------------------------
Error: insert into `jobs` (`created_at`, `finished_at`, `id`, `metadata`, `name`, `queue_entry`, `started_at`, `status`, `updated_at`) values ('2024-10-18 00:32:03', NULL, '6711ac8389d4880ed0fcad4e', NULL, 'members-migrations', NULL, NULL, 'queued', '2024-10-18 00:32:03') - SQLITE_ERROR: table jobs has no column named metadata

[2024-10-18 00:32:03] INFO Ghost URL Service Ready in 3.509s

[9larsons]

Ah awesome, thank you. I suppose that's a good thing! I knew the migration needed updating but it's nice to know that it'll fail in CI. Should've thought of that.

[cmraible]

Migration looks good to me! Have some more general thoughts which I'll share in slack, but definitely nothing blocking — looking forward to trying this out in staging 🚀