Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Added experimental background job queue #20985

Merged
merged 37 commits into from
Nov 4, 2024
Merged

Added experimental background job queue #20985

merged 37 commits into from
Nov 4, 2024

Commits on Sep 11, 2024

  1. Configuration menu
    Copy the full SHA
    fa3773a View commit details
    Browse the repository at this point in the history

Commits on Sep 12, 2024

  1. Added and wired up the jobs queue into JobManager

    Added config flags for the job queue to make it switchable
    
    config options are in services:jobs:queue and are 'enabled,reportStats,reportInterval,maxWorkers,logLevel(info/debug),pollMinInterval,pollMaxInterval,queueCapacity,fetchCount.
    9larsons committed Sep 12, 2024
    Configuration menu
    Copy the full SHA
    0944f8e View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    cbfb1a2 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    d74d339 View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    e3b68ec View commit details
    Browse the repository at this point in the history
  5. Configuration menu
    Copy the full SHA
    f47448b View commit details
    Browse the repository at this point in the history

Commits on Sep 13, 2024

  1. Removed db dependency

    This is slightly less efficient - potentially - because we do a select before inserting instead of ignoring the insert conflict. It's likely less efficient because I don't anticipate a ton of duplicates, although the analytics job can certainly result in that, which is where I'd expect the knex implementation to slightly win out.
    
    Regardless, testing is a fucking nightmare with knex directly as we have to spin up a db and use a schema for the table. Let's go down that path later if we need the performance improvements.
    9larsons committed Sep 13, 2024
    Configuration menu
    Copy the full SHA
    713dcc4 View commit details
    Browse the repository at this point in the history

Commits on Sep 16, 2024

  1. Added bulk of unit tests

    9larsons committed Sep 16, 2024
    Configuration menu
    Copy the full SHA
    4b3b928 View commit details
    Browse the repository at this point in the history
  2. Fixed linting

    9larsons committed Sep 16, 2024
    Configuration menu
    Copy the full SHA
    85c28ea View commit details
    Browse the repository at this point in the history
  3. Fixed unit tests

    9larsons committed Sep 16, 2024
    Configuration menu
    Copy the full SHA
    010369b View commit details
    Browse the repository at this point in the history
  4. Fixed linting

    9larsons committed Sep 16, 2024
    Configuration menu
    Copy the full SHA
    62807b9 View commit details
    Browse the repository at this point in the history
  5. Configuration menu
    Copy the full SHA
    e253203 View commit details
    Browse the repository at this point in the history

Commits on Sep 17, 2024

  1. Add workerpool

    9larsons committed Sep 17, 2024
    Configuration menu
    Copy the full SHA
    125f3d7 View commit details
    Browse the repository at this point in the history
  2. Added and updated jobs integration tests

    - moved to a more appropriate location (integration tests from e2e-server)
    9larsons committed Sep 17, 2024
    Configuration menu
    Copy the full SHA
    7aeb7e0 View commit details
    Browse the repository at this point in the history
  3. Fixed linting

    9larsons committed Sep 17, 2024
    Configuration menu
    Copy the full SHA
    5508cfd View commit details
    Browse the repository at this point in the history

Commits on Sep 18, 2024

  1. Configuration menu
    Copy the full SHA
    c602c67 View commit details
    Browse the repository at this point in the history
  2. add more debug logging

    9larsons committed Sep 18, 2024
    Configuration menu
    Copy the full SHA
    2cd905f View commit details
    Browse the repository at this point in the history
  3. Add even more logging

    9larsons committed Sep 18, 2024
    Configuration menu
    Copy the full SHA
    e5529c0 View commit details
    Browse the repository at this point in the history
  4. closure

    9larsons committed Sep 18, 2024
    Configuration menu
    Copy the full SHA
    aa8fc8b View commit details
    Browse the repository at this point in the history
  5. .

    9larsons committed Sep 18, 2024
    Configuration menu
    Copy the full SHA
    6694210 View commit details
    Browse the repository at this point in the history
  6. .

    9larsons committed Sep 18, 2024
    Configuration menu
    Copy the full SHA
    2fd8caf View commit details
    Browse the repository at this point in the history
  7. .

    9larsons committed Sep 18, 2024
    Configuration menu
    Copy the full SHA
    22b76b0 View commit details
    Browse the repository at this point in the history
  8. ,..

    9larsons committed Sep 18, 2024
    Configuration menu
    Copy the full SHA
    1d77f5a View commit details
    Browse the repository at this point in the history

Commits on Sep 25, 2024

  1. Configuration menu
    Copy the full SHA
    d341de1 View commit details
    Browse the repository at this point in the history
  2. Removed blank line

    9larsons committed Sep 25, 2024
    Configuration menu
    Copy the full SHA
    e5ea94f View commit details
    Browse the repository at this point in the history
  3. Removed clogs

    9larsons committed Sep 25, 2024
    Configuration menu
    Copy the full SHA
    0a1d943 View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    ff7a699 View commit details
    Browse the repository at this point in the history

Commits on Sep 26, 2024

  1. Updated integration tests

    It appears testUtils.setup MUST be called as before(testUtils.setup()); and doesn't work as an anonymous function with any consistency.
    9larsons committed Sep 26, 2024
    Configuration menu
    Copy the full SHA
    06cc10c View commit details
    Browse the repository at this point in the history

Commits on Sep 30, 2024

  1. Configuration menu
    Copy the full SHA
    55cdf52 View commit details
    Browse the repository at this point in the history

Commits on Oct 7, 2024

  1. Configuration menu
    Copy the full SHA
    1adeee1 View commit details
    Browse the repository at this point in the history
  2. Updated testutils call

    This was failing locally due to the local test db (in /tmp/ghost-test.db) missing migration columns. Not sure why. It could be due to version mismatch at this point as the migration needs to be updated before merging this. Blowing away that file and then re-running init'd a table with the correct schema.
    9larsons committed Oct 7, 2024
    Configuration menu
    Copy the full SHA
    635a6b8 View commit details
    Browse the repository at this point in the history

Commits on Oct 16, 2024

  1. Configuration menu
    Copy the full SHA
    48aef4e View commit details
    Browse the repository at this point in the history
  2. Fix merge

    9larsons committed Oct 16, 2024
    Configuration menu
    Copy the full SHA
    8790ae5 View commit details
    Browse the repository at this point in the history

Commits on Oct 31, 2024

  1. Configuration menu
    Copy the full SHA
    cad48b8 View commit details
    Browse the repository at this point in the history
  2. Added migration

    9larsons committed Oct 31, 2024
    Configuration menu
    Copy the full SHA
    c623c50 View commit details
    Browse the repository at this point in the history

Commits on Nov 4, 2024

  1. Configuration menu
    Copy the full SHA
    89d54bc View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    6da5fbf View commit details
    Browse the repository at this point in the history