Sidekiq without ActiveJob

[davidpiegza]

Using Sidekiq without ActiveJob reduces memory usage drastically.

On my development machine with Rails 5.1.5, Ruby 2.4.3 and Sidekiq 5.1.1, running a test with 250_000 sub jobs, the results for the MainJob process are:

With ActiveJob:
Memory Usage: 522 MB
Time: 379.329 sec

Without ActiveJob
Memory Usage: 60 MB
Time: 121.536 sec

The concurrency has been set to 10.

The test can be run with rake sidekiq:test (start sidekiq before).

[mperham]

In designing and building Sidekiq, I've tried to make pragmatic trade-offs between performance and developer friendliness. I can also optimize the happy path and provide special APIs for edge cases. For instance, you can make your benchmark even faster if you use the push_bulk API. This load tester creates 100,000 native jobs in 2 seconds (vs 40 sec for AJ) on my laptop:

https://github.com/mperham/sidekiq/blob/master/bin/sidekiqload#L92-L100

AJ has much more complex Ruby serialization/deserialization logic, which will also chew up memory/CPU. Sidekiq's de/serialization is explicitly "JSON only" (one of those pragmatic decisions mentioned above) and therefore is implemented in pure C within the "json" gem.

And when you actually want to process those jobs? The 100,000 jobs take 22 seconds to process on my laptop. The same AJ jobs take 65 seconds to process, so ~3x the overhead.

[davidpiegza]

@mperham Thanks a lot for your explanation and for mentioning push_bulk, this is definitely a useful option.

Process time is not really important in our case, the jobs get enqueued in batches (of 50_000 by default) and then processed by Sidekiq.

As a conclusion, if you need to enqueue/process more than 100k jobs it makes more sense to use Sidekiq without ActiveJob (to save memory/improve performance and to use advanced Sidekiq features like push_bulk).

Does it makes sense to mention this in the Limitations section in the Wiki (https://github.com/mperham/sidekiq/wiki/Active-Job#limitations)?

[mperham]

https://github.com/mperham/sidekiq/wiki/Active-Job#performance