Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature/1584 batched job #1598

Draft
wants to merge 6 commits into
base: 5.x
Choose a base branch
from
Draft

Feature/1584 batched job #1598

wants to merge 6 commits into from

Conversation

i-just
Copy link
Contributor

@i-just i-just commented Feb 12, 2025

Description

Feeds are now processed as batched jobs.

Notes:

  • Pagination is still supported, as it was before. If you have a feed that has 3 pages, each page has, e.g. 200 primary elements to be processed, then each page will take two batches.
  • services/Process.php->processFeed() - the last parameter ($feedData) is no longer optional
  • services/Process.php->afterProcessFeed() - now accepts the 4th param ($startTime); it’s needed for the time calculations to work correctly for the paginated view; if omitted, the time calculation will still work, but as before - timing only the last page
  • services/Process.php->beforeProcessFeed() is now split into 2 methods, beforeProcessFeed(), which runs before each page (if the feed is not paginated, then there’s only one page), and getFeedSettings(), which runs before each batch
  • Processing X elements finished in <number>s => it looks like this was a bit buggy before for paginated feeds; it reported the total number of processed items but only really measured the time of the past page (fixed now)
  • it looks like the logs were always a bit wonky for the paginated feeds; they were always nesting one of the processed elements under Finished preparing for feed processing. because the $stepKey was not cleared as expected (fixed now)

Related issues

#1584

@jamesmacwhite
Copy link
Contributor

This is literally the best thing ever for the Feed Me, can't wait for this to be merged!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants