[Feature]: Imeplement Queues For More Efficient Article Fetching #82
Description
Package
API (Backend)
Problem or use case
Problem: As the number of feeds grows, with the number of active users, it takes longer to fetch the content. We are now exceeding the run execution limits on cloudflare. In #80 we temporarily resolved the issue (hopefully) by batching operations and running the cron more often.
Temporary Solutions: If #81 works as we hope, we will be able to handle up to ~1,200 feeds/hour.
It's pretty clear this solution will not last long, so this is very much temporary.
Proposed solution
We have started a new plan doc in docs/planning/queue-based-feed-fetching.md. We have a little time to think this over properly and plan ahead.
We will migrate to a queue-based approach. Cloudflare Queues and BullMQ for local/docker.
With a queue, each feed will be fetched individually as a parallel task. This however, might be expensive if we start getting a lot of users. This shouldn't be a concern untill we have over ~500 active users, so we might be able to kick that down the road a little further with a queue.
Additional context
docs/planning/queue-based-feed-fetching.md contains the current plan
Would you like to contribute?
- I'd be willing to contribute this feature