Preparing for more users: code analysis and structural improvements

[emmajclegg]

IATI Publisher's user base continues to grow and we want to ensure the tool can handle this with a minimum level of performance.

We need to understand the impact of more users on different parts of the tool and identify ways the tool could be improved for better scalability. We would then prioritise these based on effort and value.

Possible tasks:

• Identify areas where more simultaneous users will inevitably slow Publisher down due to architecture (e.g. single worker queue for publishing data)
• Identify, prioritise and implement possible improvements for better scalability

This issues relates to #1656 (monitoring to prepare for more users)

[robredpath]

I've heard informal reports of IATI Publisher being "slow" or "unresponsive" but they've always been short-lived and hard to replicate - which isn't much use to a developer trying to track down the issues!

I would suggest grouping the tasks you've described into two groups, and adding a couple more.

Group 1: Metrication and Monitoring

Set up monitoring of volumes of active vs. total registered users per month to track usage over time.

Ensure automated alerts and logs are in place

And, implement user-centric monitoring of performance for key pages. For example, how long does it take between clicking the "Sign in" button and the page finishing loading? Or, how often does a page appear empty to a user because a query to get the activities has timed out? That will then let us see which pages are slowest, and help us prioritise or identify areas for future investigation.

Group 2: Code Analysis & Structural Improvements

Assess the impact of more concurrent users on IATI Publisher's different processes - e.g. time taken to import, publish or load data in the interface.

And, identify areas where more simultaneous users inevitably will slow things down due to architecture. For example, I've read that IATI Publisher has a single worker for the publishing queue; what would be involved in increasing the number of workers for that queue?

Given that I'm suggesting splitting them up, it probably makes sense to separate this into two issues once work gets underway.