Failed to load resource error When I open Datasets page

[IDPogolovkin]

When I initially overload the Datasets page, the error Internal Server Error appears on the screen, and here's what I see in the console: Failed to load resource: the server responded with a status of 500, GET https://some-domain/api/v0/search/datasets?start=0&limit=11&publishingState=published 500.

But when I do a hard reload of the Datasets page one or more times or I go to the datasets page through the organizations page, falling into the datasets that are assigned to the organization, then everything loads normally and after that I can already go directly to the datasets page and everything loads without the 500 error.

Tell me please what is the problem here and how to fix it ?

[t83714]

Hi @IDPogolovkin

Magda uses Elasticsearch/OpenSearch as the search engine.

Magda’s search API’s default timeout time is 30 seconds, which means if it can’t get a response from the search engine within 30s, you will get 500 errors.

This config can be adjusted via helm chart config search-api.appConfig.akka.http.server.request-timeout.

What happened behind the longer-than-usual query time is:

Regarding why the search engine could take longer than usual when processing your first request after some idle time is:

Elasticsearch / OpenSearch will move shards to an "idle" state when an index has not been queried for some time
This behaviour is controlled by config index.search.idle.after
You can also see this ticket: opensearch-project/OpenSearch#9707
Once the index is hit by a query request after some time, it will take some time for the search engine to make previous index data available for searching.
This strategy is designed for optimise indexing speed of the index that is not queried often.

For most use case, this won’t be an issue unless your application doesn’t have too many visitor and are rarely accessed for more than 30s.

If this is the case, I’d suggest:
Option 1: periodically send requests to /api/v0/search/datasets every 30s to avoid the search engine marking the index as idle.
E.g. you can create a cronjob in the k8s cluster or use any service monitor online tool
Option 2: Update index.search.idle.after config to a very large value
Option 3: Increase the search API timeout as mentioned above

[t83714]

Has been addressed in Magda v5.0.0 release via this ticket: #3576