Harvesting projects from a specific research domain

[nuest]

Hi RSD team,

I want to make sure I'm on the right track and am grateful for some brief pointers or confirmation regarding the following questions. I'm not good at SQL, so the PostgREST docs are leaving me behind at some points. I'm happy to get a "you can look that up yourself" answer for a few of the questions, of course, and I can offer to contribute documentation that might be useful for others in the future.

Goal: I want to harvest all software of selected research domains from the RSD API. In NFDI4Earth we're building a database of all Earth System Sciences-related "stuff" and want to include software, connect it to data, etc. For this we're writing harvesters for different sources, and I'm currently exploring how we could harvest the RSD, also keeping the potential in the back of the head that there might be more RSDs (domain specific?) that we might need to harvest in the future.

Questions

1. Am I on the right track when I use the API, or would is the OAI-PMH implementation top of your list? I could just scrape everything from there and filter on my end...

2. Can you confirm that not all Software is related to a Project ?

   Then we might combine the search results from different full-text queries with domain-based searches.

3. Can you point me to the app code where the query is built when I do a full text search in the frontend?

   I'm getting different results in my explorative tests of the API, but am probably just missing the right placeholders in the search term (to get partial matches etc.)

4. Can you confirm that there is no single query to get all software projects that are related to a research domain (via projects)?

5. How do I correctly construct the URL to get all projects with a specific research domains from rpc/research_domain_filter_for_project? I different options but feel stupid by now:

   https://research-software-directory.org/api/v1/rpc/research_domain_filter_for_project?PE10
   https://research-software-directory.org/api/v1/rpc/research_domain_filter_for_project?1=PE10
   https://research-software-directory.org/api/v1/rpc/research_domain_filter_for_project?research_domain=PE10

[ewan-escience]

1. Yes, using the API is currently the preferred way. It is reasonably stable, but requires knowledge of the underlying database structure. To help with this, you could look at the database code directly, our Swagger endpoint or our technical documentation. But we are indeed looking to make a OAI-PMH endpoint available in the near future, but this will take some work. By the way, if you happen to know useful tools/libraries or people who have done this before, please let us know.
2. Indeed, software does not have to be related to a project. As a matter of fact, projects are mostly used by the Netherlands eScience Center (NLeSC), so especially software not made by NLeSC might not have any related projects. We are also looking at categories for software, but that is still work in progress.
3. The frontend global search can be found here. It uses the global_search RPC, see also its definition.
4. There is one, e.g. this. One drawback, you don't get to see other research domains attached to the project, I haven't found a way around that. You can replace the select lists with asterisks (*) to see what fields are available, but we kindly ask you only to select the fields you need when using this in some production environment. You might also want to have a look at resource embedding.
5. Since you want to filter on an array of scalar values, you can use cs, see the operators, so the query becomes https://research-software-directory.org//api/v1/rpc/research_domain_filter_for_project?research_domain=cs.{PE10}

[nuest]

Thanks - that's very helpful. Will get back to here when I get around to continue my work on the implementation.

[nuest]

I fail to write a query that matches multiple research domains:

https://research-software-directory.org/api/v1/software?select=*,repository_url!inner(*),keyword!inner(*),project!inner(*,research_domain!inner(*))&or(project.research_domain.key.eq.PE10,%20project.research_domain.key.eq.SH7)

Any advise? Otherwise I'll just make multiple calls - since domains mainly cover NLeSC software I will likely combine this with full text and keyword searches anyway, so multiple calls is not an issue. More a question to learn the API better.

[ewan-escience]

Yes, that's possible, it took some puzzling, see the URL below. 😉 I did the following:

• removed a space (%20)
• added an equal sign after or
• moved the project.research_domain from within the or to before it

https://research-software-directory.org/api/v1/software?select=*,repository_url!inner(*),keyword!inner(*),project!inner(*,research_domain!inner(*))&project.research_domain.or=(key.eq.PE10,key.eq.SH7)