Add backend: ProQuest Federated Search Gateway #3991

maccabeelevine · 2024-10-07T18:59:19Z

The ProQuest Federated Search Gateway is an SRU API for searching across research databases licensed via ProQuest. It's old (docs last updated 2016) but still works and per ProQuest customer service:

The Federated Gateway is still in use ... There are no plans to discontinue it and we would be fine with them posting the VuFind code for integration for other customers.

On the plus side, it has a fairly rich CQL syntax for search. On the minus, there are no facets offered other than the constituent databases.

Implementation is patterned after the soon-to-be-deleted WorldCat backend as they both use SRU.

TODO

Integrate Add PropertyString, cleanHtml helper and escapeOrCleanHtml helper. #4004 for titles
Update changelog when merging (note changes to SRU backend)
Document in wiki when merging

maccabeelevine · 2024-11-06T16:35:59Z

Other than the TODOs, I think this is far enough to be worth a review. Also worth noting that there is no authentication config, it's simply IP-based access.

maccabeelevine · 2025-01-27T21:07:10Z

module/VuFindSearch/src/VuFindSearch/Backend/SRU/Connector.php

+     * @throws BackendException
+     * @return void
+     */
+    public function checkForSRUDiagnosticsError($resultBody)


@sturkel89 pointed out that if you are not on the correct IP range, the error was hidden and there were just zero results. It turns out that SRU likes to return an HTTP 200 no matter what goes wrong; there is a list linked above of dozens of different error codes, including '3' for authentication problems.

This is a general fix to make any such "diagnostic" messages throw an error akin to an HTTP 4xx.

You would think a "diagnostic" message might be benign debug content; but looking at the list, apparently it's always an error of some kind.

When I run a search off the VPN, I now get this 'red box' error. I think this is better than getting zero results. It would be nice to see text suggesting that you don't have access to ProQuest content because you are not on the campus network, but this may be the best we can do for now.

It would be nice to see text suggesting that you don't have access to ProQuest content because you are not on the campus network, but this may be the best we can do for now.

No that's just because we're both running VuFind on our local machines. In a production context, the error would be because the VuFind server was not in ProQuest's configured IP range. Would not matter whether the user had VPN or not.

And FWIW if you turn on development mode in your apache config, the error is more specific (and ugly). I unfortunately can't generate a screenshot of that today because I'm on campus.

We can consider this issue resolved, right? No more work to be done?

I think so!

demiankatz · 2025-01-28T15:24:00Z

This display logic is in templates/RecordDriver/DefaultRecord/result-list.phtml. I could make a ProQuest-specific version of that, but I think if we make this change it's best done in at the DefaultRecord level in a separate PR for more visibility. I do see that the EDS-specific record driver template displays volume, issue and page number, although that's because it's all just coming from a single EDS field rather than being an intentional choice, but in any case it would be a consistent change. IMHO I'm on the fence about this, it maybe information overload on the results page.

Probably best to keep things simple for now; I agree that it would be better to implement (perhaps configurable) base template changes rather than creating a custom ProQuest-specific template. But if what we're doing now is not actively incorrect, it's probably easiest to avoid adding complexity in this initial PR and instead tackle that separately as needed as a follow-up.

maccabeelevine · 2025-01-28T16:49:20Z

I think this is at a good functional point now.

I've implemented most of @sturkel89 's good suggestions, and as discussed above I think follow-up PRs are worth considering for others, that would be done in a more backend-agnostic way at higher-level templates.
Ideally I'd like to implement Add PropertyString, cleanHtml helper and escapeOrCleanHtml helper. #4004 as part of this, but for the moment I'm stripping HTML tags and decoding entities, and that's at least on par with any other backend. So integrating Add PropertyString, cleanHtml helper and escapeOrCleanHtml helper. #4004 could happen later.
I may also try something with pulling out subjects from the API into a separate facet, even though in the API it's used the same as a single database/source. But that should be a separate PR.

So I think this is ready for some final reviews and whatever cleanup.

sturkel89 · 2025-01-28T20:19:31Z

Thanks for all of the incremental improvements! This is looking a lot better than version 1.0. It's probably just about ready to merge (perhaps pending a response to my question about how date display relates to number of 045 fields).

However, I just noticed one other thing that is probably a bug. (I showed Demian and he agrees.)

Here it is:

When you run a search and view results, you see the Source filter on every page of search results as you page through the search results list.
When you're on the first page of search results, view a record, and then use the browser's back arrow or click Search in the breadcrumbs to go back to the full results list, the Source filter is visible on the results page.
BUT! When you are on a later page of search results, view one record, and then go back to the search results list, the Source filter disappears!

First page of results after viewing a record and then going back to see the results list:

Later pages of results after viewing a record and going back to see the results list:

maccabeelevine · 2025-01-28T20:54:48Z

BUT! When you are on a later page of search results, view one record, and then go back to the search results list, the Source filter disappears!

@sturkel89 This is a good catch. As you and @demiankatz say this is a bug. When I query the underlying API with any startRecord param, it refuses to pass me back any facets. The reason the bug doesn't appear when you page normally is that it's using AJAX to retrieve the next few records, and leaving the sidebar alone -- if you turn off load_results_with_js then the problem happens even with normal paging.

I'm going to report the issue to ProQuest, but I'm still waiting on a response to an email from November so not holding my breath. In the meantime I could fix this by doing a duplicate query every time for the facet sidebar, but that seems wasteful when probably 90% of faceting will happen on page 1? Could be convinced otherwise.

sturkel89 · 2025-01-28T21:05:48Z

BUT! When you are on a later page of search results, view one record, and then go back to the search results list, the Source filter disappears!

@sturkel89 This is a good catch. As you and @demiankatz say this is a bug. When I query the underlying API with any startRecord param, it refuses to pass me back any facets. The reason the bug doesn't appear when you page normally is that it's using AJAX to retrieve the next few records, and leaving the sidebar alone -- if you turn off load_results_with_js then the problem happens even with normal paging.

I'm going to report the issue to ProQuest, but I'm still waiting on a response to an email from November so not holding my breath. In the meantime I could fix this by doing a duplicate query every time for the facet sidebar, but that seems wasteful when probably 90% of faceting will happen on page 1? Could be convinced otherwise.

Hard to say what the typical user would do. My preference is to always have the filters available, esp. since PQ can provide such a firehose of results. If the user enters a simple search and gets a huge results set, it may take a couple of screens of browsing for them to realize that they should probably limit the results SOMEHOW.

demiankatz · 2025-01-29T13:26:31Z

BUT! When you are on a later page of search results, view one record, and then go back to the search results list, the Source filter disappears!

@sturkel89 This is a good catch. As you and @demiankatz say this is a bug. When I query the underlying API with any startRecord param, it refuses to pass me back any facets. The reason the bug doesn't appear when you page normally is that it's using AJAX to retrieve the next few records, and leaving the sidebar alone -- if you turn off load_results_with_js then the problem happens even with normal paging.
I'm going to report the issue to ProQuest, but I'm still waiting on a response to an email from November so not holding my breath. In the meantime I could fix this by doing a duplicate query every time for the facet sidebar, but that seems wasteful when probably 90% of faceting will happen on page 1? Could be convinced otherwise.

Hard to say what the typical user would do. My preference is to always have the filters available, esp. since PQ can provide such a firehose of results. If the user enters a simple search and gets a huge results set, it may take a couple of screens of browsing for them to realize that they should probably limit the results SOMEHOW.

I guess there are two ways we could go with this:

1.) If doing the redundant query to fetch the facet data is easy to implement and very fast (e.g. if you set record count to 0 and instantly get summary data back), maybe it's worth doing to make things work better...

BUT

2.) If fixing this problem adds complexity or significantly impacts performance, maybe the best course of action is to open a bug-level JIRA ticket documenting the problem so we don't forget it, but giving it some time to stew in case ProQuest can offer a better solution on their end.

I suspect option 2 is probably the path of least resistance for now...

maccabeelevine · 2025-01-29T15:00:22Z

BUT! When you are on a later page of search results, view one record, and then go back to the search results list, the Source filter disappears!

@sturkel89 This is a good catch. As you and @demiankatz say this is a bug. When I query the underlying API with any startRecord param, it refuses to pass me back any facets. The reason the bug doesn't appear when you page normally is that it's using AJAX to retrieve the next few records, and leaving the sidebar alone -- if you turn off load_results_with_js then the problem happens even with normal paging.
I'm going to report the issue to ProQuest, but I'm still waiting on a response to an email from November so not holding my breath. In the meantime I could fix this by doing a duplicate query every time for the facet sidebar, but that seems wasteful when probably 90% of faceting will happen on page 1? Could be convinced otherwise.

Hard to say what the typical user would do. My preference is to always have the filters available, esp. since PQ can provide such a firehose of results. If the user enters a simple search and gets a huge results set, it may take a couple of screens of browsing for them to realize that they should probably limit the results SOMEHOW.

I guess there are two ways we could go with this:

1.) If doing the redundant query to fetch the facet data is easy to implement and very fast (e.g. if you set record count to 0 and instantly get summary data back), maybe it's worth doing to make things work better...

BUT

2.) If fixing this problem adds complexity or significantly impacts performance, maybe the best course of action is to open a bug-level JIRA ticket documenting the problem so we don't forget it, but giving it some time to stew in case ProQuest can offer a better solution on their end.

I suspect option 2 is probably the path of least resistance for now...

Ok, I found a compromise (that I should have thought of in the first place) to only repeat the search when it's actually needed i.e. when startRecord > 1. Doing the extra search adds about 600ms, which I don't love, but it's not horrible especially if it's only done when necessary. (Setting record count to 0 doesn't appear to speed things up.)

maccabeelevine · 2025-01-29T15:02:47Z

Well no, I'm doing something wrong now and the offsets are not stable. Looking into it.

maccabeelevine · 2025-01-29T15:06:12Z

Well no, I'm doing something wrong now and the offsets are not stable. Looking into it.

Never mind, now I can't reproduce the issue.

sturkel89 · 2025-01-29T16:11:46Z

Well no, I'm doing something wrong now and the offsets are not stable. Looking into it.

Never mind, now I can't reproduce the issue.

This seems to be fixed for me!

sturkel89 · 2025-01-29T16:18:22Z

Another question/possible bug:

When I do a search, then limit results by source, I see that limit across the top of the screen. I'm finding that if I either change the search term or change the search type dropdown (All fields, Title) my limit goes away. Is that intentional? (In Villanova's implementation, these limits are sticky in both the main catalog and in EDS.)

demiankatz · 2025-01-29T16:40:51Z

Another question/possible bug:

When I do a search, then limit results by source, I see that limit across the top of the screen. I'm finding that if I either change the search term or change the search type dropdown (All fields, Title) my limit goes away. Is that intentional? (In Villanova's implementation, these limits are sticky in both the main catalog and in EDS.)

There are these two settings in searches.ini that control the filter retention behavior for Solr:

https://github.com/vufind-org/vufind/blob/dev/config/vufind/searches.ini#L103-L110

Maybe something equivalent is missing in the ProQuest-specific configuration or configuration processing. Let me know if you need me to dig deeper into this, @maccabeelevine!

maccabeelevine · 2025-01-29T18:13:28Z

When I do a search, then limit results by source, I see that limit across the top of the screen. I'm finding that if I either change the search term or change the search type dropdown (All fields, Title) my limit goes away. Is that intentional? (In Villanova's implementation, these limits are sticky in both the main catalog and in EDS.)

Thanks @sturkel89 for catching this. During my early testing I had changed the two config settings that @demiankatz cited. Just restored them to their defaults and it works as expected.

demiankatz

Thanks, @maccabeelevine and @sturkel89 -- see below for another round of (mostly very minor) comments.

demiankatz · 2025-01-29T19:41:20Z

config/vufind/ProQuestFSG.ini

+; [NoResultsRecommendations] sections below.
+; See the comments above those sections for details on legal settings.  You may
+; repeat these lines to load multiple recommendations.
+;default_top_recommend[] = TopFacets:FacetsTop:Primo


Stray Primo reference here. Should we just remove this line?

This one is not necessarily wrong (you could??? bring in Primo results at the top) but it's definitely confusing and shout not be there. Removed.

demiankatz · 2025-01-29T19:42:07Z

config/vufind/ProQuestFSG.ini

+
+; Checkbox facets are facets, which are getting displayed as checkboxes
+; pcAvailability is a special facet. It's used to show all records found in
+; PrimoCentral without checking the local availability against a holdingsfile.


This comment also seems to be copied from elsewhere and probably not relevant here.

Thanks, yes clearly using Primo.ini as a source was not ideal. I've reverted this to (almost) what it says in facets.ini.

demiankatz · 2025-01-29T19:42:27Z

config/vufind/ProQuestFSG.ini

+
+; This section shows which search types will display in the basic search box at
+; the top of Primo pages.  The name of each setting below corresponds with an
+; index defined in the Primo API.  The value of each setting is the text to


More Primo references here.

demiankatz · 2025-01-29T19:43:46Z

config/vufind/ProQuestFSG.ini

+view=full
+
+; This section provides settings for optional caching of search requests.
+[SearchCache]


I'm pretty sure that your connector here does not support search caching. Am I missing something, or should we remove these settings for now?

This should be working! It's actually the SRU Connector that uses ConnectorCacheTrait and uses it in search(); looks like I added that in this PR.

Fantastic! Sorry for missing that detail -- didn't look high enough in the class hierarchy.

No worries, I had to check myself anyway, I wrote that part back in October!

demiankatz · 2025-01-29T19:44:20Z

config/vufind/ProQuestFSG.ini

+; NOTE: This feature is incompatible with SyndeticsPlus content; please use
+;       regular Syndetics if necessary.
+[List]
+view=full


Has anyone tested whether tab/accordion modes work here? Let us know if you'd like @sturkel89 or me to give it a try!

I definitely tested it during development (only added stuff to ProQuest.ini that I tested -- except obviously I copied over a lot of bad Primo comments :) ) but that's definitely not proof it works as designed.

demiankatz · 2025-01-29T19:52:32Z

module/VuFind/src/VuFind/Search/ProQuestFSG/Results.php

+     */
+    public function getFacetList($filter = null)
+    {
+        if (null === $this->simplifiedResponseFacets) {


Should we also check here whether any facets are actually turned on? You should be able to check $this->getParams()->getFacetConfig() and bypass the extra lookup if the Database facet has not been requested.

This is a good point on efficiency in case someone turns off the Database facet, but I think I can just check $filter passed into this method to see if it's empty?

A null $filter value means "get all configured facets," so you might need to check the config if you get a null value... but you're right that if it's a non-null value, you can look at the filter and not worry about the config.

I implemented the $filter check and it seems to work correctly.

Ah sorry posted my comment before seeing your last one...will review.

I think you might want to add a && $filter !== null check in there to be safe; I can't remember under what circumstances the null value is passed in, but if you don't check for null, then the facets will be excluded when somebody asks for all the facets, which seems likely to cause a problem somewhere.

You were right the first time around (I think) -- I swapped out the $filter logic with getFacetConfig and it seems to work as expected in all cases, including if I hard code $filter = null they still display.

demiankatz · 2025-01-29T19:53:25Z

module/VuFindSearch/src/VuFindSearch/Backend/ProQuestFSG/Backend.php

+            $this->setRecordCollectionFactory($factory);
+        }
+        $this->connector    = $connector;
+        $this->identifier   = 'ProQuestFSG';


Minor style point: do the extra spaces before = really serve a purpose here?

No, but copied from six other Backend classes that do the same :)

demiankatz · 2025-01-29T19:54:36Z

module/VuFindSearch/src/VuFindSearch/Backend/ProQuestFSG/Connector.php

+ *
+ * PHP version 8
+ *
+ * Copyright (C) Andrew Nagy 2008.


Might want to update this copyright statement. You can certainly give Andrew credit in the author list if this is heavily derived from his original code, but I suspect this is largely your work and justifies a 2025 date.

Yeah that was copy/paste mistake from the SRU connector. I'm leaving this all at 2024 tho.

demiankatz · 2025-01-29T19:57:53Z

themes/bootstrap5/templates/RecordDriver/DefaultRecord/result-list.phtml

@@ -51,7 +51,7 @@
        <?php else: ?>
          <?php
            $journalTitle = $this->driver->getContainerTitle();
-            $summDate = $this->driver->getPublicationDates();
+            $summDate = $this->driver->getHumanReadablePublicationDates();


Have you done any analysis to be sure this change won't have weird side effects outside of ProQuest? (I haven't looked closely, just double-checking that this is safe and intentional since it impacts a base template; happy to dig in deeper if you'd like me to invest the time).

Yeah fair point, I did some quick display of Solr results and didn't see any difference, and I figured if there was going to be a difference, it would be a good once since this is a template and so the 'human readable' thing presumably applies. But that's not proof, and if it's safer I can roll it back and adjust the new driver accordingly.

I'll take a closer look at this tomorrow with a fresh brain and see how concerned I feel about it. ;-)

demiankatz · 2025-01-29T19:59:02Z

themes/bootstrap3/templates/Recommend/ProQuestFSGResults.phtml

@@ -0,0 +1,32 @@
+<?php


Looks like the bootstrap3 templates you added for this PR didn't get deleted when you merged in the removal of that theme. We should get rid of them now!

D'oh! When I merged in dev I saw all of the deletions of the existing bootstrap3 templates I had edited, but didn't think to also remove the new ones. Fixed.

…l $filter means display all of them

maccabeelevine added 22 commits October 4, 2024 16:34

Add ProQuest Federated Search Gateway backend. First working draft.

9263c83

Support basic search indices

518de13

Implement sorting

e6c1cd1

Add search result limits

300c51f

Display database facets

df23ce6

Apply database filter

0fb4406

Support checkbox filters

0f3021c

Add OOTB facet settings

ebab982

Add SearchCache support

8305dd5

Fix NPE

a23fefc

Fix single record load

d77a08d

Add next_prev_navigation

fc89632

Fix copy/paste typo

5d5b9d0

Remove dead code

896ba94

Add retain/reset filters configs that work OOTB

6ffeaa3

Add pagination and recommendation settings, work OOTB

d856c11

Simply constant backend params

b5a7ddd

Support list view options

2d8b4c8

Fix typo

f40b553

Remove dead code

a14a801

Add advanced search support

86abb67

Fix styles and add missing comments

ec2dc6c

maccabeelevine mentioned this pull request Oct 9, 2024

Allow safe tags in escapeHtml #3998

Closed

maccabeelevine added 5 commits October 9, 2024 13:49

Strip html from title

4a0b7b5

Merge branch 'dev' into proquest-federated-search-gateway

190abbe

Convert to bootstrap5

d698830

Add recommendation module

9670ee6

Fix styles

2e85a0f

maccabeelevine marked this pull request as ready for review November 6, 2024 16:34

maccabeelevine commented Jan 27, 2025

View reviewed changes

maccabeelevine added 2 commits January 27, 2025 21:10

Fix style

2a8fc17

Be safer about the error message

0814acd

maccabeelevine added 3 commits January 28, 2025 16:12

Merge branch 'dev' into proquest-federated-search-gateway

2f7531a

Suppress container info for books

35ce055

Decode html entities

60de692

maccabeelevine requested a review from demiankatz January 28, 2025 16:49

Perform a second search for facets when needed

96cd4fb

maccabeelevine added 2 commits January 29, 2025 15:06

Merge branch 'dev' into proquest-federated-search-gateway

11b8543

Define 'book' based on ProQuest data

ed7c53f

Restore defaults for retain/reset filters

d42f516

demiankatz requested changes Jan 29, 2025

View reviewed changes

maccabeelevine added 5 commits January 29, 2025 20:38

Remove references to Primo

5694042

Fix author / copyright

039db68

Remove new bootstrap3 templates created before that theme was removed

cab3c23

Save an API call if no facets are configured

c2e6276

Look to facet config for whether any should be displayed, since a nul…

8248d16

…l $filter means display all of them

Add backend: ProQuest Federated Search Gateway #3991

Are you sure you want to change the base?

Add backend: ProQuest Federated Search Gateway #3991

Conversation

maccabeelevine commented Oct 7, 2024 • edited Loading

maccabeelevine commented Nov 6, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

demiankatz commented Jan 28, 2025

maccabeelevine commented Jan 28, 2025

sturkel89 commented Jan 28, 2025

maccabeelevine commented Jan 28, 2025 • edited Loading

sturkel89 commented Jan 28, 2025

demiankatz commented Jan 29, 2025

maccabeelevine commented Jan 29, 2025

maccabeelevine commented Jan 29, 2025

maccabeelevine commented Jan 29, 2025

sturkel89 commented Jan 29, 2025

sturkel89 commented Jan 29, 2025

demiankatz commented Jan 29, 2025

maccabeelevine commented Jan 29, 2025

demiankatz left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

maccabeelevine commented Oct 7, 2024 •

edited

Loading

maccabeelevine commented Jan 28, 2025 •

edited

Loading