Feature/404 send full text for search

[anusharanganathan]

This is a PR for https://github.com/antleaf/jct-project/issues/404

It sends the full text to the API, but the auto suggest doesn't kick in until something other than the stop words are typed
For example: Journal of Nu or University of Ca

[richard-jones]

I've had a play around with this locally, and I found some slightly odd behaviour in the journal and institution boxes:

• When searching "journal of" I got results immediately after typing "jo" which makes sense, because that's not a stop word. That found a bunch of journals like JOM and JOR. But after I got to typing "jou" those results stayed, and I wasn't sure why.

• I found if I typed the journal of (with the trailing space) it would give me search results. Perhaps we need to strip whitespace as part of the stop word stripping?

• If I type "university of o" in the institution box I don't get a result, then if I type university of ox in the box, then I get to see Oxford. If I then delete the x leaving me with university of o it is still showing me Oxford as an option. In fact it continues to show me Oxford until I get down to un and then it give me a different set of results.

I'm not sure exactly what the fixes here are - they are mostly about user expectation rather than being technically correct, I think. I will try to have some thoughts about what behaviour might make most sense.

[anusharanganathan]

When searching "journal of" I got results immediately after typing "jo" which makes sense, because that's not a stop word. That found a bunch of journals like JOM and JOR. But after I got to typing "jou" those results stayed, and I wasn't sure why.

The rules for auto completion when typing in the journal, funder or institution boxes

1. There needs to be atleast 2 characters before we ask for auto completion suggestions.

2. The pattern matching kicks in if atleast 3 characters are matched from the word. The patterns matched are

   • Journal: of, the, spaces, jou, jour, journ, journa, and journal
   • Funder stripped words: of, the and spaces
   • Institution stripped words: of, the, spaces, uni, univ, unive, univer, univers, universi, universit and university.

   If after stripping the matched pattern, 2 characters or more remain, we will ask for auto completion suggestions.

For journal auto suggestion, if the user wants to search for journal of neurology
* Auto completion will receive suggestions after typing jo.
* From jou until you have finished typing journal of ne, there will be no new auto completion suggestions. The initial suggestions obtained after typing jo will continue to be displayed until new auto suggestions can be obtained.

For institution auto suggestion, if the user wants to search for university of oxford
* Auto completion will receive suggestions after typing un.
* from uni until you have finished typing university of ox, there will be no auto completion suggestions. The initial suggestions obtained after typing un will continue to be displayed until new auto suggestions can be obtained.

For funder auto suggestion, if the user wants to search for the wellcome trust
* Auto completion will receive suggestions after typing the
* from the until you have finished typing the we, there will be no auto completion suggestions. The initial suggestions obtained after typing th will continue to be displayed until new auto suggestions can be obtained.

One possible improvement, would be to start pattern matching after 2 characters to match with the auto completion expectations. So match jo, jou.... and un, uni, univ.... By doing so though, there will be no auto suggestion until the user types journal of ne or university of ox

The best way to handle this I think, is to not strip any words and use the garbage in, garbage out approach

[richard-jones]

One possible improvement, would be to start pattern matching after 2 characters to match with the auto completion expectations. So match jo, jou.... and un, uni, univ.... By doing so though, there will be no auto suggestion until the user types journal of ne or university of ox

I would remove the substrings of "journal of" and "university of" (jou, univ, etc) from our stop list, as they could be substrings of other words that the user is trying to type. Like, I guess, "JOUrney" or "UNIVerse", and we may inadvertently prevent them from searching those terms.

Perhaps what we really need to do is go back to the original issue and hash out exactly how we want this to work and get agreement from the whole team?