article selector (on author page) doesn't always correctly find existing articles

[eplebel]

steps:

1. add 10 new articles (saving them as "article 1" (for title and authors), "article 2", etc. up to "article 10"
2. go to a different author page
3. click "Add Preexisting Article"
4. type the letter 'a'
   notice that the 10 articles are correctly being listed (though it's difficult to see given what appears to be a minor UI bug within the selector dropdown)
5. select any article and add it to the article list
6. repeat the last two steps (steps 4 and 5) until all 10 articles are linked

notice that eventually the figure selector won't list any more available articles, even though not all of the 10 articles were linked. i'm usually able to add 6 or 7, and then the remaining 3-4 don't show up somehow (and therefore cannot be linked).

I believe this WAS working correctly before the prevent already-linked articles FIX (i.e., a9edead ), but i'm not 100% sure.

[onejgordon]

Investigated this, and I believe it's a result of a page limit in the autocomplete API.

The query 'a' returns at most 10 articles currently, and if other articles match, then they may displace the 10 newly created test articles. Then when we filter the results in the UI to remove those that are already linked to the author, the result can be an empty list making it impossible to link additional test article(s).

We can confirm this by seeing that the problem goes away when additional characters are added to the query (e.g. 'art' correctly filters to the remaining unlinked test articles).

@alexkyllo it looks like we're using Select2QuerySetView for this. Possibilities I can see:

1. Increase default page size for autocomplete, perhaps to 20-30
2. Add the option to exclude a set of articles (author's already linked articles) via query parameters
3. Additional endpoint that performs a query including the exclude in 2, but all on the server side (API would have to take author_id as a parameter)
4. Change UI to only perform query when > 2-3 characters are entered

Note that none of these comprehensively solves this problem, since no matter what the implementation we can imagine a scenario where there are a sufficient number of articles beginning with the same text that the one the user is searching for doesn't make it to the top of the list. I do think we can functionally solve this with any of the above, however, thereby making it very unlikely users will run into this.

[eplebel]

Investigated this, and I believe it's a result of a page limit in the autocomplete API.

i'm not sure about this diagnosis given that it also happens when only a few articles need to be returned and when searching by entire words (rather than by individual letters). (i initially detected the issue with real articles, but decided to log the simpler 'a' cases to simplify bug reporting and investigation):

for example, if you're on Lorne Campbell's author page and try to add the article "EP LeBel (2017) A new replication norm for psychology", typing "new" or even "new norm" yields 0 results. but if you type "lebel", the article is correctly found.

but this doesn't happen with other articles. for example, if you try to add the article "EP LeBel & CJ Wilbur (2014) Big secrets do not necessarily cause hills to appear steeper" to Lorne's author page. typing "big" or "big secrets" or "wilbur" all correctly finds the article. (though typing "big secrets wilbur" does not).

i also saw a case where i was not able to find an article by author name, but could find it via title words, but i can't reproduce this right now.

hope that helps.

[onejgordon]

Ah ok, it seems there are multiple causes for these symptoms. I still believe there is a page limit affecting the Article 1-10 test, but for the issue you just discovered on Lorne's profile, it appears the reason is actually that author search is a full text match, while title search looks to match at beginning of string only. In ArticleAutocomplete we have:

            queryset = queryset.filter(Q(author_list__icontains=self.q)
                                       | Q(title__istartswith=self.q))

This seems to explain why the search for "new" and "new norm" don't work, but "lebel" and "a new" do find the article.

I assume this is an easy fix __istartswith --> __icontains, but @alexkyllo should confirm.

[alexkyllo]

Yes, I believe it's that easy. Of course icontains is slower than istartswith but that probably won't matter until we have millions of articles.

[eplebel]

this fixes my main examples ("new" or "new norm" yielding 0 results), however the combined title and author search (also mentioned) still doesn't work (e.g., "big secrets wilbur" does not find the article ""EP LeBel & CJ Wilbur (2014) Big secrets do not necessarily cause hills to appear steeper"").

is there an easy/quick fix for this?

if not i can live with this for now, but please let me know, thanks.

[onejgordon]

As far as I can tell, combined title/author search is not supported by the current API as it is returning matches when either the title, or the author list, includes the entire query term (e.g. "big secrets wilbur"). It seems likely we could add a combined (possibly computed) field merging authors and title and run a single contains match against the query term. That said, I haven't implemented this kind of query in django/postgres so @alexkyllo will have to confirm.

[eplebel]

ok thanks for additional info. definitely doesn't sound easy, so i'm going to close this for now and we can re-visit this later.