Background : Those who develop search hedges and systematic review strategies often use a process where they measure how effective a given search is at retrieving items from a set of “gold standard” articles. While this is a robust method, it is also resource-intensive. There are many occasions when a searcher needs to refine a search strategy in an iterative way, but she does not have time to manually assemble a set of exemplar citations. This describes an application that would allow a searcher to test different iterations of a search strategy against a set of “known-good” citations in a streamlined way. Description : This application, currently under development, will allow a user to test and refine searches against PubMed, which is both eminently suited to biomedical searchers and possessed of a robust API for developers. The task flow will begin with a user’s initial search, after which she will be presented with a limited, randomly chosen set of search results. She will then examine the citation (and abstract, if available) of each result and determine whether it is relevant to her search task or not. Citations marked as relevant will be stored as “known-good” (or at least “likely-good”) items for that search, while those that are obviously irrelevant can be deemed “known-bad”. From there, successive search strategies can be automatically tested in the application as to how well they retrieve the desired citations while also avoiding the bad ones. Conclusion : By having a way to easily quantify the performance of different strategies, it is possible that this application could improve the rigor of “everyday” searches for complex topics. User testing will be necessary to refine this concept. In particular, the author would wish to determine the minimum number of items needed for examination in the initial stage in order to provide reliable information as to how well successive searches perform.

Presentation given at 2019 Medical Library Association Annual Meeting in Chicago, IL