John Tait FBCS, Secretary of the IRSG, attended the Tony Kent Strix Award Lecture on Friday 29 November 2019 at the Royal Geographical Society in London. This is his report of the meeting
If you don’t know it, the Tony Kent Strix Award is probably the second most prestigious life time achievement award in IR (after ACM SIGIR’s Salton Award). The award commemorates Tony Kent, a pioneer of early information retrieval systems, and awarded through a joint effort form the BCS IRSG, the UKeiG of CILIP, ISKO and Royal Institute of Chemistry’s CICAG. In the past it has been won by Stephen Robertson, Donna Harman, Keith van Rijsbergen and Bruce Croft amongst others. More information may be found here: https://en.wikipedia.org/wiki/Tony_Kent_Strix_award.
The afternoon began with a brief welcome by Doug Beale, the long standing chair of the committee which organises the award, who announced the winner of the 2019 Strix Award, Professor Ingemar Cox of University College London, and the University of Copenhagen. Professor Cox is a former member of the IRSG Committee and has made a number of notable contributions to IR in particular in image retrieval.
As is customary next was an introductory lecture on a topical or historical area by someone close the award and its organisation. This year Andrew Macfarlane of City University London (the IRSG’s longest standing committee member), focused on the tension between issues like searcher privacy and control a versus the way modern public information systems like Google and Facebook select and rank information to be delivered.
https://drive.google.com/file/d/1WUjKqyZwzLEJb5sljGzWePCBQDDOTIZg/view
He pointed out that as long ago as 1976 Belkin and Robertson (the latter the first Strix Award winner) pointed out the potential abuses caused by authors using their knowledge of readers for their own ends, and their analysis remains sound despite the many changes in technology which have taken place in the intervening period. This work rests on fundamental work by Shannon on his Communication Theory of 1946.
A number of principles to which information systems should conform were put forward, including searcher control, elimination of bias in results, provision of complete results lists and so on.
Andy’s talk was timely and addressed and important set of topics. However I felt much of it needed greater refinement, or was based on some doubtful assumptions: for example whether unbiased ranking is feasible in principle or even desirable in practice; or the assumption that social media have had a wholly negative influence on politics: when I know from personal experience they have helped enormously with small and new parties getting their message across to potential voters. These are reasonable shortcomings in research which is at an early stage, and I wish Andy well with the development of his work.
We then moved to the main event of the day: The Strix Award Lecture presented by Professor Pia Borlund of Oslo Metropolitan University, the winner of the 2018 Tony Kent Strix Award. Pia began with thanks to a number of people who had laid the foundations of her work, including previous Strix Award winners Keith van Rijsbergen, Kalervo Jarvelin, and Peter Ingwersen.
https://drive.google.com/file/d/1BosRT0sDMvVjPxLCR4dpxuYT9ip-FSWW/view
Professor Borlund’s talk was a plea for greater realism in information retrieval evaluation. Much evaluation work is based on the Cranfield model which in its modern interpretation has user judgements about the relevance of results only at a distance. Her own work from her PhD onwards was based on work task simulations in which searchers were asked to make judgements about the effectiveness of searches which they would actually undertake in their day-to-day work. This has yielded more realistic evaluations of information retrieval systems than are produced by the Cranfield model.
She pointed out many user studies in IR use students as subjects and then ask them to undertake search tasks, make relevance judgements, and so on, with queries and searches which they were unlikely to undertake in their day-to-day work.
Pia concluded that tailoring experimental tasks to the actual searchers used in the experiments was an important element in ensuring IR continues to have sound empirical underpinnings.
I think this is an important conclusion when too much research work in IR and related disciplines like Computational Linguistics and Machine Learning is based tiny incremental improvements of doubtful statistical validity on standardised benchmarks and tasks. The development of such benchmarks and tasks has contributed much to improving the reproducibility and rigour of research, but runs the danger that the results do not reflect the performance of real systems by real users in the real world.
Unbeknown to the two speakers the Belkin and Robertson (1976) paper referred to above, and therefore Shannon’s Communication Theory, was an important foundation of the both the lectures, although in Professor Borlund’s the link had only recently come to light.
Both the talks provoked likely debate in the question sessions.
I found the afternoon a stimulating and useful one, and I look forward to next year’s lecture.