2020 was a notable year in information retrieval research and development. First there was the realization that AI applications were going to need a level of transparency that would reassure customers that they were supporting, not controlling, the search process. Second there was the realization that the search community had a responsibility to think beyond the Holy Grail of perfect relevance and take into account issues around bias, fairness and reproducibility.
Back in March 2020 I was looking forward to attending a workshop being run by Graham McDonald, Iadh Ounis and Craig Macdonald at the University of Glasgow on the subject of Fair Information Retrieval in Industry. It turned out to the first of many cancelled events last year! So I was delighted that the team were able to run a virtual version in March 2021 with a slightly different title. The papers are listed below, with video links where available.
The keynote was presented by Professor Maarten de Rijke (https://youtu.be/pmSXvPbm1lo) who laid out what he described as the operating principles, which he then considered with great clarity and insights. It was a master class in terms of content and delivery. He encouraged us focus on robust AI, which he defined as involving
- Accuracy, including well-defined and explained contexts of usage
- Reliability, including exhibiting parity with respect to sensitive attributes
- Repeatable and reproducible results, including audit trails
- Resilience to adversarial examples
- Resilience to distributional shifts
- Safety, including privacy preservation
The other presentations were
Leif Azzopardi, University of Strathclyde, UK. “The Fairness Hypothesis: Is fairer, better?”. https://youtu.be/36gsEYeop8g
Tim Gollins, National Records of Scotland, UK. “The Challenges of Fair Retrieval in Archives and the Public Sector”. https://youtu.be/Q34dCf7BtNs
Rishabh Mehrotra, Spotify, London, UK. “Diversity & Fairness in multi-stakeholder Marketplaces”. https://youtu.be/1B0MP_jBezE
Claudia Hauff, Delft University of Technology (TU Delft), The Netherlands. “SIREN: A Simulation Framework for Understanding the Effects of Recommender Systems in Online News Environments”. https://youtu.be/9F1Y0H8OjaQ
Pablo Castells, Universidad Autónonoma de Madrid, Spain. “Bias in recommendation: avoid it or embrace it?”. https://youtu.be/EGTSbQWr1EA
Carlos Castillo, Universitat Pompeu Fabra, Spain “Fairness in Ranking”. https://youtu.be/hxkXNql78IA
Debasis Ganguly, University of Glasgow, UK. “Towards Socially Responsible AI: Cognitive Bias-Aware Multi-Objective Learning”. https://youtu.be/05v-FDQc-Tw
Frank Hopfgartner, University of Sheffield, UK. “Promoting Algorithmic Transparency in Information Access”. https://youtu.be/IUHsnjAUlZo
Jean-Michel Renders, Naver Labs Europe, France. “Delivering Fair and Useful Rankings: a Generic Geometrical Projection Framework ”
Graham McDonald, University of Glasgow, UK. “TREC 2021 Fair Ranking Track: Task Overview ”
Amifa Raj, Boise State University, USA. “Comparing Fair Ranking Metrics”. https://youtu.be/vwwHIUotpQs
Graham McDonald tells me that there were 384 registrations. Attendance varied through the day but each session had at least 135 people. He was not yet sure of the peak session numbers but it was probably about 150. That is quite an outstanding achievement and a strong indication of the level of interest in this subject. The technology and timing worked to perfection, and there was no shortage of questions and comments.
If this area is still new to you I can recommend a very good preprint
The lead authors are at The Cyprus Center for Algorithmic Transparency (CyCAT)which is hosted at the Open University of Cyprus, the second public university in Cyprus and the only one dedicated to open and distance education. There is a link between the Centre and the Information School at the University of Sheffield, and the work of the Centre was the subject of Frank Hopfgartner’s presentation.
My take-away from the papers I was able to listen to (it conflicted with a major project meeting) was that the topic was so much more complex than I had imaged. Moreover we are only just beginning to identify the issues, much less come up with solutions and are still miles away from validating the solutions with search users.
If I had to give a best paper award (accepting that Maarten de Rijke’s was outstanding) I would give it to Tim Gollins from the National Records of Scotland, for reminding us that fairness and transparency are not just challenges to the management and discovery of digital records but to the collection of documents and artifacts from 1127 to the present day.