The 10th annual meeting of the Forum for Information Retrieval Evaluation (FIRE 2018) took place in Gandhinagar, India. The conference took place at the Dhirubhai Ambani Institute of Information and Communication Technology (DA-IICT). Several participants dubbed the venue as a mini zoo because of the varied wildlife, especially monkeys, squirrels and peacocks on the campus. FIRE 2018 was one of the biggest FIRE till date with over 80 participants and 11 invited speakers. Besides the main conference FIRE-2018 hosted six evaluation tracks and two tutorials.
FIRE 2018 consisted for the following tracks:
- Event Extraction from Newswires and Social Media Text in Indian Languages (EventXtract-IL)
- Information Extractor for Conversational Systems in Indian Languages (IECSIL)
- Information Retrieval from Microblogs during Disasters (IRMiDis)
- Multi-lingual Author Profiling on SMS Messages (MAPonSMS)
- Verb Phrase Translation in English and Indian languages (VPT-IL)
- Indian Native Language Identification (INLI)
With the MAPonSMS task, Urdu became the latest inclusion in FIRE, which has continuously expanded to include evaluation tracks in new languages from the subcontinent.
Day 1: Thursday, 6th December
The first day of conference constituted of two half day tutorials. The first tutorial was on ‘Deep Learning based Approach to Build a Conversational Agent and Practical Challenges’. The tutorial was offered by Manoj Kumar Chinnakotla, Kedhar Nath Narahari, Puneet Agrawal from the AI & Research division in Microsoft India and USA. The tutorial covered a wide range of topics including the basics of conversational systems, specific challenges like code mixing and evaluation techniques. It also touched upon topics like handling emotions and dealing with offensive and troll behaviours.
The second tutorial was offered by Sumit Bhatia, who talked about ‘Knowledge graphs and Information Retrieval’. In the first part, Sumit talked about the uses cases of knowledge graphs in information retrieval tasks such as query understanding, query expansion, and entity-oriented search. And then he complemented this with techniques in information retrieval that can help improve the search and management of structured data in large-scale knowledge graphs. Both the tutorials were well received with a volley of questions from the participants, a lot of whom were masters students, aspiring to begin their research journeys soon.
Day 2: Friday, 7th December
The second day of FIRE started with a brief introduction by Prasenjit Majumder. He recalled the journey from the first edition back in 2008, when he and Mandar, with support from the IR community, started the journey with 30 participants, to where we stand today. This was followed by a congratulatory message from Donna Harman on the occasion of the tenth anniversary of FIRE. The message was read out by Mandar Mitra. The conference began with an invited talk by Stephen Robertson on “The gaps between the words, a small history of space”. Stephen talked about how the use of ‘spaces’ in written texts originated and evolved through time, and how this has played a pivotal role in making Information Retrieval possible, the way we know today. This was followed by another invited talk on “Machine Translation of Low Resource Languages”, by Pushpak Bhattacharya. In this talk he spoke about the work on low resource MT, exploiting the techniques of factors, subwords, pivots and unsupervised Neural MT for translating between pairs of languages that have no or little parallel corpora. This was followed by track overview of the Indian Native Language Identification Task (INLI) and Information Extractor for Conversational Systems in Indian Languages (IECSIL) tracks.
The post lunch session saw a talk by Amlan Dasgupta on interpreting archival evidence of musical sound. Next, Dharmendra Saha talked about his journey with Rekhta, which works towards promotion and preservation of Urdu langauge and making it more accessible to public. This was followed by an invited talk by Gareth Jones over skype. Gareth talked about Information Retrieval from spoken content. The day ended with an overview of Multi-lingual Author Profiling on SMS Messages (MAPonSMS) track and presentations from the teams participating in this track.
The lunch break and both the tea breaks for the day saw the participants flocking to the sponsor booths to know more about the work being done by them. The folks from Microsoft demonstrated how AI is helping villagers in Rajasthan to retain their artistic heritage and at the same time create more business opportunity for them. On the other hand, people from Rygbee demonstrated their product RAXter, which is an upcoming digital research assistant. The long day ended with participants enjoying the perfect weather at a poolside conference dinner. One of the hot topics of discussion at the conference dinner was the lack of beer in the ‘Dry State’ of Gujarat and the majority opinion in favour for conducting (atleast one of) the next editions of FIRE in Goa.
Day 3: Saturday 8th December
Day three started with a riveting talk by Doug Oard, who talked about the MATERIAL programme at IARPA. Machine Translation for English Retrieval of Information in Any Language (MATERIAL) is a project aimed at finding speech and text content in low-resource languages that are relevant to domain dependent English queries. Doug talked about the systems which use minimal training data and can be rapidly ported to new languages. Doug’s talk was followed by oral presentations from the authors of selected conference papers. After the tea break, Shourya Roy from American Express talked about AI-driven decision making in the real world. He provided a nice bigger picture of how AI is shaping the world around us and the crucial role that humans play in the whole scenario. Shourya’s talk was followed by another session of conference oral presentations.
The second half started with the final session of oral presentations followed by an invited talk from Jaap Kamps. Jaap talked about “Learning Sparse Representations for Neural Rankers”. An animated talk and a ‘popular’ topic kept the audience wide awake even in the post-lunch session, making the tea break almost redundant. The spotlight during the break was Karin Verspoor sharing the details about her upcoming trip to Antarctica, with many from the audience wondering if it would be possible to replace the invited talk she was going to deliver later that day with a talk about Antarctica and penguins.
The tea break was followed by presentations from participants of the Information Extractor for Conversational Systems in Indian Languages (IECSIL) track. After the track presentations, Karin Verspoor talked about the kind of information biologists usually require and how semantic search will help solve this. She also talked about several research directions that can contribute to building a better semantic search system for biomedical literature. The day ended with a couple more presentations from the track participants.
This was followed by a cultural event, where the ‘Khelaiya Group’ from DAIICT presented the traditional ‘Garba’ dance from Gujarat, among various other performances. It was amazing to see invited speakers and other guests join the group, with some of them picking up the dance really well.
Day 4: Sunday, 9th December
The final day of FIRE started with an invited talk from Allan Hanbury, who talked about advances in word embeddings and their use cases in information retrieval. This was followed by presentations from participants of the Indian Native Language Identification (INLI) track. Next, Pavel Pecina gave a talk on ‘Machine Translation for Cross-Lingual IR in medical domain’. He talked about the Khresmoi translator that provides MT for the biomedical domain and supports seven European languages apart from English. Next, there was an overview presentation for the Information Retrieval from Microblogs during Disasters (IRMiDis) track and a few more presentations from the track participants some of which rolled over to the post-lunch session. This was followed by a talk from Paulo Quaresma on the topic of ‘automatic identification of arguments in legal documents’ which covered the specifics of such an architecture and possibility of extending it to other languages and domains. Post the tea break participants of the Event Extraction from Newswires and Social Media Text in Indian Languages (EventXtract-IL) track presented their systems and results. The ultimate talk of the day and of FIRE 2018 was a very informative invited talk from Nicola Ferro. Nicola talked about ‘Rich Explanatory Models of IR System Performance’. The talk covered state-of-the-art in performance modelling and a discussion on how to use these results for performance prediction of an IR system.
The closing of 10th FIRE was a bit special. FIRE 2018 ended with a recorded message from Carol Peters:
“There’s TREC and CLEF and NTCIR, but never forget there’s also FIRE,
America, Europe, Asia, near and far, Donna, Noriko, Nicola and Mandar,
They all follow their guiding star, IR systems to meet every desire,
The aim has been to provide a space, where researchers could meet face to face,
To discuss their ideas for system creation, And the best ways to perform evaluation,
The results are clear for all to see, Multimedia and multilinguality, So much science from A to Z(ee),
Sharing ideas and sharing data, Such an experience couldnt have been greater,
Above all let me make it clear, The achievement to my heart most dear,
The result that was most important to me, The building of a strong vibrant community,
So there’s TREC and CLEF and NTCIR, But never forget there’s also FIRE.”
FIRE 2019 will be held in Kolkata, also known as ‘The city of joy’ from 12-15th December. The call for papers can be found at fire.irsi.res.in