Fresh back from Miyazaki, Japan and my mind is buzzing with new ideas about how language researchers can leverage the power of games to create and annotate language resources. The Games4NLP workshop aimed to promote and explore the possibilities for research and practical applications of using games and gamification for the creation of language resources for Natural Language Processing.
As one of the opening workshops of the Language Resources and Evaluation Conference (LREC), we were delighted to see a packed room of 50+ participants hoping to find out more about using games and gamification for NLP.
Keynote speaker Dr. Ivan Habernal discussed why he chose to research gamification, the current state of games for NLP, the challenges those games face, and how he approached those challenges in their open source game for gathering annotations of different fallacy types, Argotario. Ivan spoke of some negative experiences when surveying current NLP games including, not being well explained, having broken features, receiving in-game rewards without providing valid annotations and unconvincing in-game agents. He identified one of the challenges to be the difference of the speed of thinking they require. Games or fun activities often require fast paced thinking, whereas annotation tasks typically require slow thinking. Additionally, games for NLP present users with challenging annotation tasks, the answers to which, may be unclear, even to experts. Ivan suggested games for NLP should choose tasks that people are good at. In Argotario they teach users about the different kinds of fallacies, and have them label those fallacy types. Ivan suggested targeting mobile as the best choice of platform, adding a learning element to make games intrinsically motivating and leveraging users’ behaviour patterns for improved player retention.
Hend Al-Khalifa shared her team’s mobile game Faseeh. Faseeh is a serious game for language education. Targeted at children in and around the 10-15 year old range, Faseeh provides a learning experience for expanding the child’s vocabulary at the same time as collecting Arabic synonyms – a collection which Hend said no one else has previously gathered. Faseeh, aside from the name of the game, is also an in-game avatar. Children can challenge the in-game avatar in a different game mode. Gamification is woven into Faseeh through use of levels, a leaderboard and continuously unlocking information. Faseeh is available on multiple mobile app stores.
My personal contribution to the workshop was a report on a recent experiment on my game with a purpose, TileAttack. With the goal of reaching as broad an audience of players as possible, I compared the performance of the three separate audiences: paid crowdsource workers; gamers; and linguists. I also discussed the upcoming aggregation work to extract more value from the noisy annotation TileAttack is gathering.
Sarah Ita Levitan and James Shin, from the University Of Columbia, presented their work on deception detection. Motivated by the cost of expert annotation, and the slow speed and quality control problems of crowdsourcing, Sarah and James gather their annotations using LieCatcher, their game with a purpose. In LieCatcher, players are given a text question, and listen to a pre-recorded human response. Given the text question and the pre-recorded human response they have to decide whether the user was right or not by clicking the “TRUE” or “FALSE” buttons. In their study of 40 participants with pre and post surveys they achieved very positive results for ease of use.
On behalf of Alain Joubert et al, Karen Fort presented experiences from the past 10 years of Jeux De Mots. One of the first games with a purpose in the NLP domain, having gone live in September 2007, Jeux De Mots players have played over 30,000 hours and contributed 230 million lexical relations. Karen says that one of the benefits of using GWAPs in comparison to other methods of crowdsourcing, is the opportunity to train users about complex tasks. Those players can then go on to become experts. Karen said of Jeux De Mots, the contributions from players has a long tail. A few players do the bulk of the work. However, she said that they find it is better to find a few good players, a strategy that has evidently worked for Jeux De Mots with Karen reporting that over the 10 years of operation, some dedicated players have racked up over 3,000 hours of play time. The Jeux De Mots lexical network still continues to be regularly updated, free and downloadable today.
Following on from the previous talk, Karen discussed a series of voting games her team uses to complement Jeux De Mots. Collectively, they are known as, “A galaxy of voting games”. These games serve to enrich the network, clean the network, and drive new players to Jeux De Mots. With minimal effort their team can produce new voting games to target a specific area of interest for the Jeux De Mots lexical network. These include games like ColorIt, that enrich the network by adding new colour related relations (e.g. clinical -> white), or Emot that gathers emotion/sentiment relations by allowing players to click what a specific term makes them feel. Individually, these games have attracted player votes in the millions.
Before the project updates there were two lightning talks from participants of the workshop. The first was from Kevin Yansey from Waseda University who talked about work he was doing with a serious game for language learners of Korean. Leaning heavily on gamification principles, the interface has produced some interesting results that we look forward to hearing more about in the future. Patrick Littell from the National Research Centre of Canada put out a call for help with building mobile games for indigenous under-represented languages and dialects, indicating a real need for this approach to be made more mainstream.
Verena Lyding and Lionel Nicolas provided a project update of the enetCollect COST Action. This is an EU-funded effort to further collaboration between researchers and universities around an emerging theme, for example by providing reciprocal visits to research centres and to arrange group meetings. Considerable progress from the first year was reported, as well as plans for the future.
Chris Cieri from the Linguistic Data Consortium discussed multiple exciting projects they were working on. Their new game “Name that language” plays a short audio clip and players are asked to guess what the language is. This game is built on top of a framework that is heavily parameterised and allows extensive configuration making it easy to produce similar games. He also spoke of LingoBoingo, the new portal they have started specifically for GWAPs for NLP. They have been marketing the portal for around five months, starting with a linguistic audience and increasing to broader audiences. Chris concluded his talk with an invitation to anyone who’s creating a GWAP for language to contact them and potentially benefit from appearing on the new LingoBoingo platform.
The main objective according to the organisers of the workshop was to provide a forum for researchers and practitioners to discuss and share ideas regarding how the NLP research community can benefit from using game and gamification strategies. During the breaks (and the rest of the conference) it was clear that a community of researchers are using a gaming approach and it was great to hear about so many exciting projects and learn from the experiences of other teams.
The proceedings and presentations from the workshop are available online: http://anawiki.essex.ac.uk/dali/games4nlp/