Games4NLP was an independent symposium co-located with the 15th European Chapter of the Association for Computational Linguistics (EACL) Conference that took place on 4 April 2017 in Valencia, Spain.
The symposium aimed to promote and explore the possibilities for research and practical applications of using games and gamification for Natural Language Processing (NLP). The main objective was to provide a forum for researchers and practitioners to discuss and share ideas regarding how the NLP research community can benefit from using game/gamification strategies. For example, games can be used to collect large numbers of annotations of human language provided that there are sufficient numbers of players who are motivated to play, and these annotations, when aggregated, can be used as labels that replace or compliment the effort of expert annotators.
In the first session games that were under development were presented. Michael Föls presented the uComp Language Quiz, a multilingual platform for gathering natural language resources. Users are recruited via the CrowdFlower platform through financial incentivisation, but continue to play and contribute to a variety of tasks in a variety of languages. Chris Madge presented TileAttack, a game-with-a-purpose approach to annotating text segmentation that also sets out to study the effect of using different game mechanics in that domain. Kevin Forand presented three separate games with a purpose for knowledge base curation. These include a pinball game, and a balloon game. The games are being used on top of an information extraction pipeline to filter out noisy triples.
The second session focused on player motivation. Bruno Guillaume discussed the results of a player survey carried out on their ZombiLingo game (a GWAP for dependency syntax annotation). Surveys on GWAP players are something of a rarity and the ZombiLingo team find some interesting trends. Alice Millour presented Bisame, a gamified crowdsourcing platform for part of speech annotations for the Alsatian language. Jason Radford discussed his exploration of user behaviour in games for science and the challenges in quality of participation in large paid subject pools that many services now offer.
Dagmara Dziedzic and Wojciech Włodarczyk take a different approach to motivating contribution in through use of games, by applying the Free-to-Play mechanic in their game RoboCorp. Rather than presenting the task in its entirety as the game, RoboCorp presents a game in which the player performs the task in a mini-game to enhance their main game experience. This is similar to the payments made by players in Free-to-Play games, but players contribute work instead of play. Their work also tied into the theme of the final session: methods for evaluating game performance. Jon Chamberlain discussed the adaptation of Free-to-Play metrics for evaluating games-with-a-purpose and, finally, Pontus Stenetorp presented the lessons learned from the first version of Word Sheriff, and plans to incorporate social features and target the mobile platform in version 2.
During the panel session (and over lunch) lively discussions were had regarding using a gaming approach for language processing and resource creation. To break the ice all presenters were asked for their reasons for choosing games with a purpose. For Bruno Guillaume from the ZombiLingo team, one of those reasons was the challenge. Pontus Stenetorp from the WordSheriff team mentioned the link between how languages are about communication, and how games can also be about communication. Jon Chamberlain spoke about how games are effective in encouraging collaboration and design opportunities that match the semi structured nature of language.
One of the key questions for a games-based approach is whether the participants want to play the game or whether they want to help with the research that is behind the game. Games like Phrase Detectives have a hardcore community of players who are driven to participate because they enjoy being part of the language research, and the interface amplifies that enjoyment with leaderboards, high scores, inter-player competitiveness etc. On the other hand, RoboCorp is designed specifically to be an enjoyable platform-type game, but in order to play the game the participants must complete language tasks.
The conversation then shifted to the opportunities games provide for the study of language, including language that is scraped from in game discussions. It was proposed that a larger online presence should be created to discuss and advertise games of this nature, including a specific subreddit on the popular bulletin board like website, Reddit. Massimo Poesio mentioned the plans of the Linguistic Data Consortium (LDC) to set up a portal for this very purpose. It was questioned whether such collections of games may dilute player bases and suggested that we gain a greater understanding of what motivates the players to play to mitigate this and similar problems.
Many other experiences were shared by the presenters and one that stood out was from Wojciech Włodarczyk who described a simple game he built called “Watching Grass Grow”. The game was a single image of a lawn that never changed but he reported that some people played the game for several hours waiting, and hoping, that something might happen. The use of “cheap psychological tricks”, as described to vivid effect by Richard Bartle in his book “Designing Virtual Worlds“, may be an important method to engage people in games for language processing and other crowdsourcing tasks.