The 1st Gamification in Information Retrieval (GamifIR) workshop was held at ECIR 2014 on April 13 in Amsterdam. The workshop focused on the challenges and opportunities that gamification may present for the information retrieval (IR) community.
Gamification, which is the application of game elements and mechanics in non-gaming environments with the aim to increase user engagement, data quality or cost effectiveness, is rapidly moving beyond being a buzzword and becoming a key strategy for a wide range of applications. A core aspect of gamification solutions is to infuse intrinsic motivations to participate by leveraging people’s natural desires for achievement and competition. While gamification, on the one hand, is emerging as the next big thing in industry, e.g., an effective way to generate business, on the other hand, it is also becoming a major research area. However, its adoption in IR is still in its infancy, despite the wide ranging IR tasks that may benefit from gamification techniques. These include the manual annotation of documents for IR evaluation, the participation in user studies to study interactive IR challenges, or the shift from single-user search to social search, just to mention a few.
The purpose of the GamifIR workshop was to bring together researchers and practitioners from a wide range of areas including game design, IR, human-computer interaction, computer games, and natural language processing in order to start a discussion and an exchange of research ideas and results relating to emerging areas of gamification within the context of IR.
The workshop was organised by Frank Hopfgartner (TU Berlin, Germany), Gabriella Kazai (Microsoft, UK and US), Udo Kruschwitz (University of Essex ,UK) and Michael Meder (TU Berlin, Germany). In response to the call for papers, 18 submissions were received, out of which 14 were accepted for presentation at the workshop. Over 40 people attended the workshop, representing both industry and academia. The program of the day has been structured into three sessions: keynote, poster presentations and interactive discussion (a’la fishbowl style).
The keynote address was given by Prof. Richard Bartle. Richard is a Senior Lecturer and an Honorary Professor of Computer Game Design at the University of Essex, UK. He is well known for his Player Types model, which has seen widespread adoption by the massively multiplayer online game (MMO) industry. He was also the co-writer of the first ever virtual world, MUD, in 1978. His book “Designing Virtual Worlds” (2003) is the standard text on the subject and he is an influential writer on all aspects of MMO design and development. In 2010, he was the first recipient of the prestigious GDC “Online Game Legend” award.
In his keynote address, titled “Information Reconstruction: Unpicking the GamifIR Call for Papers”, Richard launched into a very deliberate, deeply serious while superbly entertaining dissection of the workshop’s call for papers (CFP) text. He started by a seriously playful comment on the fact that he is not in fact an expert in IR, nor in gamification, but, that after reading the CFP, he realised that he is also not alone in that regard. This is in part because of the emerging gamification community’s “play” on the very meaning of the term “gamification”. Gamification used to mean the act of turning things that weren’t games into games, but it now covers the somewhat paradoxical attempt of making things with games into not-games.
Another confounding factor that is adding to the cross-community confusion between the fields of game design and gamification is the somewhat interchangeable use of the terms “game elements” and “game mechanics”. He argued that while gamification currently focuses on the application of game elements, such as leader boards, badges or achievement points, it has yet to explore the role of true game mechanics, which represent the “moving parts” or the dynamic processes of a system. As an example of a real game mechanic, Richard referred to the Indian lottery, where the winning number is the one that the least number of people bet on. In contrast, the common parts in most gamification applications are only the tokens, e.g., badges, which are simply the things (nouns) that a game operates on (e.g., infantry, cannons). However, you also need rules (e.g., cannon kills infantry, cannons create holes, infantry can hide in holes), the interaction of which leads to game features (e.g., investing in infantry or cannons), the interaction of which in turn defines the game play (e.g., strategy).
On the other hand, Richard made the point that gamification is actually more akin to gaming as opposed to games. For example, how casinos plan their layout to maximise income from passers-by as opposed to how game designers go about developing the next version of Grand Theft Auto. Another parallel with gaming is the value of token. In casinos, tokens can be exchanged for money, but without that they are worthless. Similarly, in gamification, the points or badges are worthless unless people associate value to them. The most common value is represented by tangible goods and, thus, most corporate gamification applications are essentially forms of “bribes”, where users are offered discounts, cash or goods in return for their accumulated points (e.g., loyalty programs). However, when everyone starts using gamification, the value of such rewards may diminish and can no longer be considered in isolation from others on the “game” market.
Aside from monetary incentives, gamification aims to infuse intrinsic motivations for participation. The two relevant intrinsic motivations are: fun (participate for the sake of enjoyment) and worthiness (part taking because the subject or the outcome is of interest to the user). Richard argued this to be a more promising direction, but noted that most commercial gamification applications use neither of these. Another key point of intrinsic motivations is that they supposed to work by leveraging people’s natural desires for achievement. However, not everyone who plays for fun has a desire to achieve. Indeed, Richard’s research showed that in MMOs there are four main types of people who play for fun: 1) achievers, 2) socialisers, 3) explorers, and 4) killers (where each type can be further broken down to two subtypes). He claimed that gamification applications that only cater for achievers are actually more likely to fail since people at the bottom of a leader board, i.e., the failed achievers, are more likely to leave the system (essentially moving the bottom of the list up). At the same time, achievers needs to believe that the tasks they accomplish are too challenging for others, i.e., there is a need to have people below them on the leader board. This can be solved by catering for other types of players who can then provide an “insulating layer” on the leader board. Thus, a successful gamification design should consider different rewards for the different types of players. For example, socialisers don’t mind if they are not at the top of the leader board while they also like to help others. Rewarding them for such a behaviour will also boost the health of the overall system.
However, it is not enough to aim to provide fun for different types of users. In particular, a lot of gamified applications focus on the competition aspect of fun. Yet, people only really like competition when they win. The rest of the potentially millions of participants may then become missed opportunities. Typical current methods to deal with such situations include the periodic clearing of the leader boards or partitioning them (e.g., only showing friends). Other methods may employ so called “cheap psychological tricks” such as variable rate reinforcement. Indeed, Richard has in fact made the specific point to say that gamification is in truth a branch of psychology, and not game design, as it involves strategies for influencing people in some ways.
He closed his keynote address by commenting that many problems in the field of IR do seem well suited for gamification and that IR as a discipline to adopt gamification is ahead of the curve, possibly because IR has a synergy with gamification at some level. However, while there are great opportunities in this space, there is still much more as-yet undiscovered territory that may be best explored by embracing not just game elements, but game mechanics, player types and psychology
(and the art of manipulation).
In the poster session, the 14 accepted papers were presented by their authors. The papers covered a wide range of topics from zombified dependency syntax annotation to gamified search tasks to study users’ browsing behaviour and from crowd-powered medical experts to mood based wardrobe recommendations. Out of the 14 posters, four were position papers, presenting reviews of the current state of the art and of possible research directions on how game elements and game mechanics may best be utilised in IR tasks [2,9,10,12], e.g., to improve users’ search experience [10]. The rest of the 11 research papers clustered around several key themes: 1) gamification of crowdsourcing tasks with the aim to increase quality or productivity [1,3,4], 2) game design and turning tasks into actual games [7,8,13], 3) user studies of incentives [6] and behaviour [11,14].
Finally, the discussion session covered several areas of interest, such as 1) the conflict between art and computer science as two different fields with very different views on game design, 2) the conflict between designing games from an egotistical point of view vs designing for all audiences, 3) how search is an exchange of values or what does search as economy mean, 4) how to motivate searchers to explicate their intent or how to support annotators in understanding user motivations, 5) how to design real time games or real time gamified relevance judging interfaces, 6) how can games improve learning experiences in education, 7) how can we go beyond game elements onto adding a layer of actionable context, and 8 ) how to deal with the biased sample of population that gamification leads to.
The workshop was closed by the organisers summing up the day of events and with the final take home message urging the community to take the lead on gamification research and capitalise on the benefits it can offer for IR. Finally, based on the interest received from the general community and the feedback given by the attendees, there seems a genuine need for the workshop being run again next year. If you would like to be involved in any shape or form, we would love to hear from you!
The Proceedings of the workshop is available via the ACM DL.
References
[1] Karën Fort, Bruno Guillaume, and Hadrien Chastant. Creating Zombilingo, a game with a purpose for dependency syntax annotation.
[2] Luca Galli, Piero Fraternali, and Alessandro Bozzon. On the application of game mechanics in information retrieval.
[3] Jon Chamberlain. The annotation-validation (AV) model: rewarding contribution using retrospective agreement.
[4] Christopher G. Harris. The beauty contest revisited: measuring consensus rankings of relevance using a game.
[5] Markus Brenner, Navid Mirza, and Ebroul Izquierdo. People recognition using gamified ambiguous feedback.
[6] Dinesh Pothineni, Pratik Mishra, Aadil Rasheed, and Deepak Sundararajan. Incentive design to mould online behavior: a game mechanics perspective.
[7] Carlos Maltzahn, Arnav Jhala, Michael Mateas, and Jim Whitehead. Gamification of private digital data archive management.
[8] Leif Azzopardi, Martin Bevc, Andrew Gardner, David Maxwell, and Abdullah Razzouk. PageFetch 2: gamification the sequel.
[9] Juan M. Fernández-Luna, Juan F. Huete, Humberto Rodríguez-Avila, and Julio C. Rodríguez-Cano. Enhancing collaborative search systems engagement through gamification.
[10] Mark Shovman. The game of search: what is the fun in that?
[11] Jiyin He, Marc Bron, Leif Azzopardi, and Arjen de Vries. Studying user browsing behavior through gamified search tasks.
[12] Carsten Eickhoff. Crowd-powered experts: helping surgeons interpret breast cancer images.
[13] Mathias Lux, Mario Guggenberger, and Michael Riegler. PictureSort: gamification of image ranking.
[14] Bojana Dumeljic, Martha Larson, and Alessandro Bozzon. Moody closet: exploring intriguing new views on wardrobe recommendation.