One size does not fit all: Rethinking recognition system design for behaviorally heterogeneous online communities

https://doi.org/10.1016/j.im.2019.103245Get rights and content

Abstract

Online knowledge-sharing communities typically acknowledge their top contributors by implementing recognition systems. Extant recognition systems suffer from several limitations such as treatment of the entire community as homogeneous and inflexibility of customization. We propose a framework based on socio-technical design principles for designing a multi-criterion- and multi-segment-based recognition system that exploits multiple user characteristics for differential recognition according to community goals. We apply this framework on data gathered from Yelp.com and show how it can be used to recognize top members of different identified behavioral segments (amateurs, adepts, and enthusiasts) based on their performance on various relevant factors.

Introduction

Internet-enabled knowledge-sharing is evolving from the conventional ways of hosting anonymous user contribution toward a more community-based sharing model, where users maintain their social profiles and interact with other community members [1]. Different types of online communities have emerged, such as customer review communities (Amazon, Yelp, etc.), social Q&A forums (Quora, Yahoo Answers, etc.), etc. To survive and stay relevant, the natural inclination of these communities is to encourage higher contribution by their members (usually in terms of volume and quality) and to foster inter-member interaction and reciprocation [2,3]. To that end, community managers typically use the help of either one or both of the two, broad extrinsic motivating mechanisms – a recognition system such as ranks, badges, titles, etc., and a reward system such as offering discounts, gift vouchers, etc. [4,5]. In this study, we focus on the domain of recognition system, which is a critical aspect in the management of online knowledge-sharing communities.

Recognition systems have gained prominence in recent years with many popular communities implementing them in various ways. For example, the online travel review site TripAdvisor uses a hierarchical recognition system of providing various levels of badges (starting from “Reviewer” to “Top Contributor”), whereas Yelp selectively rewards yearly “Elite” status to users based on different aspects of community contribution. In addition to being sources of external motivation for the contributors, recognition systems also act as signals of contributor expertise and credibility for other users, and in turn help in building trust on the platform [6]. Existing literature reports contradictory findings regarding the efficacy of online recognition systems as extrinsic motivators, with some studies showing a positive influence of such systems on user contribution [[7], [8], [9]] and community commitment [10], and some reporting detrimental effects of the same [11,12]. Users contribute to online communities with an expectation of receiving recognition and appreciation for their efforts [13]. Research has shown that users’ motivation to participate increases when their contribution is duly valued [2,7]. Hence, it could be argued that poor design and ineffective management of recognition systems, resulting in an unfair selection of recognized members, may dissuade users to participate further. Thus, it is critical to tackle the unintended negative consequences and promote community contribution and trust on the platform by focusing on an effective design and implementation of recognition systems.

The design of online community recognition systems largely refers to the mechanism of their operationalization in different communities. The most important aspect of this is the user selection criteria i.e., how certain members are selected as the recipients of community recognition. Community behavior of members may consist of several aspects, only some of which may be considered for designing and implementing recognition systems. For example, positivity, involvement, experience, reputation, competence, sociability, etc., are some of the relevant reviewer characteristics in the context of online reviews [14]. An effective recognition system should ideally focus on fostering frequent and high-quality content contribution from the community members, and also on increasing the level of community commitment and camaraderie among the members [6]. However, in practice, online communities seldom design their recognition systems by considering the different aspects of contribution behavior. Some platforms encourage only a higher volume of contribution, while some focus only on the quality of the contribution, thus compromising on the overall value that can be derived by an effective recognition system. Recognition systems that are designed using only one parameter can be referred to as single-criterion recognition systems. For example, Mozilla Support Forum, Microsoft TechNet Forum, Discogs, etc. rank their contributors based on their quantity of contributions. TripAdvisor, on the other hand, has separate badges for high-volume contributors and high-quality contributors, which means that they value both quantity and quality, albeit rather disparately and can be considered as an example of using multiple single-criterion recognition systems. Other critical success factors of online platforms such as social interaction among the users [15], individuals’ community commitment, [16,17], etc. are typically not taken into consideration when developing a recognition system. Thus, by using single-criterion recognition systems, a community discourages diversity of behavioral patterns and biases its members toward acting out only a particular kind of behavior. This may lead to demotivation and eventually voluntary exit of those community members who are valuable contributors in some other aspects but fall behind in one that leads to recognition in the community. Hence, if online review communities can implement a multi-criterion recognition system for not only simultaneously promoting frequent and useful content but also facilitating social exchange and community commitment, it will be beneficial for community members. A good example of this is Yelp, which uses multiple parameters simultaneously for recognizing some members as Elite. However, it is not transparent and is perceived to be subjectively implemented by community managers, without a proper systematic mechanism.

It may be noted that both single-criterion and multi-criterion recognition systems suffer from one common flaw – they use a one-size-fits-all approach by treating the entire community as a homogeneous population. However, users in virtual communities typically differ in their behavioral patterns, e.g., some focus on frequent content generation, some concentrate on the quality of content, some engage more socially and appreciate others, some try to balance two or more aspects, etc. Generally, it can be expected that users will not always extremelyrate high on one behavioral criterion and extremely low on others, but rather exhibit certain combinations of the different criteria. Recognition systems should ideally be able to differentiate among various dominant user behaviors and customize the rewards and recognitions accordingly. The multiple single-criterion system addresses this issue to a certain extent by providing different recognitions for different behavioral criteria. However, it fails to appreciate the interplay and combinations among various behavioral parameters. For instance, it would recognize a member who rates high on quantity but low on all other factors, over those who are medium-high on all aspects including quantity. Another issue is that community members may be operating at different points of the learning curve. For instance, a new member of a community might not be able to match the level of contribution of a veteran. An approach to address these issues would be to segment the entire community into meaningful categories based on multiple relevant parameters and then distribute the recognitions according to the relative importance of various behavioral segments. A simple example could be to segment the community based on one parameter such as experience (say amateurs, pros, and veterans) and using a weighted average of multiple contribution factors to rank and recognize top contributors in each segment. A more intricate system may use multiple parameters for segmentation based on the requirements of the community. Such a multi-criterion segment-based recognition system (MSR system) can provide community managers an improved and comprehensive way to strategically tailor their recognitions and enable them to differentially encourage/discourage various combinations of community behavior based on community goals. While it might have been difficult to implement such a complex system a few years ago, the advent of contemporary data analytics and machine learning tools and techniques has greatly simplified the process. Time is now ripe to evolve from a simplistic recognition system to a more intricate, generic, and flexible MSR system.

Extending the arguments presented, it follows that virtual communities can benefit from a systematic way of designing an MSR system, which can use state-of-the-art machine learning techniques to semiautomate and enrich the process. However, existing literature does not suggest any objective approach for designing and managing such recognition systems, which propels us to take up the following research objectives:

Research objective I: To propose a general guiding framework for design and management of a multi-criterion segment-based system for online communities.

Research objective II: To demonstrate the applicability of the proposed framework by designing a recognition system for a real-life online review community.

We address the need for developing a decision support framework for community managers, which provides them the flexibility to include various user parameters and behavioral segments and to choose from multiple high-level recognition-distribution strategies in alignment with the needs of the community. Accordingly, we propose a five-step MSR system framework, based on socio-technical design principles and data analytics applications, which can be adopted by community managers. We demonstrate the applicability of our framework on Yelp.com, a popular review website having 77 million average monthly unique desktop visitors (in Q4 2017)1 . Essentially, our design proposes the following high-level steps for a community manager:

  • (i)

    identify relevant behavioral variables for community members

  • (ii)

    extract a few independent core factors

  • (iii)

    use machine learning classifiers to gauge the influence of each core factor on the recognition status, and assign factor weights

  • (iv)

    use clustering techniques to segment the community and assign segment weights

  • (v)

    use factor weights to rank members in each segment and then use segment weights to select a specific number of top-ranked members from each segment for recognition

Our proposed approach can be used to design a recognition system from scratch or upgrade existing recognition systems in online communities. It should be noted that the main purpose of our proposed approach is to provide community managers the flexibility and control to systematically distribute recognition across different behavioral segments, that can help in motivating those users who exhibit better behavioral outcomes, in alignment with the overall objectives of the community. Through the example of Yelp, we demonstrate the implementation of the proposed MSR system and empirically show that it is more effective than Yelp’s existing recognition system in recognizing reviewers who display better behavioral outcomes.

To the best of our knowledge, this is the first attempt to provide a systematic guiding framework to design and manage an MSR system, which enriches the body of literature pertaining to online communities. This is also the first attempt to use contributor segments based on behavioral clusters for community management. Although we demonstrate our framework in the context of an online reviewer community, Information Systems researchers can extend the generic framework with minor modifications to study other virtual platforms such as social networking sites, Q&A forums, etc. For practitioners, especially online community managers, this paper is a useful reference to assess and improve their existing recognition system or develop a recognition system from scratch.

The remainder of the paper is organized as follows: Section 2 explores the background literature pertaining to the area. Section 3 provides the theoretical background for this research. Section 4 presents and explicates the five-step framework for designing an MSR system. Section 5 demonstrates the application of the guidelines in the context of Yelp. Section 6 discusses the academic and managerial implications, limitations of the study, and future research directions, and section 7 concludes the paper.

Section snippets

Influence of recognition systems on community behavior

Extant literature on online community recognition systems has primarily focused on evaluating the efficacy of online recognition systems on community behavior [5,8,9,11,18]. Studies have found that in most cases such systems are useful for driving sustained content contribution, higher quality content, and increased user engagement and commitment [5,8,9,18]. However, some studies have also reported unintended negative consequences of recognition systems where badges, ranks, etc. undermine

Socio-technical principles of system design

The theoretical foundation of our paper is based on the socio-technical principles of system design. The underlying philosophy of socio-technical system design is that system design should capture the interaction of human factors (social or organizational) and technical factors because both kinds of factors influence the usage and functionality of any information system. The socio-technical approach of system design leads to outcomes in alignment with organizational goals, better control over

A five-step design framework

The proposed five-step design framework provides generalized guidelines for community managers to design an MSR system based on community goals. The framework enables community managers to make use of the accessible user data pertaining to their community behavior to systematically select deserving members for recognizing them. Fig. 1 shows the basic steps of the framework.

The framework adheres to the principles of socio-technical system design as discussed earlier. On the basis of the

Application of the framework

Yelp hosts millions of reviews for local businesses in several metro cities across multiple countries written by thousands of reviewers. At the same time, it maintains an online community of reviewers who not only share reviews but also constitute an active social network. It provides community recognition to selected reviewers every year by conferring them with the Elite title based on their contribution and community behavior. Users can nominate themselves or others for receiving the Elite

Discussion – implications and limitations

This study proposes a generalized five-stage guiding framework to design and manage MSR systems for online communities and demonstrates how it can be implemented by community managers by using real data from Yelp. While our system empirically showed significantly better results in terms of selecting candidates for recognition in Yelp, it is not our intention to claim that our proposed approach is superior to the existing systems at Yelp. This is because of the fact that any assessment (either

Conclusion

Recognition systems play a pivotal role in ensuring the success of online communities by retaining contributors, encouraging them to contribute more and useful content, and signaling their expertise and credibility. While existing studies agree on the importance of a well-designed recognition system for the sustenance of online platforms, this paper is the first one to put forth a novel approach of systematically designing recognition systems based on socio-technical design principles. We use

Author contribution

The authors do not wish to include any author contribution statement for the paper.

Samadrita Bhattacharyya is a doctoral student of Management Information Systems at the Indian Institute of Management Calcutta. She holds B.Tech in Electronics and Communication Engineering from West Bengal University of Technology and M.Tech in VLSI Design from Indian Institute of Engineering Science and Technology, Shibpur (formerly Bengal Engineering and Science University, Shibpur). Her research interests include social commerce, online reviews, online communities, data analytics, and

References (61)

  • S. Okazaki

    What do we know about mobile Internet adopters? A cluster analysis

    Inf. Manag.

    (2006)
  • F. Hamka et al.

    Mobile customer segmentation based on smartphone measurement

    Telemat. Inform.

    (2014)
  • J.W. van Dam et al.

    Online profiling and clustering of Facebook users

    Decis. Support Syst.

    (2015)
  • J.G. Dias et al.

    Latent class modeling of website users’ search patterns: implications for online market segmentation

    J. Retail. Consum. Serv.

    (2007)
  • H.H. Chang et al.

    Social capital and individual motivations on knowledge sharing: participant involvement as a moderator

    Inf. Manag.

    (2011)
  • I. Kang et al.

    Investigation of online community voluntary behavior using cognitive map

    Comput. Human Behav.

    (2007)
  • R. Houari et al.

    Dimensionality reduction in data mining: a Copula approach

    Expert Syst. Appl.

    (2016)
  • Z. Liu et al.

    What makes a useful online review? Implication for travel product websites

    Tour. Manag.

    (2015)
  • H.F. Lin

    Determinants of successful virtual communities: contributions from system characteristics and social factors

    Inf. Manag.

    (2008)
  • P.J. Rousseeuw

    Silhouettes: a graphical aid to the interpretation and validation of cluster analysis

    J. Comput. Appl. Math.

    (1987)
  • I. Fernando

    Community creation by means of a social media paradigm

    Learn. Organ.

    (2010)
  • R.E. Kraut et al.

    Building Successful Online Communities: Evidence-Based Social Design

    (2012)
  • R. Gazan

    Social Q & amp;A

    J. Am. Soc. Inf. Sci. Technol.

    (2011)
  • C. Dellarocas

    Online reputation systems: how to design one that does what you need

    MIT Sloan Management Review.

    (2010)
  • Z. Li et al.

    Quantifying the impact of badges on user engagement in online Q&A communities

  • L. Khansa et al.

    Understanding members’ active participation in online Question-and-Answer communities: a theory and empirical analysis

    J. Manag. Inf. Syst.

    (2015)
  • M. Schuckert et al.

    Stars, votes, and badges: how online badges affect hotel reviewers

    J. Travel Tour. Mark.

    (2016)
  • L. Zhao et al.

    Sharing knowledge in social Q&A sites: the unintended consequences of extrinsic motivation

    J. Manag. Inf. Syst.

    (2016)
  • J. Lampel et al.

    The role of status seeking in online communities: giving the gift of experience

    J. Comput. Commun.

    (2007)
  • P.J. Bateman et al.

    The impact of community commitment on participation in online communities

    Inf. Syst. Res.

    (2011)
  • Cited by (7)

    • Administrator-users contribute more to online communities

      2022, Information and Management
      Citation Excerpt :

      Collectively, our study synthesized the literature to create a typology of three positive community behaviors, namely public space contribution, private space contribution, and OL. The online community literature assumes that users are typically community participants, while some can be “top members”, basing on their performance [43]. However, past research has not considered how users who serve in administrator roles may behave differently from other users, showing a research gap.

    • Recognise me or pay me? How rewards for online restaurant reviews impact continuity: A cross-country investigation

      2022, International Journal of Hospitality Management
      Citation Excerpt :

      The literature suggests consumers are incentivized to share consumption-related information in online communities, such as online reviews, not only for social capital (e.g., status) but also economic capital (e.g., monetary value) (Husemann, 2012); in the current study, both are examined via recognition rewards and monetary rewards. The first type of reward examined in this study, recognition rewards, are defined as social-capital-focused incentives (e.g., badges and ranking) and enhance the reviewer’s social recognition (Bhattacharyya et al., 2019). Given tourism and hospitality review sites, such as TripAdvisor use recognition rewards as mechanisms to increase reviewers’ contributions in posting reviews (Moro et al., 2019), such rewards are particularly relevant to this study.

    • Impact of mobile technology-enabled HR gamification on employee performance: An empirical investigation

      2022, Information and Management
      Citation Excerpt :

      For example, Goes et al. [31] study the effect of incentive hierarchies on users’ contributions to a user-generated content context. Bhattacharyya et al. [10] provide guidelines about how to implement a recognition system in online communities. Holzer et al. [38] study the effect of feedback on a knowledge management system, and Sheffler et al. [81] examined how different badge designs affect bicycle ridership.

    View all citing articles on Scopus

    Samadrita Bhattacharyya is a doctoral student of Management Information Systems at the Indian Institute of Management Calcutta. She holds B.Tech in Electronics and Communication Engineering from West Bengal University of Technology and M.Tech in VLSI Design from Indian Institute of Engineering Science and Technology, Shibpur (formerly Bengal Engineering and Science University, Shibpur). Her research interests include social commerce, online reviews, online communities, data analytics, and optimization and algorithms. Her research articles have appeared in Decision Support Systems and proceedings of Workshop of e-Business, Australasian Conference on Information Systems, and IEEE.

    Shankhadeep Banerjee is a doctoral student of Management Information Systems at the Indian Institute of Management Calcutta (IIMC). He holds an MBA from IIMC, an IMP certificate from NEOMA Business School, France, and a B.Tech in Computer Science and Engineering from National Institute of Technology Durgapur. His research interests are primarily related to human behavior around contemporary technologies like crowdfunding, online reviews, virtual communities, e-commerce adoption, etc. He has prior IS research experience at Indian School of Business Hyderabad, and also has extensive IS practitioner experience working at top technology firms like Microsoft, Amazon, and eBay. His research articles have appeared in Decision Support Systems journal, and in the proceedings of several esteemed conferences including International Conference on Information Systems and Australasian Conference on Information Systems.

    Indranil Bose is Professor of Management Information Systems at the Indian Institute of Management, Calcutta. He acts as Coordinator of IIMC Case Research Center. He holds a B. Tech. from the Indian Institute of Technology, MS from the University of Iowa, MS and Ph.D. from Purdue University. His research interests are in business analytics, telecommunications, information security, and supply chain management. His publications have appeared in MIS Quarterly, Communications of the ACM, Communications of AIS, Computers and Operations Research, Decision Support Systems, Ergonomics, European Journal of Operational Research, Information & Management, International Journal of Production Economics, Journal of Organizational Computing and Electronic Commerce, Journal of the American Society for Information Science and Technology, Operations Research Letters etc. He serves as Senior Editor of Decision Support Systems and as Associate Editor of Information & Management, Communications of AIS, Information Technology & Management and member of the editorial board for Journal of AIS.

    View full text