One size does not fit all: Rethinking recognition system design for behaviorally heterogeneous online communities
Introduction
Internet-enabled knowledge-sharing is evolving from the conventional ways of hosting anonymous user contribution toward a more community-based sharing model, where users maintain their social profiles and interact with other community members [1]. Different types of online communities have emerged, such as customer review communities (Amazon, Yelp, etc.), social Q&A forums (Quora, Yahoo Answers, etc.), etc. To survive and stay relevant, the natural inclination of these communities is to encourage higher contribution by their members (usually in terms of volume and quality) and to foster inter-member interaction and reciprocation [2,3]. To that end, community managers typically use the help of either one or both of the two, broad extrinsic motivating mechanisms – a recognition system such as ranks, badges, titles, etc., and a reward system such as offering discounts, gift vouchers, etc. [4,5]. In this study, we focus on the domain of recognition system, which is a critical aspect in the management of online knowledge-sharing communities.
Recognition systems have gained prominence in recent years with many popular communities implementing them in various ways. For example, the online travel review site TripAdvisor uses a hierarchical recognition system of providing various levels of badges (starting from “Reviewer” to “Top Contributor”), whereas Yelp selectively rewards yearly “Elite” status to users based on different aspects of community contribution. In addition to being sources of external motivation for the contributors, recognition systems also act as signals of contributor expertise and credibility for other users, and in turn help in building trust on the platform [6]. Existing literature reports contradictory findings regarding the efficacy of online recognition systems as extrinsic motivators, with some studies showing a positive influence of such systems on user contribution [[7], [8], [9]] and community commitment [10], and some reporting detrimental effects of the same [11,12]. Users contribute to online communities with an expectation of receiving recognition and appreciation for their efforts [13]. Research has shown that users’ motivation to participate increases when their contribution is duly valued [2,7]. Hence, it could be argued that poor design and ineffective management of recognition systems, resulting in an unfair selection of recognized members, may dissuade users to participate further. Thus, it is critical to tackle the unintended negative consequences and promote community contribution and trust on the platform by focusing on an effective design and implementation of recognition systems.
The design of online community recognition systems largely refers to the mechanism of their operationalization in different communities. The most important aspect of this is the user selection criteria i.e., how certain members are selected as the recipients of community recognition. Community behavior of members may consist of several aspects, only some of which may be considered for designing and implementing recognition systems. For example, positivity, involvement, experience, reputation, competence, sociability, etc., are some of the relevant reviewer characteristics in the context of online reviews [14]. An effective recognition system should ideally focus on fostering frequent and high-quality content contribution from the community members, and also on increasing the level of community commitment and camaraderie among the members [6]. However, in practice, online communities seldom design their recognition systems by considering the different aspects of contribution behavior. Some platforms encourage only a higher volume of contribution, while some focus only on the quality of the contribution, thus compromising on the overall value that can be derived by an effective recognition system. Recognition systems that are designed using only one parameter can be referred to as single-criterion recognition systems. For example, Mozilla Support Forum, Microsoft TechNet Forum, Discogs, etc. rank their contributors based on their quantity of contributions. TripAdvisor, on the other hand, has separate badges for high-volume contributors and high-quality contributors, which means that they value both quantity and quality, albeit rather disparately and can be considered as an example of using multiple single-criterion recognition systems. Other critical success factors of online platforms such as social interaction among the users [15], individuals’ community commitment, [16,17], etc. are typically not taken into consideration when developing a recognition system. Thus, by using single-criterion recognition systems, a community discourages diversity of behavioral patterns and biases its members toward acting out only a particular kind of behavior. This may lead to demotivation and eventually voluntary exit of those community members who are valuable contributors in some other aspects but fall behind in one that leads to recognition in the community. Hence, if online review communities can implement a multi-criterion recognition system for not only simultaneously promoting frequent and useful content but also facilitating social exchange and community commitment, it will be beneficial for community members. A good example of this is Yelp, which uses multiple parameters simultaneously for recognizing some members as Elite. However, it is not transparent and is perceived to be subjectively implemented by community managers, without a proper systematic mechanism.
It may be noted that both single-criterion and multi-criterion recognition systems suffer from one common flaw – they use a one-size-fits-all approach by treating the entire community as a homogeneous population. However, users in virtual communities typically differ in their behavioral patterns, e.g., some focus on frequent content generation, some concentrate on the quality of content, some engage more socially and appreciate others, some try to balance two or more aspects, etc. Generally, it can be expected that users will not always extremelyrate high on one behavioral criterion and extremely low on others, but rather exhibit certain combinations of the different criteria. Recognition systems should ideally be able to differentiate among various dominant user behaviors and customize the rewards and recognitions accordingly. The multiple single-criterion system addresses this issue to a certain extent by providing different recognitions for different behavioral criteria. However, it fails to appreciate the interplay and combinations among various behavioral parameters. For instance, it would recognize a member who rates high on quantity but low on all other factors, over those who are medium-high on all aspects including quantity. Another issue is that community members may be operating at different points of the learning curve. For instance, a new member of a community might not be able to match the level of contribution of a veteran. An approach to address these issues would be to segment the entire community into meaningful categories based on multiple relevant parameters and then distribute the recognitions according to the relative importance of various behavioral segments. A simple example could be to segment the community based on one parameter such as experience (say amateurs, pros, and veterans) and using a weighted average of multiple contribution factors to rank and recognize top contributors in each segment. A more intricate system may use multiple parameters for segmentation based on the requirements of the community. Such a multi-criterion segment-based recognition system (MSR system) can provide community managers an improved and comprehensive way to strategically tailor their recognitions and enable them to differentially encourage/discourage various combinations of community behavior based on community goals. While it might have been difficult to implement such a complex system a few years ago, the advent of contemporary data analytics and machine learning tools and techniques has greatly simplified the process. Time is now ripe to evolve from a simplistic recognition system to a more intricate, generic, and flexible MSR system.
Extending the arguments presented, it follows that virtual communities can benefit from a systematic way of designing an MSR system, which can use state-of-the-art machine learning techniques to semiautomate and enrich the process. However, existing literature does not suggest any objective approach for designing and managing such recognition systems, which propels us to take up the following research objectives:
Research objective I: To propose a general guiding framework for design and management of a multi-criterion segment-based system for online communities.
Research objective II: To demonstrate the applicability of the proposed framework by designing a recognition system for a real-life online review community.
We address the need for developing a decision support framework for community managers, which provides them the flexibility to include various user parameters and behavioral segments and to choose from multiple high-level recognition-distribution strategies in alignment with the needs of the community. Accordingly, we propose a five-step MSR system framework, based on socio-technical design principles and data analytics applications, which can be adopted by community managers. We demonstrate the applicability of our framework on Yelp.com, a popular review website having 77 million average monthly unique desktop visitors (in Q4 2017)1 . Essentially, our design proposes the following high-level steps for a community manager:
- (i)
identify relevant behavioral variables for community members
- (ii)
extract a few independent core factors
- (iii)
use machine learning classifiers to gauge the influence of each core factor on the recognition status, and assign factor weights
- (iv)
use clustering techniques to segment the community and assign segment weights
- (v)
use factor weights to rank members in each segment and then use segment weights to select a specific number of top-ranked members from each segment for recognition
Our proposed approach can be used to design a recognition system from scratch or upgrade existing recognition systems in online communities. It should be noted that the main purpose of our proposed approach is to provide community managers the flexibility and control to systematically distribute recognition across different behavioral segments, that can help in motivating those users who exhibit better behavioral outcomes, in alignment with the overall objectives of the community. Through the example of Yelp, we demonstrate the implementation of the proposed MSR system and empirically show that it is more effective than Yelp’s existing recognition system in recognizing reviewers who display better behavioral outcomes.
To the best of our knowledge, this is the first attempt to provide a systematic guiding framework to design and manage an MSR system, which enriches the body of literature pertaining to online communities. This is also the first attempt to use contributor segments based on behavioral clusters for community management. Although we demonstrate our framework in the context of an online reviewer community, Information Systems researchers can extend the generic framework with minor modifications to study other virtual platforms such as social networking sites, Q&A forums, etc. For practitioners, especially online community managers, this paper is a useful reference to assess and improve their existing recognition system or develop a recognition system from scratch.
The remainder of the paper is organized as follows: Section 2 explores the background literature pertaining to the area. Section 3 provides the theoretical background for this research. Section 4 presents and explicates the five-step framework for designing an MSR system. Section 5 demonstrates the application of the guidelines in the context of Yelp. Section 6 discusses the academic and managerial implications, limitations of the study, and future research directions, and section 7 concludes the paper.
Section snippets
Influence of recognition systems on community behavior
Extant literature on online community recognition systems has primarily focused on evaluating the efficacy of online recognition systems on community behavior [5,8,9,11,18]. Studies have found that in most cases such systems are useful for driving sustained content contribution, higher quality content, and increased user engagement and commitment [5,8,9,18]. However, some studies have also reported unintended negative consequences of recognition systems where badges, ranks, etc. undermine
Socio-technical principles of system design
The theoretical foundation of our paper is based on the socio-technical principles of system design. The underlying philosophy of socio-technical system design is that system design should capture the interaction of human factors (social or organizational) and technical factors because both kinds of factors influence the usage and functionality of any information system. The socio-technical approach of system design leads to outcomes in alignment with organizational goals, better control over
A five-step design framework
The proposed five-step design framework provides generalized guidelines for community managers to design an MSR system based on community goals. The framework enables community managers to make use of the accessible user data pertaining to their community behavior to systematically select deserving members for recognizing them. Fig. 1 shows the basic steps of the framework.
The framework adheres to the principles of socio-technical system design as discussed earlier. On the basis of the
Application of the framework
Yelp hosts millions of reviews for local businesses in several metro cities across multiple countries written by thousands of reviewers. At the same time, it maintains an online community of reviewers who not only share reviews but also constitute an active social network. It provides community recognition to selected reviewers every year by conferring them with the Elite title based on their contribution and community behavior. Users can nominate themselves or others for receiving the Elite
Discussion – implications and limitations
This study proposes a generalized five-stage guiding framework to design and manage MSR systems for online communities and demonstrates how it can be implemented by community managers by using real data from Yelp. While our system empirically showed significantly better results in terms of selecting candidates for recognition in Yelp, it is not our intention to claim that our proposed approach is superior to the existing systems at Yelp. This is because of the fact that any assessment (either
Conclusion
Recognition systems play a pivotal role in ensuring the success of online communities by retaining contributors, encouraging them to contribute more and useful content, and signaling their expertise and credibility. While existing studies agree on the importance of a well-designed recognition system for the sustenance of online platforms, this paper is the first one to put forth a novel approach of systematically designing recognition systems based on socio-technical design principles. We use
Author contribution
The authors do not wish to include any author contribution statement for the paper.
Samadrita Bhattacharyya is a doctoral student of Management Information Systems at the Indian Institute of Management Calcutta. She holds B.Tech in Electronics and Communication Engineering from West Bengal University of Technology and M.Tech in VLSI Design from Indian Institute of Engineering Science and Technology, Shibpur (formerly Bengal Engineering and Science University, Shibpur). Her research interests include social commerce, online reviews, online communities, data analytics, and
References (61)
- et al.
An empirical study of the motivations for content contribution and community participation in Wikipedia
Inf. Manag.
(2015) - et al.
Why users contribute knowledge to online communities: an empirical study of an online social Q&A community
Inf. Manag.
(2015) Do badges increase user activity? A field experiment on the effects of gamification
Comput. Human Behav.
(2017)- et al.
Perceived online community support, member relations, and commitment: differences between posters and lurkers
Inf. Manag.
(2017) - et al.
Whose online reviews to trust? Understanding reviewer trustworthiness and its impact on business
Decis. Support Syst.
(2017) - et al.
Relationships among community interaction characteristics, perceived benefits, community commitment, and oppositional brand loyalty in online brand communities
Int. J. Inf. Manage.
(2013) - et al.
How to design gamification? A method for engineering gamified software
Inf. Softw. Technol.
(2018) Increasing participation in online communities: a framework for human-computer interaction
Comput. Human Behav.
(2007)- et al.
A reference model for reputation systems
Decis. Support Syst.
(2014) Sociotechnical principles for system design
Appl. Ergon.
(2000)
What do we know about mobile Internet adopters? A cluster analysis
Inf. Manag.
Mobile customer segmentation based on smartphone measurement
Telemat. Inform.
Online profiling and clustering of Facebook users
Decis. Support Syst.
Latent class modeling of website users’ search patterns: implications for online market segmentation
J. Retail. Consum. Serv.
Social capital and individual motivations on knowledge sharing: participant involvement as a moderator
Inf. Manag.
Investigation of online community voluntary behavior using cognitive map
Comput. Human Behav.
Dimensionality reduction in data mining: a Copula approach
Expert Syst. Appl.
What makes a useful online review? Implication for travel product websites
Tour. Manag.
Determinants of successful virtual communities: contributions from system characteristics and social factors
Inf. Manag.
Silhouettes: a graphical aid to the interpretation and validation of cluster analysis
J. Comput. Appl. Math.
Community creation by means of a social media paradigm
Learn. Organ.
Building Successful Online Communities: Evidence-Based Social Design
Social Q & amp;A
J. Am. Soc. Inf. Sci. Technol.
Online reputation systems: how to design one that does what you need
MIT Sloan Management Review.
Quantifying the impact of badges on user engagement in online Q&A communities
Understanding members’ active participation in online Question-and-Answer communities: a theory and empirical analysis
J. Manag. Inf. Syst.
Stars, votes, and badges: how online badges affect hotel reviewers
J. Travel Tour. Mark.
Sharing knowledge in social Q&A sites: the unintended consequences of extrinsic motivation
J. Manag. Inf. Syst.
The role of status seeking in online communities: giving the gift of experience
J. Comput. Commun.
The impact of community commitment on participation in online communities
Inf. Syst. Res.
Cited by (7)
Administrator-users contribute more to online communities
2022, Information and ManagementCitation Excerpt :Collectively, our study synthesized the literature to create a typology of three positive community behaviors, namely public space contribution, private space contribution, and OL. The online community literature assumes that users are typically community participants, while some can be “top members”, basing on their performance [43]. However, past research has not considered how users who serve in administrator roles may behave differently from other users, showing a research gap.
Recognise me or pay me? How rewards for online restaurant reviews impact continuity: A cross-country investigation
2022, International Journal of Hospitality ManagementCitation Excerpt :The literature suggests consumers are incentivized to share consumption-related information in online communities, such as online reviews, not only for social capital (e.g., status) but also economic capital (e.g., monetary value) (Husemann, 2012); in the current study, both are examined via recognition rewards and monetary rewards. The first type of reward examined in this study, recognition rewards, are defined as social-capital-focused incentives (e.g., badges and ranking) and enhance the reviewer’s social recognition (Bhattacharyya et al., 2019). Given tourism and hospitality review sites, such as TripAdvisor use recognition rewards as mechanisms to increase reviewers’ contributions in posting reviews (Moro et al., 2019), such rewards are particularly relevant to this study.
Impact of mobile technology-enabled HR gamification on employee performance: An empirical investigation
2022, Information and ManagementCitation Excerpt :For example, Goes et al. [31] study the effect of incentive hierarchies on users’ contributions to a user-generated content context. Bhattacharyya et al. [10] provide guidelines about how to implement a recognition system in online communities. Holzer et al. [38] study the effect of feedback on a knowledge management system, and Sheffler et al. [81] examined how different badge designs affect bicycle ridership.
Samadrita Bhattacharyya is a doctoral student of Management Information Systems at the Indian Institute of Management Calcutta. She holds B.Tech in Electronics and Communication Engineering from West Bengal University of Technology and M.Tech in VLSI Design from Indian Institute of Engineering Science and Technology, Shibpur (formerly Bengal Engineering and Science University, Shibpur). Her research interests include social commerce, online reviews, online communities, data analytics, and optimization and algorithms. Her research articles have appeared in Decision Support Systems and proceedings of Workshop of e-Business, Australasian Conference on Information Systems, and IEEE.
Shankhadeep Banerjee is a doctoral student of Management Information Systems at the Indian Institute of Management Calcutta (IIMC). He holds an MBA from IIMC, an IMP certificate from NEOMA Business School, France, and a B.Tech in Computer Science and Engineering from National Institute of Technology Durgapur. His research interests are primarily related to human behavior around contemporary technologies like crowdfunding, online reviews, virtual communities, e-commerce adoption, etc. He has prior IS research experience at Indian School of Business Hyderabad, and also has extensive IS practitioner experience working at top technology firms like Microsoft, Amazon, and eBay. His research articles have appeared in Decision Support Systems journal, and in the proceedings of several esteemed conferences including International Conference on Information Systems and Australasian Conference on Information Systems.
Indranil Bose is Professor of Management Information Systems at the Indian Institute of Management, Calcutta. He acts as Coordinator of IIMC Case Research Center. He holds a B. Tech. from the Indian Institute of Technology, MS from the University of Iowa, MS and Ph.D. from Purdue University. His research interests are in business analytics, telecommunications, information security, and supply chain management. His publications have appeared in MIS Quarterly, Communications of the ACM, Communications of AIS, Computers and Operations Research, Decision Support Systems, Ergonomics, European Journal of Operational Research, Information & Management, International Journal of Production Economics, Journal of Organizational Computing and Electronic Commerce, Journal of the American Society for Information Science and Technology, Operations Research Letters etc. He serves as Senior Editor of Decision Support Systems and as Associate Editor of Information & Management, Communications of AIS, Information Technology & Management and member of the editorial board for Journal of AIS.