Vocal minority and silent majority - How do online rating reflect population perceptions of quality

RESEARCH ARTICLE

VOCAL MINORITY AND SILENT MAJORITY: HOW DO

ONLINE RATINGS REFLECT POPULATION

PERCEPTIONS OF QUALITY1

Guodong (Gordon) Gao

Robert H Smith School of Business, University of Maryland, 4325 Van Munching Hall,

College Park, MD 20742-1815 U.S.A. {ggao@rhsmith.umd.edu}

Brad N. Greenwood

Fox School of Business, Temple University, 1801 Liacouras Walk,Philadelphia, PA 19122-6083 U.S.A. {greenwood@temple.edu}

Ritu Agarwal

Robert H Smith School of Business, University of Maryland, 4325 Van Munching Hall,

College Park, MD 20742-1815 U.S.A. {ragarwal@rhsmith.umd.edu}

Jeffrey S. McCullough

School of Public Health, University of Minnesota, 420 Delaware Street S.E.,

Minneapolis, MN 55455 U.S.A. {mccu0056@umn.edu}

1Consumer-generated ratings typically share an objective of illuminating the quality of a product or service forother buyers. While ratings have become ubiquitous and influential on the Internet, surprisingly little empiricalresearch has investigated how these online assessments reflect the opinion of the population at large, especiallyin the domain of professional services where quality is often opaque to consumers. Building on the word-of-mouth literature, we examine the relationship between online ratings and population perceptions of physicianquality. We leverage a unique dataset which includes direct measures of both the offline population’sperception of physician quality and consumer-generated online reviews. As a result, we are able to examinehow online ratings reflect patients’ opinions about physician quality. In sharp contrast to the widely voicedconcerns by medical practitioners, we find that physicians who are rated lower in quality by the patientpopulation are less likely to be rated online. Although ratings provided online are positively correlated withpatient population opinions, the online ratings tend to be exaggerated at the upper end of the quality spectrum. This study is the first to provide empirical evidence of the relationship between online ratings and theunderlying consumer-perceived quality, and extends prior research on online word-of-mouth to the domain ofprofessional services.

Keywords: Online ratings, physician quality, online word-of-mouth, professional services, informativeness

1

Kai Lim was the accepting senior editor for this paper. Bin Gu served as the associate editor.

MIS Quarterly Vol. 39 No. 3, pp. 565-589/September 2015565

Gao et al./Online Ratings and Population Perceptions of Quality

Introduction

Recent years have witnessed a rapid growth of online ratingsby consumers across many products and services. Extantliterature has shown that these ratings can significantly affectsales (e.g., Chevalier and Mayzlin 2006; Clemons et al. 2006;Liu 2006), underscoring their importance in the online mar-ketplace. However, when consumers rely on online ratingsto make decisions, legitimate concerns emerge that the gener-ation of online ratings may be subject to bias as a result ofvarious environmental and behavioral factors, including theself-selection of early buyers, the influence of existing ratings,and different types of managerial intervention (e.g., Godesand Silva 2012; Gu and Ye 2014; Li and Hitt 2008; Moe andTrusov 2011; Muchnik et al. 2013; Wang et al. 2010). To thedegree that online ratings are typically generated by a fractionof users, the ability of ratings to inform consumers about thequality of products or services is questionable. For example,not all products receive ratings, begging the question of howthe quality of products with ratings compares with those thatare not rated. If rated, how closely do online ratings reflectthe perceived quality of the product? Further, does the dis-criminatory power of online ratings vary across the qualityspectrum? Although answering these questions has impor-tant implications for the usefulness of online ratings forconsumer choice, few studies have investigated these criticalissues.

In this paper we provide insights into the relationship betweenonline ratings and the underlying consumer population’s per-ception of quality. While the majority of prior researchattempts to infer product quality from online ratings (e.g., Liand Hitt 2008; Moe and Trusov 2011), our work is distinctivein that we examine how quality is reflected in online ratingsby using direct measures of quality. We use detailed data on1,425 primary care physicians in three metropolitan areaswhich contains, among other variables, population measuresof physician quality as rated by patients, constructed usingrigorous standards set by the Agency for Healthcare Researchand Quality (AHRQ) of the U.S. Department of Health andHuman Services. We match these data to online ratings fromone of the largest physician rating websites as well as otherdemographic and economic variables. This unique datasetenables us to investigate the relationship between populationquality measures and online ratings.

The research presented here makes several important contri-butions. First, to the best of our knowledge, this study is thefirst to provide empirical evidence of the relationship betweenonline ratings and population perceptions of quality. Second,we extend research on online word-of-mouth into the domainof professional services. Although online reviews for con-sumer goods such as books and movies are well-known,

566MIS Quarterly Vol. 39 No. 3/September 2015

online word-of-mouth (WOM) is increasingly expanding toinclude professional service providers such as auto mech-anics, lawyers, and physicians. To date, however, studies ononline WOM have focused limited attention on the expandingexpert service market (a brief review of existing literature isprovided in Table 1). Online ratings may play an even greaterrole in affecting consumer decisions for professional servicesas they often possess experience and credence qualities andhave been characterized as “natural candidates” for WOMcommunication (Harrison-Walker 2001).

Background and Prior Literature

Word-of-Mouth and Social Communication

WOM has long been acknowledged as an important deter-minant of consumer choice (e.g., Arndt 1967; Bass 1969;Katz and Lazarsfeld 1955). Until approximately a decadeago, WOM was, by its very nature, limited to individuals withclose social connections (Brown and Reingen 1987) and/orclose physical proximity. In recent years, however, the notionof WOM has been extended to include a wide range of socialcommunication enabled by technology, and growing empi-rical evidence suggests that these new forms of communi-cation are highly influential in guiding consumer decisions. As a result, an increasing number of studies have examinedthe phenomenon of online consumer reviews (e.g., Clemonset al. 2006; Dellarocas et al. 2004; Dellarocas and Narayan2006; Duan et al. 2008; Godes and Mayzlin 2004; Li and Hitt2008; Moe and Trusov 2011; Muchnik et al. 2013; Sun 2012;Zhu and Zhang 2010).

Table 1 provides a brief summary of recent, representativeempirical literature on WOM. Researchers have studiedWOM across diverse product categories, ranging from videogames to movies to beer, and the effect of these onlinereviews has been explored on a variety of outcomes, such asproduct sales, market penetration, and box-office perfor-mance. Extant literature on online reviews has also enrichedtheoretical models of the relationship between online WOMand outcomes with multiple moderating and mediatingvariables including product experience, the location of WOM,consumer characteristics, and trust.

Despite the breadth of literature in this area, three critical gapsexist that represent fruitful opportunities for research. First,there is limited work in the context of credence goods such asprofessional services. The dominant contexts studied in priorresearch include online retailing (Chen and Xie 2008; Che-valier and Mayzlin 2006; Dellarocas and Wood 2008; Li andHitt 2008; Richins 1983), movies (Chintagunta et al. 2010;

Gao et al./Online Ratings and Population Perceptions of Quality

Table 1. Overview of Major Empirical Studies in Online Word-of-Mouth

PhenomenonGeneration ofWOM

Paper

Dellarocas and Narayan(2006)

Berger and Iyengar(2012)

Dellarocas et al. (2010)Brown et al. (2007)Mudambi and Schuff(2010)

Godes and Mayzlin(2004)

Dellarocas et al. (2004)Moe and Trusov (2011)

Finding

Curvilinear Relationship between satisfaction and WOMSynchronicity of conversation channel impacts which pro-ducts receive WOM based on how interesting the object isNiche products are more likely to receive online reviewsSocial platforms are viewed as actors in social networksand have an effect on the perception and value of WOMDepth of review is correlated with helpfulness, howeverextremity is less helpful for experience goods

Online conversation is an effective proxy for WOM and hasexplanatory power when measuring ratingsOnline ratings are an effective proxy for WOM

WOM affects sales and is subject to social dynamics inthat ratings will affect future rating behavior

As the number of ratings increase the average scoredecreases. This increases with positivity, which inducespurchase errors and dissatisfaction

Social influence biases the rating dyanmics

Customer reviews are not correlated with sales, howeverrecommendations are highly significant

Online recommender systems are more influential thanexpert reviews when determining customer product choicePosted reviews are highly correlated with box office salesOnline Reviews positively relate to sales

Both the mean and dispersion of WOM are associated withsale growth

WOM is not an unbiased indication of quality and will affectsales

Consumer WOM can be used as a valuable tool for onlinemarketing and can be strategically manipulated to increasesales

Box office revenue is correlated with WOM, which in turn iscorrelated with performance

Reviewer disclosure of identity is a strong indicator of bothfuture sales and future geographic sales

WOM references increase social network tenure

WOM is correlated with film box office performanceConsumer characteristics moderate the relationshipbetween WOM and sales

WOM on external review websites is a more effectiveindicator of sales for high-involvement productsIncreases in Micro WOM valence (as moderated by

volume and customer characteristics) significantly impactsproduct adoption

When average rating is low, higher variance is associatedwith greater demand

ContextFilmExperimentFilm

TV programsBooks, CD, etc.

TV programsFilm

Bath, fragranceand beautyproducts Books

Trust and

value in WOM

WOM andRatingsDynamics

Godes and Silva (2012)

Effect of WOMon Sales/Performance

Muchnik et al (2013)Chen et al. (2004)Senecal and Nantel(2004)Liu (2006)

Chevalier and Mayzlin(2006)

Clemons et al. (2006)Li and Hitt (2008)Chen and Xie (2008)

NewsBooks

Electronics andwineFilmBooksCraft BeersBooksCameras

Duan et al. (2008)Forman et al. (2008)Trusov et al. (2009)Chintagunta et al. (2010)Zhu and Zhang (2010)Gu et al. (2011)Hennig-Thurau et al.(2004)Sun (2012)

FilmBooksSocialNetworksFilm

Video GamingCamerasFilm

Books

MIS Quarterly Vol. 39 No. 3/September 2015567

Gao et al./Online Ratings and Population Perceptions of Quality

Dellarocas et al. 2010; Dellarocas and Narayan 2006; Duan etal. 2008), and financial markets (Hong et al. 2005). Profes-sional services, however, are distinctive insofar as there isconsiderable information asymmetry between sellers andbuyers, and consumers typically lack the specialized knowl-edge required to evaluate the quality of the service (Arrow1963). In principle, online customer opinions in such con-texts should create significant welfare because informationacquisition is not only costly, but often infeasible. Inhealthcare settings in particular, the information deficit thatconsumers confront has been characterized as particularlyacute, underscoring the need for rigorous research to promoteadditional information transparency (Christianson et al. 2010;Harris and Buntin 2008).

A second gap relates to the question of which products/services are more likely to receive online WOM: althoughresearchers have studied when products are likely to receivereviews from consumers (e.g., Anderson 1998; Dellarocas etal. 2010; Dellarocas and Narayan 2006; Dellarocas and Wood2008); the question of which products receive ratings hasreceived limited attention. For example, research by Ander-son (1998) and Dellarocas and Narayan (2006) examines howthe marginal experience consumers have with a product islikely to affect their propensity to rate that product online. Dellarocas et al. (2010) address the question of whether nicheor hit films are likely to receive more or fewer online reviews. Likewise, after observing that “less is known about its[WOM’s] causes or what leads people to talk about certainproducts and brands rather than others,” Berger and Iyengar(2012, p. 2) find that more interesting products and brandsgenerate more online WOM but not face-to-face WOM. Although these studies shed light on the characteristics ofproducts (e.g., popularity and interestingness, respectively)that lead to online WOM, the question of how product orservice quality affects the availability of online ratingsremains unanswered. To the degree that the main purpose ofreading online ratings is to learn about quality and facilitatecomparison, understanding how quality correlates with theavailability of online ratings is vitally important. If, forexample, most rated physicians came from the low-qualitycohort, the utility of the ratings would be considerablydegraded as the available information would be limited to oneend of the quality spectrum.

The final gap we note relates to the implicit assumption in thisbody of literature that higher online ratings reflect higherquality, resulting in a higher likelihood of purchase (e.g.,Chevalier and Mayzlin 2006; Chintagunta et al. 2010; Zhuand Zhang 2010). Surprisingly, in spite of the pervasivenessof this assumption, little empirical evidence exists demon-strating the relationship between online ratings and productquality. Clearly, online ratings are contributed by self-

568MIS Quarterly Vol. 39 No. 3/September 2015

motivated individuals. While it has been suggested thatproducts yielding a customer experience that significantlydeviates from a priori expectations are more likely to be rated(both in the case of positive or negative experiences) (Ander-son 1998), this research does not provide an answer to thequestion of how representative these ratings are of the offlinepopulation’s perception of the product quality. Given thedocumented selection biases in the decision to rate online,together with evidence of punitive or reinforcing effects ofearly reviews on long-term sales (Godes and Silva 2012; Liand Hitt 2008), there is a critical need to examine therelationship between online ratings and intrinsic quality.In summary, our review of the literature on WOM reveals thata majority of studies focus on establishing the value propo-sition of WOM by relating reviews to financial measures. Consumer reviews have been examined from various perspec-tives, including the dispersion of WOM (Godes and Mayzlin2004), the distribution of the ratings (Clemons et al. 2006,Sun 2012), product segment (Dellarocas et al. 2010), valence(Chintagunta et al. 2010), and consumer characteristics (Zhuand Zhang 2010). All of these studies have provided impor-tant insights into the relationship between online consumerreviews and sales; however, even though ratings are largelyused by consumers to infer product quality and make con-sumption choices, there is surprisingly limited understandingof how product/service quality affects the likelihood ofreceiving ratings, as well as how ratings reflect quality. Weaddress this gap and contribute to the existing knowledgeabout the value of online ratings to consumers (Mudambi andSchuff 2010).

Word-of-Mouth for Physicians

Our research setting, online ratings for physicians, is conse-quential for both policy and practice. Within the professionalservices spectrum, healthcare is the largest sector of the U.S.economy, accounting for nearly 18 precent of the U.S. GDPin 2010. More recently, healthcare has experienced a strikinggrowth in online WOM (Gao et al. 2012; Lagu et al. 2010). These ratings may help resolve a crucial information asym-metry as, prior to their emergence, very little informationabout the quality of an individual physician made its way intothe public domain.2 From the consumer’s perspective, theinability to learn about the characteristics and qualities of aphysician prior to engaging in a “service” encounter islimiting, and can result in erroneous physician choice. Given

2

To address the lack of transparency in physician quality, the federalgovernment has initiated programs such as Physician Compare led by theCenters for Medicare & Medicaid Services. Despite this and other initiatives,consumers continue to have limited access to physician quality information.