Wikipedia:Wikipedia Signpost/2011-06-06/Recent research

CHI 2011: Shared leadership, nerd stereotypes and the fruit fly of social software

A session on "Incentives & User Generated Content" at the annual "Conference on Human Factors in Computing Systems" (CHI 2011) last month featured two papers about Wikipedia (abstracts):

• Evidence of "shared leadership" found in 4 million talk page messages: As described in a paper titled "Identifying Shared Leadership in Wikipedia" , four researchers from Carnegie Mellon University used machine learning to train an algorithm to classify the text of talk page messages, based on evaluation of a set of formal criteria into four kinds of behavior indicating different kinds of "leadership", using the following descriptions and examples:
  • Providing Positive Feedback (Transactional Leadership: Energize people through acknowledging work and provides rewards): "I’m so impressed. This is a very fine article!"
  • Providing Negative Feedback (Aversive Leadership: Regulate people through reprimands): "If you continue in this manner you will be blocked from editing without further warning. Please stop, and consider improving rather than damaging the work of others."
  • Directing (Directive Leadership: Direct people through issuing instructions, commands, assigning tasks, setting goals): "Here is a new article on a former airport I thought you might want to check out."
  • Social exchange (Transformational Leadership: Promote emotional engagement through for example talking nice, starting off-topic conversation, etc.). "Drop me a line on my talk page sometime, we’ll get a coffee over at Hot Rize or the new King Kocoa…".

They used this to classify four million user talk page messages from the English Wikipedia, from a January 2008 dump, "sent by 130,000 distinct users (who had edited Wikipedia for an average of 13.6 months) and were received by 1.1 million distinct users (who averaged 10.8 months of editing)". Aiming to find differences between Wikipedians in central and peripheral roles, the researcher compared results for admins and non-admins, and according to membership in a WikiProject (non-members, regular members or core members, the latter defined as founder or top three contributor of a WikiProject). They found that "the more central editors perform more leadership behaviors per person because they are generally more active. However, these differences are not huge. For example, 2.8% of administrators’ work consists of sending directive messages compared to 2.0% for non-administrators". They interpreted this as "strong evidence of shared leadership in Wikipedia" (defined as "a dynamic, interactive influence process among individuals in groups for which the objective is to lead one another to the achievement of group or organizational goals"), with "a large proportion of leadership behaviors performed by editors in peripheral as well as central roles", in contrast to traditional models of leadership. Although editors in all roles showed leadership behaviors, there were differences: "the role of core members in Wikiprojects may be less task-focused and more person-focused, with social or motivational messages to keep members active".

• Negative stereotypes about Wikipedians may deter newbies: For a paper titled "My Kind of People? Perceptions About Wikipedia Contributors and Their Motivations" (slides), a Yahoo! researcher conducted 20 in-person interviews with participants who had all edited Wikipedia before, but infrequently. When asked how they imagined Wikipedia contributors, interviewees used three "primary stereotypes": That of "regular folks", reflecting a perception of Wikipedia as an egalitarian community, secondly "well educated, credentialed group", and thirdly, "By far the most common image that participants invoked to describe Wikipedia’s contributors was that of the solitary techno-geek, ... an unflattering picture [where Wikipedians] are 'geeky' or 'nerdy,' technologically adept, unkempt, unhealthily obsessive, and absorbed with online life." The author stressed the potential damage caused by such negative stereotypes (even if their factual accuracy was questionable), as they might prevent new editors from joining the community. He states that a wiki's "deliberate design decision to hide the identities of individual authors in favor of a kind of collective authorship ... has consequences which are to date insufficiently investigated", possibly allowing readers to fill the void with preconceived stereotypes. "Wikipedia’s ongoing educational efforts could include 'meet the author' informational campaigns which highlight the identities of heavy contributors and emphasize their pro-social motivations. In other words, Wikipedia can combat speculative answers to the question 'Who writes Wikipedia?' by explicitly revealing and promoting that information to its users." The paper has already been noted by the Wikimedia Foundation's "Account Creation Improvement Project".
• Wikipedia on a DVD player: Also at CHI 2011, a note titled "Utilizing DVD players as low-cost offline Internet browsers", describing a method that "enables communities in the developing world to access Wikipedia and other resources at very low cost", received a honorable mention. According to the abstract, the researchers put "the entirety of schools-wikipedia.org – encompassing 5,500 articles and 259,000 screens – to a double-layer DVD. We evaluate our system via a study of 20 low-income users in Bangalore, India. Using our DVD as reference, participants are able to answer factual questions with over 90% success."

A saying reported at the conference: "Wikipedia is the Drosophila of social software."

WebSci'11: Diversity, edit-wars, wikilink distance and reference blindness

The program of next week's ACM WebSci'11 conference contains several presentations and posters about Wikipedia:

The paper for a short presentation titled "Towards a diversity-minded Wikipedia" consists largely of a review of existing literature on the demographics of the Wikipedia community and other topics which to the authors make it "seem likely that high barriers exist for new viewpoints to be accepted in Wikipedia, even if they objectively contain useful information". The authors then proceed to say that "together with the German chapter of the Wikimedia Foundation, the European research project RENDER will work on building a truly diversified Wikipedia", by providing "representations, techniques and tools to discover, understand, and use the following types of information: the multitude of opinions, sentiments and viewpoints, the points of dissent, content that would otherwise disappear from view, the quality of articles, and controversies surrounding specific topics." (An employee at Wikimedia Germany has been working on the RENDER project since about March, see the chapter's April report. Expect more thorough coverage of RENDER in a future Signpost issue.)

Posters:

• Frequency of edit wars: Aiming at a "Characterization and prediction of Wikipedia edit wars", five authors from Budapest University of Technology and Economics suggest a relatively simple formula to measure the "controversiality" of a Wikipedia article, based on identifying the editors who have reverted or were reverted in that article, and counting their total contributions to it. Having tested it on five different language Wikipedias, also against the presence of templates warning about controversies, they consider it efficient at detecting edit wars (as discerned from mere vandalism reverts), and use it to measure the total proportion of controversial articles: "Only one page in a hundred becomes even a candidate for war (less than 30k out of over 3m articles in the English WP). Less than 0.5% of pages shows significant signs of war".
• How far do wikilinks go?: In a "Wikipedia Case Study" about "Measuring Hyperlink Distances" two researchers from the Federal University of Rio de Janeiro examined the distribution of the distance between a Wikipedia article and another that it links to, as measured by various distance functions based on which categories these two articles share. They found evidence "that hyperlinks in Wikipedia are more likely to point to documents that are not related" than to similar ones. In passing, they note that "the document with the most categories is 'List of mathematics categories', with 1391 categories."
• Reference Blindness: In a third poster titled "Reference Blindness: The Influence of References on Trust in Wikipedia", three researchers from the University of Twente asked a survey group of 23 college students to rate the credibility of four articles from the English Wikipedia, in various versions altered such that the same article text appeared with fewer or unrelated references. As "the most remarkable observation" they found that "only 6 of the 23 participants noticed that the references were not related to the topic of the article in the low-quality condition. However, 17 participants indicated that they had paid attention to the references. We coin this phenomenon reference blindness: users consider references important for credibility, but as long as they are present, the quality of the references mostly does not seem to matter. " In addition, two of the authors evaluate another method to assess credibility, namely the WikiTrust software, in last month's issue of First Monday ("Evaluating WikiTrust: A trust support tool for Wikipedia" – finding that "the participants in our experiment rated usefulness of WikiTrust low"), and published another article titled "Factual accuracy and trust in information: The role of expertise" (abstract) in the Journal of the American Society for Information Science and Technology.

Dissertation: Wikipedia's "cyborg individual" ideal

In his dissertation titled "Hackers, Cyborgs, and Wikipedians: The Political Economy and Cultural History of Wikipedia" (submitted at Bowling Green State University last month), Andrew Famiglietti argues that Wikipedia "was shaped by an ideal I call, 'the cyborg individual,' which held that the production of knowledge was best entrusted to a widely distributed network of individual human subjects and individually owned computers. I trace how this ideal emerged from hacker culture in response to anxieties hackers experienced due to their intimate relationships with machines." Yochai Benkler's ideas are referred to, among those of others. One chapter, titled "Wikipedia and Google", rejects a blogger's claim that Wikipedians decide on the notability of article subjects solely based on Google hits. A detailed analysis of the fate of new articles from one entire day (which the author provides online in the form of a blog, tagged by their eventual fate – e.g. those speedily deleted under CSD A7), finds that while Google searches indeed play an important role in the corresponding deletion processes, Wikipedians "are justifiably confident in their ability to skillfully use" it, avoiding the "if it is not on Google it doesn't exist" trap.

Briefly

• Wikipedia research survey: Wikipedia researcher Finn Årup Nielsen has announced an updated version of his survey paper on more than 1000 Wikipedia research publications (see also the last "Recent research").
• The upcoming Open Knowledge Conference 2011 will see a Workshop on Wikipedia & Research, organized by members of the Wikimedia Foundation' Research Committee.
• Historical discussions about "No Personal Attacks" policy analyzed: In a paper titled "Self-Governance Through Group Discussion in Wikipedia: Measuring Deliberation in Online Groups" (appearing in the June issue of "Small Group Research", abstract), researchers from Ohio University, Cornell and Southern Illinois University Edwardsville examined "the small group discussions that undergird policy-making processes in a well-established online community, Wikipedia. Content analysis shows that these discussions demonstrated a relatively high level of problem analysis and providing of information, but results were mixed in the group’s demonstration of respect, consideration, and mutual comprehension". They note that Wikipedians "do not simply write and discuss encyclopedia articles: they also propose, collaboratively create, discuss, agree on, and enforce the policies that guide their interactions. This stands in sharp contrast to most online communities, where governance resides in the hands of a relative few community leaders". Concretely, they analyzed the postings on the talk page of Wikipedia:No personal attacks "from April 2002, when the first version of the policy was proposed, through August 2005" (it was first proposed here by Jimmy Wales), comprising 282 posts across 35 discussion threads, and coded them "on eight of the nine dimensions of deliberation: creating an information base, prioritizing values, identifying solutions, weighing solutions, making decisions, comprehension, consideration, and respect. For example, 40 postings or 14.2% were evaluated as showing "Lack of respect". They also created a social network graph of discussants on one archive page (with User:Snowspinner and User "SamS" as the two biggest nodes), and highlight concrete examples (including user names) of patterns formally interpreted as "conflict management" or "good deliberative discussion".
• Editor retention not necessarily a good thing?: In an article titled "Membership Turnover and Collaboration Success in Online Communities: Explaining Rises and Falls from Grace in Wikipedia", two researchers from Boston College examine "the longitudinal history of 2,065 featured articles on Wikipedia" and find evidence that "contributions from a mixture of new and experienced participants both increases [sic] the likelihood that an article will be promoted to featured article status and decreases the risk it will be demoted after having been promoted. These findings imply that, contrary to many of the assumptions in previous research, participant retention does not have a strictly positive effect on emerging collaborative environments."
• Web of trust: Three researchers from Paris reported on efforts to group Wikipedia editors into a "signed network" (also known as web of trust), based on the following kinds of interactions: "edits over commonly-authored articles, activities such as votes for adminship, the restoring of an article to a previous version, or the assignment of barnstars (a prize, acknowledging valuable contributions)."
• Venetian network: Researcher Paolo Massa (user:Phauly) also examined "Social Networks of Wikipedia" (a paper presented at the ACM Hypertext 2011 conference this week), consisting of users on the Venetian Wikipedia with the edges of the network determined by the number of messages one user has left on the talk page of another.
• A paper titled "Wiki-watchdog: Anomaly Detection in Wikipedia Through a Distributional Lens" describes "an efficient distribution-based methodology that monitors distributions of revision activity for changes. We show that using our methods it is possible to detect the activity of bots, flash events, and outages, as they occur. Our methods are proposed to support the monitoring of the [Wikipedia] contributors" and other things.
• Wikipedia's historiography: Dominant or alternative?: An article titled "The nature of historical representation on Wikipedia: Dominant or alterative historiography?" (appeared in this month's issue of the Journal of the American Society for Information Science and Technology) compared the "Wikipedia accounts of Singaporean and Philippine history", according to the abstract. The author argues that "information professionals [should] take a keener interest in Wikipedia, with an eye to helping include accounts of documented historical perspectives that are ignored by mainstream historiographical traditions."
• Identifying current events: A publication titled "WikiTopics: What is popular on Wikipedia and why" by three US researchers described an automated method to "identify and describe significant current events as according to Wikipedia content, and metadata", by first selecting articles with significantly increasing page views, followed by clustering, and then "generat[ing] textual descriptions for the clustered articles to explain why they are popular and what current event they are relevant to".
• 8.5% of Wikipedia articles tagged as flawed: A paper titled "Towards automatic quality assurance in Wikipedia" (prepared for the World Wide Web Conference 2011 two months ago) analyzed the frequency of cleanup templates (e.g. for NPOV or notability problems) on the English Wikipedia (as of January 2010) and found that 8.5% articles were tagged for at least one flaw – most often with Unreferenced, which was encountered in 135210 articles (4.57%). The researchers from Bauhaus-Universität Weimar then built automatic classifiers that tried to discern featured articles from those carrying one of the most frequent flaws, which worked "with a nearly perfect precision" in the case of the "orphan" and "notability" tags. They announced that "based on the lessons learned, we plan to operationalize our classification approach as a Wikipedia bot that tags articles autonomously".
• A paper describing a method for the "Quality evaluation of Wikipedia articles through edit history and editor groups" promises that it "has better performance in quality evaluation than several existing metrics", according to the abstract.