Wikipedia:Wikipedia Signpost/Single/2024-09-26

As reported here in July, India's Asian News International (ANI) has brought the Wikimedia Foundation to court. The allegation is of publishing defamatory content about ANI, at the English Wikipedia article, which stated at that time that they had "been accused of having served as a propaganda tool for the incumbent central government, distributing materials from a vast network of fake news websites, and misreporting events on multiple occasions". The Foundation is now being compelled by the Indian court to reveal personal information of some editors who have edited the article, according to Livemint (report) and The Hindu (report). The next hearing will be on 25 October 2024. Wikipedia's internal consensus of ANI's suitability as a citable source for articles (as in this 2021 discussion and WP:RSPANI) generally holds it to be somewhere between marginally reliable and generally unreliable for general reporting, prudent to give in-text attribution for potentially contentious claims, and generally unreliable in its coverage of domestic and international politics (and other topics that the government of India has a stake in). – rs

In European courts, on the other hand, things have been going better for the Wikimedia Foundation lately. In the UK, British-born Swiss lawyer Matthew Parish sued the WMF for libel, because the article about him (correctly) noted his legal issues; however, the case has been dismissed by High Court judge Karen Steyn.

And as highlighted by Techdirt ("The Wikimedia Foundation Successfully Sees Off Another SLAPP Suit, But More Protection Is Needed Globally"), WMF recently reported another legal victory in Germany ("Wikimedia Foundation defeats gambling magnate’s lawsuit in Germany"). The Foundation characterized it as having "had all the hallmarks of an illegitimate 'SLAPP' lawsuit: a strategic lawsuit against public participation. SLAPPs are lawsuits designed to force organizations and individuals to remain silent on legitimate matters of public interest. [...] The [German] Wikipedia article in question names Mladen Pavlovic as one of three co-founders of Tipico, a major European gambling company headquartered in Malta." According to the WMF, this was well-sourced public information. Yet "Pavlovic engaged a reputable German law firm to threaten the Foundation with legal action unless we agreed to censor the Wikipedia article. After consulting members of the German Wikipedia community, we refused the lawyers’ demand." Pavlovic then filed a lawsuit which "was especially intensive for our team because of the unusual number of legal briefs to which we were asked to reply. [...] These usually repeated earlier arguments, and introduced—in our opinion—increasingly thin or irrelevant new ones." As summarized by Techdirt, "that approach seems a conscious attempt to deplete Wikimedia’s limited financial resources [for legal defense], increasingly under strain" from what the WMF blog post describes as a changing legal environment:

The Foundation’s legal team now also has to deal with a wave of new and very demanding “online safety” laws across the world: for example, the EU Digital Services Act (DSA) and the UK Online Safety Act. These conditions force us to be as efficient, creative, and effective as possible, including in lawsuits like this one.

These laws may not have directly affected Pavlovic's lawsuit yet, as it predates the DSA. However, according to WMF, it fits a resource-draining pattern: "The Foundation faces several SLAPP-like cases each year" (citing examples including the still unresolved Caesar DePaço lawsuit in Portugal, see previous Signpost coverage). The Foundation's post ends with a call for anti-SLAPP reform (about which, according to Techdirt, "Some progress has already been made" in the EU and UK). However, it reiterates that insufficient anti-SLAPP protection is only part of the legal challenges affecting Wikimedia projects, briefly noting other concerning developments:

Privacy-infringing laws like France’s data retention law, and emerging online identity requirements, together with laws that give government authorities insufficiently regulated powers to order content takedowns, are also a significant issue.

– H

Wikipedia beat reporter Stephen Harrison, who is best known for his articles in Slate, has recently been busy promoting his debut novel, The Editors, focused on a fictionalized version of the platform (named "Infopendium") that is suddenly caught up in global cyberwarfare during the COVID-19 pandemic — see previous coverage from the Signpost here and here.

Now, though, he has written an article for The Guardian detailing his view on the future of Wikipedia, which is subtitled "The world's most important knowledge platform needs young editors to rescue it from chatbots – and its own tired practices". Harrison says Wikipedia is currently facing an "existential crisis" due to the emergence of AI applications and large language models, which could potentially undermine the platform's visibility. According to Harrison himself, Gen Z editors are the best-equipped to help Wikipedia survive and, possibly, even thrive in this new context: he pointed out a 2022 survey reporting that about 20% of Wikipedia editors were between the ages of 18 and 24, while also noting the role of young contributors in recent debates on the incorporation of chatbot-generated content on the encyclopedia. The article notably includes a short interview with a very prominent Gen Z editor: the latest Wikimedian of the Year, Hannah Clover.

As for those "tired old practices", Harrison has his say about the sometimes inflexible norms and normalizing institutions of Wikipedia, not to mention mobile-unfriendly editing interface, which he calls "issues that dissuade the younger generation from joining the cause". For instance, he says that the tasks taken on by new editors from a decade ago – ones letting them dip their toes in the editing experience in a low-risk, low-consequence environment – are now more highly automated, leaving a lack of "clear entry points". This in turn may lead today's new editors to unknowingly get into contentious topics where they experience off-putting "harsh feedback" from the more established editors. Harrison left unsaid that there are more contentious topics and areas under sanctions than ever before (see prior Signpost coverage that noted "policies of closure and the formalization of boundaries, rules and routines").

Whether the new generation can adapt to, or reform the tired Wiki, and eventually make it their own as they become the normies, or whether they abandon it for something new, only time can tell. – O, B

Joshua Yaffa in The New Yorker explains (paywalled) the difficulties Marlene Engelhorn had in giving away 25 million euros through the Guter Rat für Rückverteilung (Good Council for Redistribution). Engelhorn had inherited her money from a fortune that started with the founding of BASF and later grew with the Boehringer Mannheim pharmaceutical company. She felt that she should give away most of it to reduce wealth inequality in Austria and as a learning experience to guide others who have the same goal. Engelhorn was keeping about 10% of her money and about €3 million was spent on implementing a process where a citizen council – a group of 50 ordinary Austrians selected by lottery – decided where the money should go. This included the use of moderators who "wield huge power" according to an academic who studies this area. They have "an emphasis on getting things done ... it can all mean that, in the moment, you take away the possibility for improvisation or dissent.”

Nearly eighty organizations were selected by the council, with an average of €312,500 for each organization. "Wikipedia" (as they called the Wikimedia Foundation) turned out to be the most controversial choice:

The idea came from a Vienna resident in his mid-thirties [...]. He saw Wikipedia as addressing many of the council’s core values: democracy, accessibility, transparency. The idea was immediately opposed by Kyrillos, a high-school student and the council’s youngest member. “We have a lot of other, more important issues to address here,” he said. Anyway, he went on, his teachers wouldn’t allow him to use Wikipedia as a source in his papers—why give it money?
Factions emerged. Some saw Wikipedia, a nonprofit based in the U.S., as an inefficient use of the council’s resources. Others viewed the effort to nix it as a violation of the council’s ground rules. [...]
The Wikipedia debate was ultimately settled with a compromise. The members of the education group agreed to give the organization fifty thousand euros, a small portion of their total.

Thanks Marlene!

The name "Engelhorn" may ring a bell for longtime Signpost readers. In 2015, the Reiss Engelhorn Museum in Mannheim, Germany, filed lawsuits against the Wikimedia Foundation, Wikimedia Deutschland and a Wikimedia Commons user over the use of photographs of public domain artworks on Wikimedia projects. (Cf. Signpost coverage: "Wikimedia Foundation, Wikimedia Deutschland urge Reiss Engelhorn Museum to reconsider suit over public domain works of art", "Wikimedia lawsuits in France and Germany". While the museum prevailed in court against the Foundation, the EU Copyright Directive subsequently made such assertions of copyright over faithful reproductions of public domain works impossible.) Indeed the museum is so named after one of its sponsors, German industry titan Curt Engelhorn (1926–2016), a relative of Marlene Engelhorn. As detailed in the German Wikipedia article about him, back in 1997, in what was Europe's largest company takeover to date, he had controversially managed to sell off the family's company holdings for 19 billion DM without paying any taxes to the German state, and Marlene Engelhorn has publicly criticized his (lack of) philanthropy.

– Sb, H

• "Can You Trust Dr. Wikipedia?": The Office for Science and Society of McGill University has recently discussed the accuracy of Wikipedia's medical content and how difficult it is to address some questions. What's your purpose in studying "accuracy"? Which language version are you interested in? When was your sample of articles taken? Despite a horrifying lede invoking toasters (see previous Signpost coverage) and John Seigenthaler, the authors conclude that it is "useful and fast" for many purposes, including for medical students approaching their licensing exams. They also praise WikiProject Medicine and the use of classes that teach health science students to edit Wikipedia.
• New problems emerge in Iran: Pejman Amiri reports in NewsBlaze about the Iranian government's attempts to manipulate and censor the Farsi Wikipedia, while openly accusing the head of the Iranian Wikimedians User Group, Mohammad Heydarzadeh (known on-Wiki as Darafsh), of "trying to gain higher and sensitive access to the Wikimedia Foundation", among other things. While much of this report is very difficult for The Signpost to verify, you can see previous Signpost coverage of a global ban possibly linked to actions of the regime.
• Do it, or else: According to a recent report from OC Media, Farid Pardashunas, an Azerbaijani blogger who had received a presidential award in 2021, revealed the identities of contributors to the Azerbaijani Wikipedia – including admin Solavirum – who had deleted about 3,000 articles on soldiers killed in the Second Nagorno-Karabakh War; in what essentially represents blackmail, the blogger urged the contributors to restore the pages, while threatening to send their personal data to the national State Security Service.

• Bias in Judaism and Zionism related articles: Many newspapers have looked at potential bias and reliability issues in articles relating to Judaism and/or Zionism:
  • "Wikipedia defines Zionism as 'colonialism', sparking outrage" (Israel Hayom)
  • "War over Wikipedia's Definition of Zionism Pits Provoked Users Against Biased Editors" (The Jewish Press)
  • "Wikipedia blasted for 'wildly inaccurate' change to entry on Zionism: 'Downright antisemitic'" (Washington Examiner)
  • "From Bias to Balance: Jewish Editors on Wikipedia" (Times of Israel blogs)
  • "Wikipedia has an antisemitism problem" – opinion, (The Jerusalem Post)
  • "Wikipedia's anti-Israel bias is a feature, not a bug" – (The Jewish Journal of Greater Los Angeles)
  The latter source mentions there was an attempt at the reliable sources noticeboard to downgrade The Jewish Chronicle back in May. However, it should be noted that the aforementioned newspaper is just now in the middle of a notable scandal involving fabricated news stories and opaque ownership, with Israeli sources first sounding the alarm. See also the ongoing discussion over at the Noticeboard.
• Paid editing saga involving Portland commissioner Rene Gonzalez continues: Several regional media sources, including The Oregonian and KOIN, reported that Portland city commissioner and candidate for mayor Rene Gonzalez has been cleared of wrongdoing by the City Auditor's Office on September 16, as evidence about Gonzalez's use of local funds for paid editing on Wikipedia was deemed as insufficient to prove a violation of the city's campaign finance law. See previous Signpost coverage on the matter here and here.
• Do Wikipedia – responsibly: Whilst being best known as an actor and the firstborn son of Brad Hall and Julia Louis-Dreyfus, Henry Hall is also a musician. Back in August, he released his second studio album, Stop Doing Funny Stuff, which contains the single "Wikipedia-ing Poison Snakes", a solemn-sounding testimony from a terminally online man who keeps going down rabbit holes on Wikipedia, ranging from predictive text to Booksmart, in order to distract himself from "his own head, his own bed, every word he's ever [freaking] said". If you feel like this song is too relatable, just know we're in this together (and many of us need some help).

In the wake of a high-profile sexual assault in Kolkata last month, India's courts have demanded that Wikipedia remove of the name of a victim from an article on the crime. While some national and local media outlets reported the name of the victim at the time, as did various international media sources, the laws of India, prohibit media from publicizing the names of victims of especially heinous crimes.

The incident was widely reported on 10 August, the day after it occurred; English Wikipedia editors created the article a day after that. By 13 August, editors began debating whether to mention the victim's name. On 16 August, that debate became part of a discussion on how to title the article, which was ongoing until 9 September, when the disagreement about including the name split into its own discussion. All of this was part of the consensus-building process usual and familiar to editors, where editorial decisions are reached by group discussion about how to implement policies and guidelines.

On 16 September, the Supreme Court of India ordered Wikipedia to remove the name. The Free Press Journal (report), The Hindu (report), and The Times of India (report) are among the many sources to report on the court's order. In this case, and as often happens when institutions make requests of Wikipedia, the court made its request with some presumption that Wikipedia has an editorial leader who can issue binding orders. As is known among Wikipedia editors, there is no such person: even the Wikimedia Foundation does not control the content of articles under Wikipedia's policies.

In response to the court decision, the legal department of the Wikimedia Foundation posted a notice on the talk page of the article, encouraging Wikipedia editors to deliberate carefully on the issue and "explain clearly why you feel the balance of interests lies one way or the other, in order to reach consensus accordingly". Wikipedia editors did that, and reached a decision to exclude the victim's name. User:Tamzin, a volunteer editor, closed the discussion and authored the consensus statement.

The final decision was to exclude the name from the article. While closing statements typically are descriptions written by Wikipedia editors and for Wikipedia editors, Tamzin included a summary explanation of the overall process in anticipation that the court, media, and public observers may wish to examine both the consensus and discussion. It will not come as a surprise to Wikipedia editors that Wikipedians value their editorial independence. The closing statement emphasizes that Wikipedia editors did not arrive at that decision at the behest of the court, but rather because community deliberation found reasons for doing so and because a supermajority of editors supported the decision.

The article is 2024 Kolkata rape and murder incident, and it begins, "On 9 August 2024, a 31-year-old female postgraduate trainee doctor at R. G. Kar Medical College and Hospital in Kolkata, West Bengal, India, was raped and murdered in a college building."

One set of arguments about the name relates to victims' rights and women's rights. The argument in favor of naming the victim is that her story becomes known and enables activism to reduce violence against women. The argument opposed is that in some cases, and this case in particular, naming the victim greatly endangers and disturbs their family, social network, colleagues, and supporters.

Another set of arguments relates to censorship of Wikipedia and Wikipedia's own WP:NOTCENSORED policy. The argument in favor of publishing the name is that maximal freedom in publishing is the preferred position. The argument opposed is that naming the victim is not a censorship issue, as Wikipedia will definitely have an article on the crime, and that article does not benefit significantly by including the name of the victim.

Another set of arguments is about following the lead of what other media outlets do. Arguments in favor of publishing the name point to seeming WP:Reliable sources and reputable journalists who are publishing the name. Arguments opposed to publishing the name make various claims, including that sources publishing the name are mistaken, or that they have since removed the name, or that the higher quality sources do not publish the name while lower quality sources do. Wikipedia editor User:Fowler&fowler checked various sources and reported which ones do not publish the name.

A final set of arguments is on the practicality of collaboration between Wikipedia and the government of India. The argument in favor of publishing the name assumes that other arguments establish that Wikipedia editors should publish the name, and in that context, it is best for Wikipedia as an international media source outside the jurisdiction of Indian government control to disregard the government request. Arguments opposed to publishing the name include respect for the expertise of those courts, respect for national decision making to know what is best for local culture, anticipation of a good future of peaceful collaboration with the government of India by granting this request, and concern for the burden on Wikipedia editors in India if they bear the responsibility of an online global decision including non-Indian Wikipedia editors.

– BR

asilvering's request for adminship was closed as successful on September 6, 2024.

First, the bullet points:

• It turns out a week is a really, really long time;
• Even if your RfA isn't contentious or stressful, it will probably be exhausting;
• Good nominators might not be essential, but I can't imagine having done RfA without mine; a steady hand on the tiller really, really helps (thanks especially to Vanamonde93);
• Designated monitors are a blessing (thanks theleekycauldron), especially when you're doing the discussion-first format and you would like some acknowledgement that you didn't somehow transclude your RfA straight into the void;
• If I may offer advice to future RfA hopefuls: don't post answers to questions right away. Write the answers in a text file, go take a nap, and then post them.
• If I may offer advice to future RfA participants: if you could not, without being overcome by embarrassment, apply your question/vote to both a) a teenager of your acquaintance and b) your grandmother, you should probably rephrase it.

As for how I got here: Vanamonde93 asked me if I was interested in running some time ago, but I deferred for a number of reasons that became more obviously hollow as time went on. For one, "I'm too busy" doesn't work as a very good excuse if you just go find something else to do, like starting a new kind of GAN backlog drive. My second nominator was an easy choice, but that doesn't mean I wasn't nervous about asking. (Don't laugh, czar!) Reading the nomination statements made my heart grow three sizes and also made me want to disappear into the wallpaper. (I'm not good at praise.)

For a little while, I was of a mind to wait out the end of the discussion-period trial, but once I'd got my nominators lined up, it started to feel like I just needed to get it over with. Then, right after I'd said, "alright, now's good, let's do this," Femke showed up in my inbox asking if I'd ever thought of running. It felt so good to be able to tell someone, "you're just in time!" instead of brushing them off with some kind of excuse.

I tried my level best to ignore all the discussion and voting while it was happening, but it turns out that's really hard – not just because it takes willpower, but also because MediaWiki kept showing me the discussion section whenever I tried to preview my answers to the questions. And it's very hard to avoid knowing the precise count once voting starts, since that's right at the top of the page. So I gave up, and read everything. If you're thinking of RfA and you're the anxious sort who will be constantly fighting the temptation to check in, line up something to do with friends/family so you don't glue yourself to the refresh button. There's no way around this bit. It sucks, and a week is a long time.

Luckily for me, people had really nice things to say. I really appreciate everyone who participated and everyone who came by to congratulate me afterwards. I'm doing my best to avoid the mushroom effect and to remain indifferent to both praise and blame. Thank you all for ignoring and then quietly removing the joke oppose; I was a bit worried they'd be murdered. As for the discussion period, I do think it helped things remain civil and on-track, but I'm ambivalent on the experiment overall.

Being an admin has been fun so far. Everyone's been really helpful, and if I've been driving anyone crazy with all my questions, they've been kind enough not to tell me. Special thanks to Aoidh, who has already stopped me from doing something stupid, kindly and firmly, like disarming a wayward toddler running with scissors.

Maybe my opinion isn't worth much, since I had such a smooth RfA compared to so many others, but if you think you'd probably succeed at RfA and you're holding back since it sounds like a bad way to spend a week: I think you should go for it.

It's possible that someone will drag up something stupid you said four years ago and try to rub your nose in it, or that some personal flaw of yours will be magnified beyond all reason. It's possible some people will try to use your RfA as a soapbox to complain about policies or norms they dislike. It's very likely that you will spend the whole week waiting for the other shoe to drop, whether any of those things happen or not. Even an uncontentious RfA can be an exhausting and unpleasant experience – mine was all of those things.

But it's also very likely that, over the course of a week, somewhere between one and three hundred people will show up to say something nice about you. Many of those people will be folks whose opinion you really, truly value. Some of them will be people you can't remember ever hearing of before, but who nonetheless have something deeply gratifying to say. You should run.

Plus, now I can tell my colleagues that I "have tenure, on Wikipedia." I'm sure they'll all be very impressed.

A monthly overview of recent academic research about Wikipedia and other Wikimedia projects, also published as the Wikimedia Research Newsletter.

A preprint titled "Language Agents Achieve Superhuman Synthesis of Scientific Knowledge"^[1] introduces

"PaperQA2, a frontier language model agent optimized for improved factuality, [which] matches or exceeds subject matter expert performance on three realistic [research] literature research tasks. PaperQA2 writes cited, Wikipedia-style summaries of scientific topics that are significantly more accurate than existing, human-written Wikipedia articles."

It was published by "FutureHouse", a San-Francisco-based nonprofit working on "Automating scientific discovery" (with a focus on biology). FutureHouse was launched last year with funding from former Google CEO Eric Schmidt (at which time it was anticipated it would spend about $20 million by the end of 2024). Generating Wikipedia-like articles about science topics is only one of the applications of "PaperQA2, FutureHouse's scientific RAG [retrieval-augmented generation] system", which is designed to aid researchers. (For example, FutureHouse also recently launched a website called "Has Anyone", described as a "minimalist AI tool to search if anyone has ever researched a given topic.")

In more detail, the researchers "engineered a system called WikiCrow, which generates cited Wikipedia-style articles about human protein-coding genes by combining several PaperQA2 calls on topics such as the structure, function, interactions, and clinical significance of the gene." Each call contributes a section of the resulting article (somewhat similar to another recent system, see our review: "STORM: AI agents role-play as 'Wikipedia editors' and 'experts' to create Wikipedia-like articles"). The prompts include the instruction to "Write in the style of a Wikipedia article, with concise sentences and coherent paragraphs".

With an average cost of $5.50, the generated articles tended to be longer than their Wikipedia counterparts and had higher quality, at least according to the paper's evaluation method:

We used WikiCrow to generate 240 articles on genes that already have non-stub Wikipedia articles to have matched comparisons. WikiCrow articles averaged 1219.0 ± 275.0 words (mean ± SD, N = 240), longer than the corresponding Wikipedia articles (889.6 ± 715.3 words). The average article was generated in 491.5 ± 324.0 seconds, and had an average cost of $4.48 ± $1.02 per article (including costs for search and LLM APIs). We compared WikiCrow and Wikipedia on 375 statements sampled from the 240 paired articles. [...] The initial article sampling excluded any Wikipedia articles that were "stubs" or incomplete articles. Statements were then shuffled and given, blinded, to human experts, who graded statements according to whether they were (1) cited and supported; (2) missing a citation; or (3) cited and unsupported. We found that WikiCrow had significantly fewer "cited and unsupported" statements than the paired Wikipedia articles (13.5% vs. 24.9%) (p = 0.0075, χ² (1), N = 375 for all tests in this section). WikiCrow failed to cite sources at a 3.9x lower rate than human written articles, as only 3.5% of WikiCrow statements were uncited, vs. 13.6% for Wikipedia (p < 0.001). In addition, defining precision for WikiCrow as the ratio of cited and supported statements over all cited statements, we found that WikiCrow displayed significantly higher precision than human-written articles (86.1% vs. 71.2%, p = 0.0013).

For the judgment whether a particular statement was "supported" by the cited references, the concrete question asked to the graders (described as "expert researchers" in the paper) was:

"Is the information correct, as cited? In other words, is the information stated in the sentence correct according to the literature that it cites?"

In addition, among other more detailed instructions, the graders were advised to mark a statement correct as cited even if it was not directly supported by the source, as long as the statement consisted of "broad context" judged to be "undergraduate biology student common knowledge" (akin to an extreme interpretation of WP:BLUE).

The fact that these rating criteria appear to be more liberal than Wikipedia's own, combined with the well-known general reputation of LLMs for generating hallucinations, makes the "WikiCrow displayed significantly higher precision" result rather remarkable. The authors double-checked it by examining the data more closely:

The "cited and unsupported" evaluation category includes both inaccurate statements (e.g. true hallucinations or reasoning errors) and statements that are accurate with inappropriate citations. To investigate the nature of the errors in Wikipedia and WikiCrow further, we manually inspected all reported errors and attempted to classify the issues as follows: reasoning issues, i.e. the written information contradicts, over-extrapolates, or is unsupported by any included citations; attribution issues, i.e. the information is likely supported by another included source, but either the statement does not include the correct citation locally or the source is too broad (e.g. a database portal link); or trivial statements, which are true passages, but overly pedantic or unnecessary [...]. Surprisingly, we found that compared to Wikipedia, WikiCrow had significantly fewer reasoning errors (12 vs. 26, p = 0.0144, χ2 (1), N = 375) but a similar number of attribution errors (10 vs. 16, p = 0.21), suggesting that the improved factuality of WikiCrow over Wikipedia was largely due to improvements in reasoning.

The authors caution that this result about Wikipedians "hallucinating" more frequently than AI is specific to their "WikiCrow" system (and the task of writing articles about genes), and must not be generalized to LLMs in general:

Although language models are clearly prone to reasoning errors (or hallucinations), in our task at least they appear to be less prone to such errors than Wikipedia authors or editors. This statement is specific to the agentic RAG setting presented here: language models like GPT-4 on their own, if asked to generate Wikipedia articles, would still be expected to hallucinate at high rates.

A previous, less capable version of the WikiCrow system had already been described in a December 2023 blog post, which discussed the motivation for focusing on the task of writing Wikipedia-like articles about genes in more detail. Rather than seeing it as an arbitrary benchmark demo for their LLM agent system (back then in its earlier version, PaperQA), the authors described it as being motivated by longstanding shortcomings of Wikipedia's gene coverage that are seriously hampering the work of researchers who have come to rely on Wikipedia:

If you've spent time in molecular biology, you have probably encountered the "alphabet soup" problem of genomics. Experiments in genomics uncover lists of genes implicated in a biological process, like MGAT5B and ADGRA3. Researchers turn to tools like Google, Uniprot or Wikipedia to learn more, as the knowledge of 20,000 human genes is too broad for any single human to understand. However, according to our count, only 3,639 of the 19,255 human protein-coding genes recognized by the HGNC have high-quality (non-stub) summaries on [English] Wikipedia; the other 15,616 lack pages or are incomplete stubs. Often, plenty is known about the gene, but no one has taken the time to write up a summary. This is part of a much broader problem today: scientific knowledge is hard to access, and often locked up in impenetrable technical reports. To find out about genes like MGAT5B and ADGRA3, you'd end up sinking hours into reading the primary literature.

[The 2023 version of] WikiCrow is a first step towards automated synthesis of human scientific knowledge. As a first demo, we used WikiCrow to generate drafts of Wikipedia-style articles for all 15,616 of the Human protein-coding genes that currently lack articles or have stubs, using information from full-text articles that we have access to through our academic affiliations. We estimate that this task would have taken an expert human ~60,000 hours total (6.8 working years). By contrast, WikiCrow wrote all 15,616 articles in a few days (about 8 minutes per article, with 50 instances running in parallel), drawing on 14,819,358 pages from 871,000 scientific papers that it identified as relevant in the literature.

These challenges of covering the large number of relevant genes are not news to Wikipedians working in this area. Back in 2011, several papers in a special issue of Nucleic Acids Research on databases had explored Wikipedia as a database for structured biological data, e.g. asking "how to get scientists en masse to edit articles" in this area, and presenting English Wikipedia's "Gene Wiki" taskforce (which is currently inactive). In a 2020 article in eLife, a group of 30 researchers and Wikidata contributors similarly "describe[d] the breadth and depth of the biomedical knowledge contained within Wikidata," including its coverage of genes in general ("Wikidata contains items for over 1.1 million genes and 940 thousand proteins from 201 unique taxa") and human genetic variants ("Wikidata currently contains 1502 items corresponding to human genetic variants, focused on those with a clear clinical or therapeutic relevance").^[2] But it seems that at least from the point of view of the FutureHouse researchers, Wikidata's gene coverage is not a substitute for Wikipedia's, perhaps because it does not offer the same kind of factual coverage (see also the review of a related dissertation below).

The current paper is not peer-reviewed, but conveys credibility by disclosing ample detail about the methodology for building and evaluating the PaperQA2 and WikiCrow systems (also in an accompanying technical blog post), and by releasing the underlying source code and data. The PaperQA2 system is available as an open-source software package. (This includes a "Setting to emulate the Wikipedia article writing used in our WikiCrow publication". However, the paper cautions that the released version does not include some additional tools that were used, and in particular does not provide "access to non-local full-text literature searches", which are "often bound by licensing agreements".) The generated articles are available online in rendered form and as Markdown source (see full list below, with links to their Wikipedia counterparts for comparison). The annotated expert ratings have been published as well.

The authors acknowledge "previous work on unconstrained document summarization, where the document must be found and then summarized, and even writing Wikipedia-style articles with RAG" (i.e. the aforementioned STORM project). But they highlight that

"These studies have not compared directly against Wikipedia with human evaluation. Instead, they used either LLMs to judge or [like STORM] compared ROGUE (text overlap) against ground-truth summaries. Here, we measure directly against human-generated Wikipedia with subject [matter] expert grading."

The "crow" moniker (already used in a predecessor project called "ChemCrow",^{[supp 1]} an LLM agent working on chemistry tasks) is inspired by the fact that "Crows can talk – like a parrot – but their intelligence lies in tool use."

From the abstract of a dissertation titled "Exploiting semi-structured information in Wikipedia for knowledge graph construction":^[3]

"[...] we address three main challenges in the field of automated knowledge graph construction using semi-structured data in Wikipedia as a data source. To create an ontology with expressive and fine-grained types, we present an approach that extracts a large-scale general-purpose taxonomy from categories and list pages in Wikipedia. We enhance the taxonomy's classes with axioms explicating their semantics. To increase the coverage of long-tail entities in knowledge graphs, we describe a pipeline of approaches that identify entity mentions in Wikipedia listings, integrate them into an existing knowledge graph, and enrich them with additional facts derived from the extraction context. As a result of applying the above approaches to semi-structured data in Wikipedia, we present the knowledge graph CaLiGraph. The graph describes more than 13 million entities with an ontology containing almost 1.3 million classes. To judge the value of CaLiGraph for practical tasks, we introduce a framework that compares knowledge graphs based on their performance on downstream tasks. We find CaLiGraph to be a valuable addition to the field of publicly available general-purpose knowledge graphs."

Why would one want to use Wikipedia as a source of structured data and build a new knowledge graph when Wikidata already exists? First, the thesis argues that Wikidata — even though it has surpassed other public knowledge graphs in the number of entitities — is still very incomplete, especially when it comes to information about long-tail topics:

"The trend of entities added to publicly available KGs in recent years indicates they are far from complete. The number of entities in Wikidata [195], for example, grew by 26% in the time from October 2020 (85M) to October 2023 (107M) [206]. Wikidata describes the largest number of entities and comprises – in terms of entities – other public KGs to a large extent [66]. Consequently, this challenge of incompleteness applies to all public KGs, particularly when it comes to less popular entities [44]. [...]

On the other hand, an automated process for extracting structured information from Wikipedia may not yet be reliable enough to import the result directly without manual review:

While the performance of Open Information Extraction (OIE) systems (i.e., systems that extract information from general web text) has improved in recent years [159, 97, 112], the quality of extracted information has not yet reached a level where integration into public KGs like Wikidata or DBpedia [104] should be done without further filtering. [...]
[...] first "picking low-hanging fruit" by focusing on premium sources like Wikipedia to build a high-quality KG is crucial as it can serve as a solid foundation for approaches that target more challenging data sources. The extracted information may then be used as an additional anchor to make sense of less structured data.

Chapter 3 ("Knowledge Graphs on the Web") contains detailed comparisons of Wikidata with other public knowledge graphs, with observations including the following:

The main focus of DBpedia is on persons (and their careers), as well as places, works, and species. Wikidata also strongly focuses on works (mainly due to the import of entire bibliographic datasets), while Cyc, BabelNet and NELL show a more diverse distribution. [...]

[...] Wikidata has the largest number of instances and the largest detail level in most classes. However, there are differences from class to class. While Wikidata contains a large number of works, YAGO is a good source of events. NELL often has fewer instances, but a larger level of detail, which can be explained by its focus on more prominent instances.
Wikidata contains about twice as many persons as DBpedia and YAGO [..., which] contain almost no persons which are not contained in Wikidata. In conclusion, combining Wikidata with DBpedia or YAGO for better coverage of the Person class would not be beneficial"

(see also an earlier paper co-authored by the author that was titled "Knowledge Graphs on the Web -- an Overview")

• See the page of the monthly Wikimedia Research Showcase for videos and slides of past presentations.

Other recent publications that could not be covered in time for this issue include the items listed below. Contributions, whether reviewing or summarizing newly published research, are always welcome.

A Wikidata taxonomy (from "city or town" to "entity") before and after refinement

From the abstract:^[4]

"Wikidata is known to have a complex taxonomy, with recurrent issues like the ambiguity between instances and classes, the inaccuracy of some taxonomic paths, the presence of cycles, and the high level of redundancy across classes. Manual efforts to clean up this taxonomy are time-consuming and prone to errors or subjective decisions. We present WiKC, a new version of Wikidata taxonomy cleaned automatically using a combination of Large Language Models (LLMs) and graph mining techniques."

From the "Evaluation" section:

"As expected, WiKC is much simpler and much more concise than Wikidata taxonomy. Compared to WiKC, Wikidata taxonomy has a factor higher than 200 in the number of classes, and a factor higher than 10 in the average number of paths from an instance to the root class entity (Q35120)."
"WiKC consistently outperforms Wikidata across all depth ranges. WiKC shows significant accuracy gains at deeper levels (depth 10 or more), suggesting that WiKC has resolved many inconsistency issues in the lower levels of the Wikidata taxonomy."

From the abstract:^[5]

"Hundreds of thousands of articles on English Wikipedia have zero or limited meaningful structure on Wikidata. Much work has been done in the literature to partially or fully automate the process of completing knowledge graphs, but little of it has been practically applied to Wikidata. This paper presents two interconnected practical approaches to speeding up the Wikidata completion task. The first is Wwwyzzerdd, a browser extension that allows users to quickly import statements from Wikipedia to Wikidata. Wwwyzzerdd has been used to make over 100 thousand edits to Wikidata. The second is Psychiq, a new model for predicting instance and subclass statements based on English Wikipedia articles. [...] One initial use is integrating the Psychiq model into the Wwwyzzerdd browser extension."

From the paper:^[6]

"Translations help people understand content written in another language. However, even correct literal translations do not fulfill that goal when people lack the necessary background to understand them. Professional translators incorporate explicitations to explain the missing context by considering cultural differences between source and target audiences. [...] For example, the name “Dominique de Villepin” may be well known in French community while totally unknown to English speakers in which case the translator may detect this gap of background knowledge between two sides and translate it as “the former French Prime Minister Dominique de Villepin” instead of just “Dominique de Villepin”. [...]
This work introduces techniques for automatically generating explicitations, motivated by WIKIEXPL, a dataset that we collect from Wikipedia and annotate with human translators. [...]

Our generation is grounded in Wikidata and Wikipedia—rather than free-form text generation—to prevent hallucinations and to control length or the type of explanation. For SHORT explicitations, we fetch a word from instance of or country of from Wikidata [...]. For MID, we fetch a description of the entity from Wikidata [...]. For LONG type, we fetch three sentences from the first paragraph of Wikipedia."

From the abstract:^[7]

"Knowledge Graph Construction (KGC) can be seen as an iterative process starting from a high quality nucleus that is refined by knowledge extraction approaches in a virtuous loop. Such a nucleus can be obtained from knowledge existing in an open KG like Wikidata. However, due to the size of such generic KGs, integrating them as a whole may entail irrelevant content and scalability issues. We propose an analogy-based approach that starts from seed entities of interest in a generic KG, and keeps or prunes their neighboring entities. We evaluate our approach on Wikidata through two manually labeled datasets that contain either domain-homogeneous or -heterogeneous seed entities."

From the abstract:^[8]

"By analyzing the edit history of Wikipedia’s ‘hyperpop’ page, we locate ongoing debates, controversies, and contestations that point to shaping forces around online genre formation. These potentially have a huge impact on how hyperpop is understood both inside and outside of the music community. In locating the most active editors of the hyperpop Wikipedia page and scrutinizing their edit histories as well as the discussions on the hyperpop page itself, we uncovered debates about artistic notability, biases toward specific sources, and attempts at associating or dissociating musical genre from non-musical identities (such as race, gender, and nationality)."

From the abstract:^[9]

"Paradoxically, in each language [English/French/Portuguese Wikipedia], the airplane has a different inventor. Through online ethnography, this article explores the multilingual landscape of Wikipedia, looking not only at languages, but also at language varieties, and unpacking the intricate connections between language, country, and nationality in grassroots knowledge production online."

Supplementary references: