User:XOR'easter/Research under a cloud

For the condensed version of this response published at the Signpost, see here.

The March 2023 paper "'Too Soon' to count? How gender and race cloud notability considerations on Wikipedia", by Lemieux, Zhang, and Tripodi[1] claims to have unearthed quantitative evidence for gender and race biases in English Wikipedia's article deletion processes:

Applying a combination of web-scraping, deep learning, natural language processing, and qualitative analysis to pages of academics nominated for deletion on Wikipedia, we demonstrate how Wikipedia’s notability guidelines are unequally applied across race and gender.

Specifically, the authors

"[...] explored how metrics used to assess notability on Wikipedia (WP:Search Engine Test; “Too Soon”) are applied across biographies of academics. To do so, we first web-scraped biographies of academics nominated for deletion from 2017 to 2020 (n = 843). Next, we created a numerical proxy for each subject's online presence score. This value is meant to emulate Wikipedia's “Search Engine Test,” (WP:Search Engine Test) a convenient and common way editors can determine probable notability before nominating a biography for deletion. [...] We also conducted a qualitative analysis of the discussions surrounding deleted biographies labeled “Too Soon,” (WP:Too soon). Doing so allowed our research team to assess if gender and/or racial discrepancies existed in deciding whether a biography was considered notable enough for Wikipedia. We find that both metrics are implemented idiosyncratically."

However, this is making a manifestly and indefensibly incorrect claim about how Wikipedia editors judge topics for notability. Also, the paper attempts to back it up with misleading quotations and numbers that are dubious on multiple levels.

  1. ^ Lemieux, Mackenzie; Zhang, Rebecca; Tripodi, Francesca (March 29, 2023). ""Too Soon" to count? How gender and race cloud notability considerations on Wikipedia". Big Data & Society. 10. doi:10.1177/20539517231165490. S2CID 257861139.