An outside check is worth something when it can fail differently from what it checks.

In 2018 Joy Buolamwini and Timnit Gebru tested three commercial gender classifiers, from IBM, Microsoft and Face++. The benchmarks the field measured face analysis against were 79.6% (IJB-A) and 86.2% (Adience) lighter-skinned, so a system could score well on them while failing the people they barely contained. The two built their own reference: 1,270 members of the parliaments of Rwanda, Senegal, South Africa, Iceland, Finland and Sweden, public figures whose official photographs their governments publish, with skin type labelled on the dermatologists’ Fitzpatrick scale. On it, darker-skinned women were misclassified up to 34.7% of the time. The largest error for lighter-skinned men was 0.8%.Buolamwini and Gebru, FAT* 2018

Inioluwa Deborah Raji and Buolamwini measured the same systems again in August 2018. Within seven months of the original audit all three companies had released new versions. Error on darker-skinned women had fallen from 34.7% to 16.97% at IBM, from 20.8% to 1.52% at Microsoft and from 34.5% to 4.1% at Face++. Two companies the first audit had not named, Amazon and Kairos, stood at 31.37% and 22.50%.Raji and Buolamwini, AIES 2019

In June 2020 three of the companies whose systems had been audited pulled back from face recognition, each giving its own reasons and each to a different degree. On 8 June IBM wrote to Congress that it no longer offers general purpose facial recognition or analysis software. On 10 June Amazon announced a one-year moratorium on police use of Rekognition. On 11 June Microsoft said it would not sell facial recognition to police departments in the United States until a national law governs it.IBM, archived · Amazon · Microsoft, as reported by TechCrunch

A benchmark can only catch what it contains, and the same risk runs back into training data. Emily M. Bender, Timnit Gebru, Angelina McMillan-Major and Shmargaret Shmitchell named it documentation debt: training datasets “both undocumented and too large to document post hoc”.On the Dangers of Stochastic Parrots, FAccT 2021

This programme measures when one model can check another. The overview reports that a checker handed a reference defers to whatever it is told is correct, and that which model does the checking mattered more than its size. Gender Shades is the same lesson in the historical record, learned with cameras and faces rather than language models. The benchmarks were a reference that shared the systems’ blind spot, so passing them certified nothing about the people they barely contained. The audit counted because its reference was built separately and could fail differently.

Why agreement between checks that share a blind spot is worth less than it looks is ordinary arithmetic, and the overview has it as a calculator you can move.

The word has an older use that runs the other way. James C. Scott’s Seeing Like a State (Yale, 1998) studies states that simplify the societies they govern so officials can read them. Shoshana Zuboff’s The Age of Surveillance Capitalism (PublicAffairs, 2019) describes corporations that watch people in order to predict and steer what they do. The instruments above point the other way: they make systems readable to the people those systems act on.

The two directions meet in a real tension. Checking who a system fails needs to know which group each person belongs to, and that is data about people. Gender Shades is a worked answer. Its subjects were public officials whose photographs their governments already publish. It labelled skin type rather than race, because “race and ethnic labels are unstable”, with a board-certified dermatologist giving the final label, and it recorded gender as perceived and said so.Buolamwini and Gebru, FAT* 2018, section 3

The costs a public would need to see before it could govern or tax these systems are material as well. Kate Crawford’s Atlas of AI (Yale, 2021) follows AI back to the minerals, the low-wage labour and the data it is made from. Mary L. Gray and Siddharth Suri’s Ghost Work (Houghton Mifflin Harcourt, 2019) documents the online piece workers behind services sold as automatic. The energy row in the table above is the same accounting for electricity.