Skip to content
Legible AI

Has someone already asked an AI this?

AI Searches will keep the answers frontier AI models gave to questions people chose to share, with the model, the date and the human-written sources beside each one. You can read what came back last time, check it against the sources, and see how the answer changed from one model or month to the next.

No data is collected yet. The record format and the sharing rules are written first, before anyone can share anything.

The record formatSharing a search

The sky over your part of the world right now, drawn in your browser from the date and time.

Shared by a person, checked by hand · The app’s share link

The question

How do I bleed a radiator in a rented flat?

  1. Written by an AI: Model A, 3 March, search offTurn the heating off, open the valve at the top corner with a radiator key until water appears, then close it and check the boiler pressure.
  2. Written by an AI: Model B, 9 June, search onThe same steps, and a note that in most rentals the landlord is responsible for the boiler, so ask before topping up the pressure.

Human-written sources

  • A radiator maker’s fitting instructions loads · archived · supports both answers, checked by a person
  • A city’s guide to tenants’ repair rights loads · archived · supports Model B, checked by a person

It went on to help with

A building’s agreement about who fixes the boiler

An example, made up to show the format. No record exists yet.

What the research asks of each record

Legible AI started as research on checking AI from outside, now at bentleymoon.com/research/checking-ai. Each record is built so that it can be checked the way that research says a check works.

  1. A checker model told right answers from wrong ones perfectly when it was handed the correct value, and barely at all when it had nothing to compare against. What to hand a checker

    The human-written sources behind each answer, as the reference. Each answer is graded twice, once with the sources and once without, so the size of that gap can be measured on real questions.

  2. Past some difficulty, a monitor came to reject correct work more often than wrong work. The inversion

    A person checks every grade on a hard question. Machine grades only sort records for a person to read, and how often the machine agreed with people is published for each level of difficulty.

  3. A frontier model flagged a planted instruction every time it was asked to look for one, and still followed it while doing the task. Detection and obedience

    The exact question, any framing the asker added, and the pages the model read when search was on. Many people’s wordings of one question are shown together, so a reader can see when wording or text in a source moved the answer.

  4. The model behind a product name changes over time without notice. GPT-4’s answers on one task changed between March and June 2023 (Chen, Zaharia and Zou).

    The app or interface, the model name as shown, the date and time, and whether search was on. The monthly questions put a date on a change.

  5. Questions that become public end up in training data, and a model that has seen a question scores higher on it.

    A private set of monthly questions, published only after the next major model release, with scores on the public and private sets side by side.

The order of work

  1. done

    Publish the record format and the sharing rules

    Written before any data exists, so the rules are fixed before anyone shares anything. They are the two pages beside this one. The record format

  2. not started

    Ask a fixed set of questions every month

    The same questions put to four or five assistants through their developer interfaces, each answer labelled as asked by AI Searches. Part of the set stays private, so a model trained on the public questions shows up as a gap between the two scores.

  3. not started

    A first round with 10 to 20 people

    When the tool is ready, each person pastes a share link from their AI app, and every record is read with them before anything is published. The round counts how long that takes and how much personal detail the automatic check missed.

  4. not started

    Work with groups already watching AI answers

    The Internet Archive’s Ask Me Twice project and the Tow Center for Digital Journalism already record and grade AI answers. A shared question set means each record gets read by more than one group.

  5. later, if the first round supports it

    A browser extension

    Only if the first round shows records can be checked quickly and safely, and a lawyer has read the AI companies’ terms. It would do nothing until you click it, and show you exactly what it will send.

Why keep the answers

Outside tests keep finding AI answers that point to the wrong source. In March 2025 the Tow Center for Digital Journalism gave eight AI search tools 1,600 excerpts from news articles and asked each to name the article. They answered incorrectly on more than 60 percent of the queries (Columbia Journalism Review, 6 March 2025). Tests like that one are run once, on questions their authors chose. A record of questions people really asked, kept with the sources that settle them, can be graded again by anyone, and again next year.

Compute

Reading an answer that already exists can save asking for it again. For a short question that is about 0.3 watt-hours (Epoch AI, 7 February 2025), and finding and reading the old answer uses energy too, which nobody has measured against it. The saving matters for long answers, like a research report an AI spent many minutes writing. The first round will count how often someone reads a record instead of asking again.

Until October 2026 this address held my research on whether one AI can check another’s work. It now lives at bentleymoon.com/research/checking-ai, and every old link goes to the matching page there.