Hundreds of Contractors Read Your ChatGPT Chats. The Toggle Is On.

TL;DR

404 Media reported on September 14 that OpenAI is hiring hundreds of contractors to read real ChatGPT prompts, sometimes whole conversations, under an internal codename: Project Lily. Reviewers never see a username, but a task can carry a "user memories summary" that sometimes hints at where the user lives. They write down what the user wanted, then grade four ChatGPT replies from 1 to 7 against a rubric that punishes sycophancy, engagement-bait endings and decorative emojis. OpenAI says prompts pass through a version of its Privacy Filter model first, and the public model card for that filter lists what it misses, including novel credential formats. Training is on by default for Free, Plus and Pro, and OpenAI's own help pages say the same switch covers Codex tasks on personal plans.

What 404 Media saw

Reporter Joseph Cox says 404 Media reviewed instruction guides, Slack channels, the rating system and real user prompts. It quoted none of the prompts, citing source protection. Some of them asked ChatGPT to keep the content to itself, which tells you what those users assumed about who was on the other end.

This is not the safety review OpenAI already talks about publicly, where flagged chats get a human look. Lily is quality work: people reading ordinary conversations to make the next model's answers better.

• Scale: "hundreds of contractors," per the report.
• Pay: one North America-based worker told 404 Media they earn more than $50 an hour.
• Supply chain: that worker found the gig through recruiter Crossing Hurdles, which referred them to Mercor, the company that actually pays.
• The work itself: "very rote," with guidelines that change a lot and "can feel self-contradictory."
• The target: unknown. The material names no model, released or unreleased. Only the codename.

Anatomy of a Lily task

A reviewer picks a task from a dashboard and gets a real prompt. There is no username. Above the prompt there is sometimes a "user memories summary," an overview of what that person has used ChatGPT for before, which in some cases includes where they may live and other personal context.

The job has three stages. First, read the prompt. Second, write a short summary of what the user is really asking; the guide's example is "The user is asking for help on revising a work Slack message." Third, review four generated responses, highlight at least three passages as "aligned or misaligned" with explanations, score each response from 1 to 7, and write a rationale.

The instruction guide tells reviewers to escalate tasks "with potential safety concerns" or personal information. That is a sensible rule, and it also confirms personal information shows up often enough to need a rule.

The rubric is a style guide for the next ChatGPT

For anyone building on OpenAI models, the grading criteria spell out what the company's human graders reward. The target, from one guide: a response that understands the user's intent and writes "in a style that is clear, natural and appropriately warm."

• Scale anchors: 1 is "unacceptable, unusable." 7 is "would be hard to meaningfully improve." A response with useful content can still score low if it is too long or cluttered.
• Tone: "generally match the user's tone, but slightly less intensely," staying "natural, restrained, and professional without implying that it is human or experiencing emotions."
• Named failure modes: "sycophancy, forced style mimicry, engagement-bait endings, amplification of frustration, or patronizing assumptions."
• No fake lived experience: no "As a chef, I like to..." or "I know what that's like." Operational first person such as "I'll take a look" is fine.
• High-stakes sourcing: reviewers do not fact-check with outside searches ("other project teams handle content verification"), but they flag errors they notice and penalize missing sources on medical, legal and financial answers.

Then there are the emojis. One guide's example marks a list decorated with checkmark emojis as "misaligned" for "unnecessary use of emojis." Another says context matters: a tree emoji suits Arbor Day planning, while skulls in a conversation about death or planes in an update on a fatal crash do not. Somewhere, a LinkedIn growth coach felt a disturbance.

The practical reading: expect default ChatGPT output to keep drifting toward shorter, plainer, less flattering prose. If your product leans on the chatty house style, pin it in your system prompt now. And if you run your own evals, this rubric is a reasonable template: a written intent summary, forced highlighting of specific passages, and a scale whose top score means "hard to improve," not "fine."

The filter in the middle

OpenAI told 404 Media it runs conversations through "a version of" its Privacy Filter model before contractors see them. The public version is openai/privacy-filter on Hugging Face (code on GitHub), released in April under Apache 2.0: 1.5 billion parameters in total with 50 million active, small enough to run in a browser.

It is a token classifier, not a generator. It reads the text once and tags spans against a fixed set of eight labels. The model card is blunt that it "will only identify personal data spans that match the trained label taxonomy," and that it is "a redaction and data minimization aid, not an anonymization, compliance, or a safety guarantee."

Picture a redaction clerk handed a list of eight things to black out. Names, emails and phone numbers get the marker. "My manager doesn't know I'm interviewing elsewhere, and the biopsy results come Friday" contains nothing on the list, so nothing in it matches a label.

The failure modes section matters more to developers than to anyone else. It lists "missed secrets for novel credential formats, project-specific token patterns, or secrets split across surrounding syntax." That is a precise description of an internal API key pasted into a config snippet. We do not know whether OpenAI's internal version shares the public label set or failure profile, and OpenAI has not said.

Who is in the pool by default

OpenAI told 404 Media that turning off "Improve the model for everyone" keeps chats out of model improvement. The catch is the default, which is on for Free, Plus and Pro, and the switch only applies to new conversations.

By OpenAI's February count, ChatGPT had more than 900 million weekly active users and more than 50 million consumer subscribers, per TechCrunch, plus over 9 million paying business users, per Search Engine Land. Of those groups, only the business users start with training off.

The Codex detail

OpenAI's Data Controls FAQ says that if you use Codex on a personal ChatGPT plan, "the 'Improve the model for everyone' setting also applies to your Codex tasks." It also says Codex has a separate setting for training on full environments, and that changing the ChatGPT toggle or opting out through the privacy portal "does not change that Codex setting." Two switches, one of which your ChatGPT settings never touch.

To be clear about scope: 404 Media's reporting covers ChatGPT prompts. Nothing in it says Lily reviewers saw Codex tasks. The docs only establish that personal-plan Codex tasks sit behind the same training toggle.

The side doors

• Feedback overrides opt-out: OpenAI's model improvement page says that even after you opt out, clicking thumbs up or down means "the entire conversation associated with that feedback may be used to train our models."
• Deleting is not recalling: the consumer data FAQ says cleared chats are deleted within 30 days "unless they have previously been de-identified and disassociated from your account."
• The disclosure: 404 Media says OpenAI did not answer whether it had told users humans may read prompts. After publication, OpenAI pointed to that same FAQ, which lists "to improve model performance (unless you have opted out)" among the reasons service providers may access content, and adds: "Please do not enter sensitive information that you would not want reviewed or used."

The thumbs-down button, in other words, is the one place where complaining about a bad answer can volunteer your whole conversation as training data.

Not just OpenAI

Human review of chatbot conversations is industry practice, and 404 Media says so. Gemini's privacy hub says "a subset of chats are reviewed by human reviewers (including Google's trained service providers)" who flag low-quality, inaccurate or harmful responses and suggest better ones.

Anthropic confirmed to 404 Media that it uses human review to improve its models for users who turned on model improvement in privacy settings, and that it strips account identifiers such as email addresses first. Its privacy center says that setting also covers Claude Code sessions from Free, Pro and Max accounts, and that thumbs feedback stores the entire related conversation for up to five years.

What makes Lily different is detail, not existence: the memory summaries, the rubric text, the staffing chain and the pay. Michal Luria of the Center for Democracy & Technology put the core problem well: chatbot interfaces "automatically create a false sense of intimacy and privacy."

What to change today

• Personal ChatGPT or Codex for work: turn off "Improve the model for everyone" under Settings, Data Controls, then check the separate full-environment setting in Codex Settings.
• One-off sensitive questions: use Temporary Chat. OpenAI's FAQ says those chats are not used for training, are deleted after 30 days, create no memories, and "may be reviewed only to monitor for abuse."
• Secrets: never paste live credentials into a consumer chat. The filter's own model card says novel credential formats can slip past it.
• Feedback buttons: treat thumbs up and down as "send this whole thread to OpenAI," because the docs say that is what they can do.
• Teams: business plans and the API start opted out. If your people use personal Plus or Pro accounts for company work, that is the gap to close.

What we still don't know

We do not know how many prompts go through Lily, how they are sampled, how often memory summaries appear, or which model the grades train.

The "hundreds of contractors" figure and the more-than-$50 hourly rate come from 404 Media alone, and the rate is one worker's. OpenAI has not said whether its internal Privacy Filter matches the open-weights release, and the report does not say whether opted-out users' data ever reaches reviewers through feedback clicks.

Key Takeaways

• 404 Media reports OpenAI contractors under "Project Lily" read real ChatGPT prompts, sometimes with a user memories summary that can hint at location, and grade four replies from 1 to 7.
• The rubric penalizes sycophancy, engagement-bait endings, fake lived experience and gratuitous emojis, a useful preview of where default ChatGPT style is heading.
• Prompts pass through a version of OpenAI's Privacy Filter, whose public model card has eight labels and admits it can miss novel credential formats.
• Training is on by default for Free, Plus and Pro, and the same toggle covers Codex tasks on personal plans; Codex full-environment training has its own separate setting.
• Opting out does not cover thumbs feedback, which can send the whole conversation, and deleting a chat does not recall copies already de-identified.
• Google and Anthropic also use human reviewers; the difference here is how much of OpenAI's process is now on the record.

Sources: 404 Media: Inside Project Lily, OpenAI Data Controls FAQ, OpenAI: How your data is used to improve model performance, OpenAI Data Usage for Consumer Services FAQ, OpenAI Privacy Filter model card, openai/privacy-filter on GitHub, Gemini Apps Privacy Hub, Anthropic Privacy Center, TechCrunch, Search Engine Land, Crossing Hurdles.