A commitment, not a completed switch

OpenAI said on October 30 that it will honor a common robots.txt signal through which website operators can express preferences about foundation-model training.

The company aims to implement it in the coming months and says it will publish implementation details later. Existing crawler controls remain in place. The announcement does not establish that the new signal is already being applied.

The intended benefit is simpler coordination: a publisher could express the same preference to multiple participating providers instead of maintaining a separate instruction for each one. Whether that reduces work in practice will depend on implementation and adoption.

The linked document is still a proposal

OpenAI links to Machine-Readable Training Preference Signal, an individual Internet-Draft by Kevin Kelley, dated October 30. The IETF tracker explicitly says it is not endorsed by the IETF and has no formal standing in its standards process.

The proposal uses the purpose-specific token __usage_ai-training within ordinary robots.txt groups. It allows preferences to apply to particular paths while retaining the file’s familiar rule structure.

It also keeps those preferences separate from crawler-specific traffic rules. A parser that does not implement the proposal can ignore the unfamiliar group. Publishing a signal is therefore not the same as installing an access barrier.

A shared vocabulary does not guarantee compliance

The related AI Preferences vocabulary draft distinguishes training from other uses, including search. It defines training around modifying a generative model’s learned parameters.

That draft is explicit about its limits: expressing and interpreting preferences does not ensure they are followed. Technical, contractual and legal mechanisms that might require compliance are outside its scope.

These documents describe communication between operators and recipients. They should not be presented as a universal enforcement system, or as a resolution of wider disputes about content use.

Current controls still have separate jobs

OpenAI’s existing crawler documentation distinguishes GPTBot, used for potential training material, from OAI-SearchBot, used for search. It describes those settings as independent: a site can allow search while disallowing GPTBot.

That distinction matters when reading the new commitment. A training preference is not automatically a request to disappear from search, and a search setting is not a complete statement about every later use of content.

The useful next checks are OpenAI’s published implementation details and evidence of how participating providers process the signal. Model Current has inspected the documents, not tested provider compliance or changed any website’s crawler settings.

Sources

  1. OpenAI: shared training-preference commitmentOctober 30 primary announcement: voluntary common signal, intended implementation in coming months, later implementation update, existing controls retained. Commitment not presented as completed rollout or universal adoption.
  2. IETF tracker: Machine-Readable Training Preference SignalIndividual Internet-Draft 00 dated October 30 by Kevin Kelley; tracker explicitly not IETF-endorsed/no formal standards standing. Purpose token __usage_ai-training, path-scoped preferences and independent crawler rules; work in progress, not adopted standard.
  3. IETF tracker: AI usage-preferences vocabularyWorking vocabulary draft 08: training versus search categories; section 3.2 explicitly says it does not ensure preferences followed or address legal/contractual/technical enforcement. No copyright ruling or compliance guarantee inferred.
  4. OpenAI: current crawler documentationPrimary documentation checked October 31: GPTBot versus OAI-SearchBot settings independent. Used for current documented roles, not to assert proposed common signal already implemented; no compliance experiment or site-setting change.