RFC 9309: Robots Exclusion Protocol
In plain English — editorial summary, not part of the RFC
This document specifies and extends the "Robots Exclusion Protocol" method originally defined by Martijn Koster in 1994 for service owners to control how content served by their services may be accessed, if at all, by automatic clients known as crawlers. Specifically, it adds definition language for the protocol, instructions for handling errors, and instructions for caching.
Document record
- Document ID
- RFC9309
- Published
- September 2022
- Authors
- M. Koster; G. Illyes; H. Zeller; L. Sassman
- Status
- PROPOSED STANDARD
- Stream
- IETF
- Area
- —
- Pages
- 12
- Also known as
- —
Topics
Related documents
Ranked automatically by shared keywords, IETF area and stream — not by editorial selection.
Also filed under
About this page
The document record above — title, authors, date, status, stream, area, relationships, DOI and errata — is imported verbatim from the public RFC Editor index. The “in plain English” section is editorial: written by The metasystema editorial team, not part of the RFC. Where the two differ, the RFC text governs.
Last checked against the RFC Editor index on . RFCs are never revised after publication; changes are issued as new documents.
Data sources · Editorial policy · Report a correction · What is an RFC?