On July 1, Google announced that it posted a Request for Comments to the Internet Engineering Task Force to formalize the Robots Exclusion Protocol specification. This is happening after an informal 25 year old standard for the internet.
Google had this to say on its blog:
“Together with the original author of the protocol, webmasters, and other search engines, we’ve documented how the REP is used on the modern web, and submitted it to the IETF. The proposed REP draft reflects over 20 years of real-world experience of relying on robots.txt rules, used both by Googlebot and other major crawlers, as well as about half a billion websites that rely on REP.”
When asked if anything is changing, Gary Illyes said that nothing would be changing.
There isn’t any official or definitive guide for keeping this standard up to date or making any specific syntax has to be followed as the Robots Exclusion Protocol hasn’t ever been formalized. Before, each major search engine has adopted robots.txt as a crawling directive but it isn’t even as official standard. This is going to change.
Google announced they are open sourcing the portion of its robots.txt that parses the robots.txt file. “We open sourced the C++ library that our production systems use for parsing and matching rules in robots.txt files,” Google said. You can see this library on Github today if you like.