In any industry, there’s always going to be that part of the job that seems to drag on, and feels incredibly boring. In the industry of technical SEO, one of the most boring topics out there is robots.txt. For the most part, there isn’t any interesting problem that would need solving in the file, and most errors come from not understanding the directives or form typos. Overall, the idea behind a robots.txt file is to suggest to crawlers where they can and cannot go. Sounds fun, right?
But Patrick Stox will not only provide some dos and don’ts for creating you robots.txt file, but he will give some examples of companies who have gotten creative with their files! Examples include ASCII art, as well as job openings. Check it out by following the link below to find out some of the fun things found within various robots.txt files!
Search Engine Land: Fun with robots.txt