In case you weren’t in the know, robots.txt may be a small part of SEO, but it’s still important for your website. They tell the search engines which pages to access and index on your website on which pages not to. Not only that, robots.txt files are great for keeping the search engines from accessing certain pages on your site is essential for both the privacy of your site and for your SEO.
Google’s John Mueller weighed in on the subject during a Google Search Central SEO office-hours hangout where he talked about robots.txt and whether or not it’s a good SEO practice to keep them at a decent size.
David Zieger, who is the SEO manager for a large German news publisher, who joined John in the stream, says that that larger robots.txt files might be a concern. In this case, he’s talking files that are over 1,500 lines with a big list of disallows that keeps getting added to as time goes on.
Are there really negative SEO efforts that can result from a huge robots.txt file?
Check out the video below for the answer from John Mueller himself!