Frédéric Dubut, a senior program manager at Microsoft working on Bing Search, said on Twitter Wednesday that when you create a specific section in your robots.txt file for its Bingbot crawler, you should make sure to list all the default directives in that section.

“If you create a section for Bingbot specifically, all the default directives will be ignored (except Crawl-Delay),” he said. “You MUST copy-paste the directives you want Bingbot to follow under its own section,” he added.


It seems that Bing has seen some sites complaining that Bingbot is crawling areas of their websites that they don’t want to see be crawled.  There are probably some webmasters who assumed that they gave Bingbot some specific instructions which would then follow the rest of the default directives not listed.  But, if you have a section for Bingbot, it only follows the directives you specifically listed in that section of your robots.txt.  Unless you have a specific section for Bingbot, it will follow the default directives.

When setting up your robots.txt file, make sure that all the search engine crawlers can efficiently crawl your site. If you set up specific directives for blocking, crawl delays or other directives, then make sure that all the search engine crawlers are listening to those directives. They may not listen if there are syntax issues, if you do not follow their protocols or they have issues accessing such directives.

For more on setting up a robots.txt for Bing, see the help documents.

Source –