For a while, there has been some speculation around an Apple Web Crawler, and now it’s seems that there’s finally been some confirmation. Apple posted a help document that proves the existence of the company’s very on web crawler, Applebot.
So what exactly will the new Applebot crawl cover exactly? Apple said that their new web crawler is “used by products including Siri and Spotlight Suggestions.”
The user-agent will typically follow the following string but will contain “Applebot” in it always:
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/600.2.5 (KHTML, like Gecko) Version/8.0.2 Safari/600.2.5 (Applebot/0.1)
If there’s any concerns that SEOs and webmaster would have over this new web crawler, don’t worry. Apple has said it will respect the customary robots.txt rules and robots meta tags. AppleBot currently originates in the 17.0.0.0 net block. Apple will follow what you’ve mentioned for GoogleBot if you haven’t mentioned AppleBot in your robots.txt directive. Since AppleBot follows GoogleBot so closely, you can block only GoogleBot if you don’t want anything crawled. Despite the fact you only have to block Googlebot, it’s always a good idea to make sure you’ve blocked both of the web crawlers, just to be safe.
Any odd activity you see with AppleBot, you’ll be able to reach out to Apple about it at Apple-NOC “at” apple.com.
It’s also been reported by Apple Insider that Apple is “rapidly-expanding internal search group” to create their own web search engine via Spotlight. So does this mean anything as far as serious competition with Google? We’ll all just have to see.