It’s been announced by Google that they are are now supporting their crawlers with locale-adaptive web pages. This feature is targeting those pages that change the content based on the user’s country origin and/or language setting.
As an example, if you visit a page somewhere in the US, but then visit the same page, but it’s a Spanish version, the pages will automatically change the content on the page based on that language. Google didn’t handle this particular function very well at all previously.
But now, the search engine will be able to handle this type of content by sending GoogleBot from different IPs across the world, as well as allowing it to set language settings.
Two methods include:
- Geo-distributed crawling – Here, Googlebot will use IP addresses that appear to be coming from outside the US, as well the current IP addresses that appear to be from the US that Googlebot current uses.
- Language-dependent crawling – This is where Googlebot would crawl with an Accept-Language HTTP header in the request.
When it comes to having content specific to different countries or languages, Google is recommending that you use URLs or TLDs based on the different countries or languages.
This is what Google said:
“These new configurations do not alter our recommendation to use separate URLs with rel=alternate hreflang annotations for each locale. We continue to support and recommend using separate URLs as they are still the best way for users to interact and share your content, and also to maximize indexing and better ranking of all variants of your content.”
This should help solve a problem that Google has had when it comes to understanding sites that display locale-adaptive techniques. Google said, “new crawling configurations are enabled automatically for pages we detect to be locale-adaptive, you may notice changes in how we crawl and show your site in Google search results without you altering your CMS or server settings.”
Check out this help document for more details.