Stop Wasting Server Resources: Disallow Google from Crawling Your Action URLs

Have you ever felt like Googlebot is crawling your website a little too much? While Google’s search engine crawler is essential for getting your site indexed, it doesn’t need to spend time on pages where users take actions. This is where disallowing action URLs comes in.

Why Disallow Crawling Action URLs?

As Gary Illyes from Google puts it, crawlers “will not buy that organic non-GMO scented candle, nor do they care for a wishlist.” Pages like “add to cart” or “sign up” are useless for search engines and can overload your server with unnecessary requests. This can be a common source of complaints from website owners who see Googlebot crawling excessively.

Examples of Action URLs:

Imagine you have product pages with URLs like these:

• https://example.com/product/scented-candle-v1?add_to_cart
• https://example.com/product/scented-candle-v1?add_to_wishlist

These URLs with query parameters clearly indicate user actions and don’t provide valuable content for search engines.

How to Block Googlebot from Crawling Action URLs

There are two main ways to prevent Googlebot from crawling these action URLs:

1. Robots.txt: Add a disallow rule to your robots.txt file specifically targeting these URL patterns. This is the recommended approach.
2. HTTP POST Method: You can also convert your action URLs to use the HTTP POST method. While some crawlers might still be able to make POST requests, it’s less common than GET requests used for crawling.

A Word of Caution:

It’s important to note that there might be rare instances where Googlebot needs to access action URLs. A few years ago, there were reports of Googlebot adding products to carts to verify pricing data. This could be part of their merchant shopping experience score. If you’re heavily reliant on Google for product data and ranking, it might be wise to consult with an SEO specialist before completely blocking action URLs.

By taking steps to disallow crawling of action URLs, you can free up server resources and ensure that Googlebot focuses on the content that truly matters for search engine optimization.