When increasing the number of URLs webmasters could submit for indexing and crawling from 10 to 10,000 per day , Bing said, “Eventually search engines can reduce crawling frequency of sites to detect changes and refresh the indexed content.” Letting site owners submit URLs directly decreases Bing’s costs and possibly even facilitates higher quality search results. Bing partnered with Botify earlier this year in order to introduce a content submission API that enables immediate indexing, provides site owners with more control over their content and, potentially, better search visibility.
So far, Google hasn’t utilized these methods.
When asked about the two approaches to crawling and indexing content, Pedro Dias, managing partner at APIs3 and former Google search quality analyst, said “Everything is migrating more towards push than towards requests, so it kind of makes sense.”
“We have to see how much that affects infrastructure of companies, but I think it gives you more power over which information you deliver to Google, because before you [would] wait for Google to crawl everything, and if we move towards this model … you might have more control over what gets indexed, and you might be able to save more resources and do a better job,” he said.