What’s the best way to learn about a client’s website besides simply browsing around the pages and checking things out? For an SEO who needs to know as much as possible about a site, they have to crawl it. By crawling a site, they can learn about the structure of the URLs. This allows the SEO to see how each page and link are all connected to each other.
Bill Slawski, a contributor at Moz was invited to try out a beta test for a new cloud-based crawler that allows users to handle extremely large websites.
This got Bill thinking about website crawling, and he wanted to share some of his past experiences with this particular technique in learning how a website works.
Read Bill’s article about his crawling experiences here in the link below:
http://moz.com/ugc/lessons-learned-while-crawling-the-web