Bill Slawski, a contributor at Moz was invited to try out a beta test for a new cloud-based crawler that allows users to handle extremely large websites.
This got Bill thinking about website crawling, and he wanted to share some of his past experiences with this particular technique in learning how a website works.
Read Bill’s article about his crawling experiences here in the link below:
http://moz.com/ugc/lessons-learned-while-crawling-the-web