How to scan thousands of domains for page rank at a time.

I developed a Java program to scan unlimited URLs for pagerank. The first 3,000 URLS went by at a rate of 5-6 URLs / second (it finished in just over five minutes). Then, it started giving me errors. The program didn't crash, but I could tell that Google was no longer accepting my requests.
Is there a way to bypass these limitations? Or should I distribute the program as is, and recommend users to only scan 3,000 URLs at a time?

See original post by xtmx

Leave a Reply