School of Computer Science and Technology, Harbin Institute of Technology, Harbin, China

---

Abstract—Inspired by the concept of internet computing, DHT-based distributed Web crawling model is proposed to solve the bottlenecks of the traditional Web crawling systems. Based on this system model, we propose optimizations to reduce the download time of the Web crawling tasks in order to increase the efficiency of the system. The improvement on the download time is achieved by shortening the crawler-crawlee network distance. By utilizing the mapping mechanism of Content Addressable Network (CAN) over Network Coordinate System (NC), the issue can be mapped onto a problem of minimizing the distances between peers and resources on the DHT overlay. This paper focuses on reducing such distances, seeking to provide an improved location-aware infrastructure for distributed Web crawling. A new DHT-based distributed Web crawling model is proposed first. Then, under this model, a new method based on CAN’s splitting schemes is proposed which shows a significant decrease in crawlercrawlee distance against existing schemes. In addition, the issue of load balancing is also solved by combining the new method with old ones.

Index Terms—DHT, distributed Web crawling, Content Addressable Network, Network Coordinate System, location-aware

[PDF]

Cite: Xiao Xu, Weizhe Zhang, Hongli Zhang, and Binxing Fang, " Exploiting Location-aware Mechanism for Distributed Web Crawling over DHTs," Journal of Computers vol. 5, no. 11, pp. 1646-1654, 2010.