Around the Web in Six Weeks: Documenting a Large-Scale Crawl
Conference Paper
Overview
Identity
Additional Document Info
Other
View All
Overview
abstract
2015 IEEE. Exponential growth of the web continues to present challenges to the design and scalability of web crawlers. Our previous work on a high-performance platform called IRLbot [28] led to the development of new algorithms for realtime URL manipulation, domain ranking, and budgeting, which were tested in a 6.3B-page crawl. Since very little is known about the crawl itself, our goal in this paper is to undertake an extensive measurement study of the collected dataset and document its crawl dynamics. We also propose a framework for modeling the scaling rate of various data structures as crawl size goes to infinity and offer a methodology for comparing crawl coverage to that of commercial search engines.
name of conference
2015 IEEE Conference on Computer Communications (INFOCOM)