Around the Web in Six Weeks: Documenting a Large-Scale Crawl Conference Paper uri icon

abstract

  • 2015 IEEE. Exponential growth of the web continues to present challenges to the design and scalability of web crawlers. Our previous work on a high-performance platform called IRLbot [28] led to the development of new algorithms for realtime URL manipulation, domain ranking, and budgeting, which were tested in a 6.3B-page crawl. Since very little is known about the crawl itself, our goal in this paper is to undertake an extensive measurement study of the collected dataset and document its crawl dynamics. We also propose a framework for modeling the scaling rate of various data structures as crawl size goes to infinity and offer a methodology for comparing crawl coverage to that of commercial search engines.

name of conference

  • 2015 IEEE Conference on Computer Communications (INFOCOM)

published proceedings

  • 2015 IEEE CONFERENCE ON COMPUTER COMMUNICATIONS (INFOCOM)

author list (cited authors)

  • Ahmed, S. T., Sparkman, C., Lee, H., & Loguinov, D.

citation count

  • 2

complete list of authors

  • Ahmed, Sarker Tanzir||Sparkman, Clint||Lee, Hsin-Tsang||Loguinov, Dmitri

publication date

  • January 2015