Identifying Soft 404 Error Pages: Analyzing the Lexical Signatures of Documents in Distributed Collections Conference Paper uri icon

abstract

  • Collections of Web-based resources are often decentralized; leaving the task of identifying and locating removed resources to collection managers who must rely on http response codes. When a resource is no longer available, the server is supposed to return a 404 error code. In practice and to be friendlier to human readers, many servers respond with a 200 OK code and indicate in the text of the response that the document is no longer available. In the reported study, 3.41% of servers respond in this manner. To help collection managers identify these "friendly" or "soft" 404s, we developed two methods that use a Nave Bayes classifier based on known valid responses and known 404 responses. The classifier was able to predict soft 404 pages with a precision of 99% and a recall of 92%. We will also elaborate on the results obtained from our study and will detail the lessons learned. 2012 Springer-Verlag.

published proceedings

  • Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

author list (cited authors)

  • Meneses, L., Furuta, R., & Shipman, F.

citation count

  • 13

complete list of authors

  • Meneses, Luis||Furuta, Richard||Shipman, Frank

publication date

  • October 2012