Identifying “Soft 404” Error Pages: Analyzing the Lexical Signatures of Documents in Distributed Collections Conference Paper uri icon

abstract

  • Collections of Web-based resources are often decentralized; leaving the task of identifying and locating removed resources to collection managers who must rely on http response codes. When a resource is no longer available, the server is supposed to return a 404 error code. In practice and to be friendlier to human readers, many servers respond with a 200 OK code and indicate in the text of the response that the document is no longer available. In the reported study, 3.41% of servers respond in this manner. To help collection managers identify these "friendly" or "soft" 404s, we developed two methods that use a Naïve Bayes classifier based on known valid responses and known 404 responses. The classifier was able to predict soft 404 pages with a precision of 99% and a recall of 92%. We will also elaborate on the results obtained from our study and will detail the lessons learned. © 2012 Springer-Verlag.

author list (cited authors)

  • Meneses, L., Furuta, R., & Shipman, F.

citation count

  • 8

publication date

  • January 2012