Springer International Publishing Switzerland 2016. In a number of application areas, such as materials and genomics, where one wishes to classify objects, sample sizes are often small owing to the expense or unavailability of data points. Many classifier design procedures work well with large samples but are ineffectual or, at best, problematic with small samples. Worse yet, small-samples make it difficult to impossible to guarantee an accurate error estimate without modeling assumptions, and absent a good error estimate a classifier is useless. The present chapter discusses the problem of small-sample error estimation and how modeling assumptions can be used to obtain bounds on error estimation accuracy. Given the necessity of modeling assumptions, we go on to discuss minimum-meansquare- error (MMSE) error estimation and the design of optimal classifiers relative to prior knowledge and data in a Bayesian context.
