Reusing Search Data in Ranking and Selection: What Could Possibly Go Wrong? Academic Article uri icon


  • It is tempting to reuse replications taken during a simulation optimization search as input to a ranking-and-selection procedure. However, even when the random inputs used to generate replications are identically distributed and independent within and across systems, we show that for searches that use the observed performance of explored systems to identify new systems, the replications are conditionally dependent given the sequence of returned systems. Through simulation experiments, we demonstrate that reusing the replications taken during search in selection and subset-selection procedures can result in probabilities of correct and good selection well below the guaranteed levels. Based on these negative findings, we call into question the guarantees of established ranking-and-selection procedures that reuse search data. We also rigorously define guarantees for ranking-and-selection procedures after search and discuss how procedures that only provide guarantees in the preference zone are ill-suited to this setting.

published proceedings

  • ACM Transactions on Modeling and Computer Simulation

author list (cited authors)

  • Eckman, D. J., & Henderson, S. G.

citation count

  • 7

complete list of authors

  • Eckman, David J||Henderson, Shane G

publication date

  • January 2018