Applying calibration to improve uncertainty assessment Conference Paper uri icon

abstract

  • The oil and gas industry is full of uncertainty. In addition to significant subsurface uncertainty and uncertainty in oil and gas prices, there are other risks, e.g., political, that contribute to uncertainty in oil and gas projects. Capen (1976) demonstrated that industry routinely underestimates uncertainty, often significantly. The cost associated with underestimating uncertainty, or overconfidence, can be substantial. According to McVay and Dossary (2012), moderate overconfidence and optimism can result in expected portfolio disappointment of more than 30%. While McVay and Dossary assessed the cost of underestimating uncertainty, they did not fully address how to better assess uncertainty. Capen and other authors have suggested that uncertainty can be best assessed through look-backs and calibration, i.e., comparing actual results to probabilistic predictions over time. While many recognize the importance of look-backs, calibration of probabilistic estimates does not appear to be commonly practiced in the industry. Part of the reason for this is lack of systematic processes and software for calibration. We developed a relational database application to track probabilistic estimates and assess their reliability over time. The Brier score and its components, mainly calibration, are used for assessing reliability. The system is general in the types of estimates and forecasts that it can monitor, such as production, reserves, times, costs, and quarterly earnings. Probabilistic forecasts may be assessed visually, using calibration charts, and quantitatively, using calibration and Brier scores. The calibration information can be used to modify probabilistic estimation and forecasting processes as needed to be more reliable. Calibration results from historical forecasts may be used to externally adjust future forecasts so they are better calibrated. Three experiments with historical data sets of predicted vs. actual quantities are presented to demonstrate that use of probabilistic forecast calibration results improves future forecasts. Consistent application of this approach and database application over time should improve probabilistic forecasts, resulting in improved company and industry performance. Copyright 2013, Society of Petroleum Engineers.

published proceedings

  • Proceedings - SPE Annual Technical Conference and Exhibition

author list (cited authors)

  • Fondren, M. E., McVay, D. A., & Gonzalez, R. A.

complete list of authors

  • Fondren, ME||McVay, DA||Gonzalez, RA

publication date

  • December 2013