THE QUALITY OF EXPERT JUDGMENT: AN INTERDISCIPLINARY INVESTIGATION
Files
Publication or External Link
Date
Authors
Advisor
Citation
DRUM DOI
Abstract
The potential impact of expert judgment on vital tasks, contributes to the desire of decision makers, to know the quality of expert judgment. For years, decision makers and stakeholders have struggled to find answers to "How accurate are experts" and "How close are the experts' estimates to the true values of quantities". Most models or tools used for the prediction of expert performance accuracy and estimates are based on historical performance records of individual experts. Decision makers are often limited to the knowledge of the attributes of the experts and their associated estimates. This dissertation focuses on two frameworks: (1) to estimate the true value of an unknown quantity, given the estimate of an expert, and (2) to effectively predict expert performance based on the attributes or qualifications of an expert. An extensive meta-analysis of validated expert judgment literature was conducted. The analysis identified the most commonly recommended attributes and evaluated the strength of association between attributes and expert performance.
Results from the analysis demonstrate nonlinear multiple regression relationships between the attributes of experts and their resulting performances. The validation case studies show that the empirical regression equation was effective in forecasting 50% of
the elicited experts' ability to provide accurate responses within 5% of their actual performance. Also, the model predicted 75% if the experts' performance with 15% of actual scores. . The results of the demonstrate that the equations derived to predict performance based on attributes are effective, and can be used to inform decision makers of the expected performance of their experts.
Results also demonstrated that the Bayesian equations developed to predict the true values of unknown quantities based on the estimates of experts are moderately effective. In the validation studies revealed a wide range of possible values for a given quantity, a result influenced by the large variance in the distribution from error.