A META-DATA INFORMED EXPERT JUDGMENT AGGREGATION AND CALIBRATION TECHNIQUE

Abstract

Policy makers use expert judgment opinions elicited from experts as probability distributions, quantiles or point estimates, as inputs to decisions that may have economic or life and death impacts. While challenges in estimating probabilities in general have been studied, research that distinguished between non-probabilistic, i.e., physical, variables and probabilistic variables specifically in the context of meta-data based expert judgment aggregation techniques, and the errors associated with the predictions developed from such variables, was not identified. This research demonstrated that for two combined expert judgment meta-data bases, the distinction between physical and probabilistic variables was significant in terms of the extent of multiplicative error between elicited medians and realized values both before and after aggregation. The distinction also impacts the widths of bounds around aggregated point estimates. The research compared nine methods of aggregating estimates and obtaining calibrated bounds, including ones based on alpha stable distributions, quantile regression, and a Bayesian model. Simple parametric distributions were also fit to the meta-data. Methods were compared against criteria including accuracy, bounds coverage and width, sensitivity to outliers, and complexity. No single method dominated all criteria for either variable type. The research investigated sensitivity of results to level of realized value for a variable, such as infrequent events for probabilistic variables, as well as sensitivity of results to number of experts.