Computer-Based Test Interpretation Software: Its Effect on School Psychologist Decision Making

Abstract

The purpose of this study was to assess the utility of interpretative software for school psychological decision making in its role as a decision aid. One hundred two professional school psychologists were provided case data and asked to make a series of diagnostic and prognostic decisions based on the case material. One subject group received case material only and each of the other two groups received one of two variations of the narrative output generated by a computer-based test interpretation software package in addition to the case material. The subjects were also asked which data sources were most influential as they made their decisions. Diagnostic agreement among the psychologists within each group was analyzed by Kendall's coefficient of concordance or weighted Kappa. For each decision there were no significant differences in agreement between those psychologists who had access to the decision aids and those who did not. Chi-square and Freidman analysis of variance results for similarity of diagnosis across groups were mixed with some trends suggestive of greater similarities of decisions among the subjects utilizing different variations of the computer output than among decisions made by unaided psychologists. Further the school psychologists overwhelmingly indicated that test data and behavioral observations were the most influential data sources for their decisions and that computer-based data sources were the least influential . Also there appeared to be no significant relationship between school psychologist professional experience and the perceived influence of the case data sources as well as little relationship between degree of experience in using computers to the data sources considered to be useful in the decision making process. The results were discussed in terms of psychological decision theory. Trends in the data suggested the computer narrative was most effective in situations where it was necessary to discriminate among ambiguous decision choices rather than in more clear cut situations. It was concluded that computer-based decision aids have the potential to debias the decision process, but that definitive changes will not come until the technology is improved and school psychologists become more familiar with the use of computers.