The Critical Race Framework Study: Standardizing Critical Evaluation for Research Studies That Use Racial Taxonomy

Abstract

Introduction: Race is one of the most common variables in public health surveillance and research. Yet, studies involving racial measures show poor conceptual clarity and inconsistent operational definitions. There does not exist a bias tool in the public health literature for structured qualitative evaluation in critical areas of critical appraisal – reliability, validity, internal validity, and external validity – for studies that use racial taxonomy. This study developed the Critical Race (CR) Framework to address a major gap in the literature.

Methods: The study involved three iterative phases to answer five research questions (RQs). Phase I was a pilot study of the CR Framework among public health faculty and doctoral students to assess measures of fit (RQ1) and to identify areas of improvement in training, instrumentation, and study design (RQ2). Study participants received training and performed a single article evaluation. Phase II was a national cross-sectional study of public health experts to assess perceptions of the revised training and tool to assess measures of fit (RQ1), to determine the influence of demographic and research factors on perceptions (RQ3), and to gather validity evidence on constructs (RQ4). In Phase III, three raters performed article evaluations to support reliability evidence (RQ4) and to determine the quality of health disparities and behavioral health research studies against the CR Framework (RQ5).

Analysis: We assessed the reliability of study results and the CR Framework using non-differentiation analysis, thematic analysis, missingness analysis, user data, measures of internal consistency for adopted instruments, interrater agreement, and interrater reliability. Validity was assessed using content validity (CVI and k*), construct validity, and exploratory factor analyses (EFA).

Results: The study recruited 30 highly skilled public health experts across its three phases as part of the final analytic sample. Phase I had poor reliability in which the results could not be confidently interpreted (RQ1) and indicated needed improvement in study design, training, and instrumentation (RQ2). Based on Phase II results, we met or exceeded acceptable thresholds for measures of fit – acceptability, appropriateness, feasibility, and satisfaction (RQ1).

Demographic or research factors were not associated with responses (RQ3). Interrater agreement was moderate to high among rater pairs (RQ4). Due to lack of confidence in significance testing, interrater reliability results were inconclusive. Overall data results showed excellent content validity. Based on EFA results, construct validity for reliability and validity items was poor to fair (RQ4). Data results were inconclusive on internal validity and external validity. The twenty studies used in critical appraisal showed low quality or no discussion when the Critical Race Framework was used (RQ5).

Discussion: The CR Framework study developed a tool and training with quality evidence for implementation effectiveness, content validity, and interrater reliability to fill a major gap in the public health literature. It contributed an innovative theory-based tool and training to the literature. Future research should seek to study individual perceptions and practices that influence outcomes of CR Framework application and to reduce barriers to ensure that minimum sample sizes can be met for additional testing.