The key role of base rates : systematic review and meta-analysis of the predictive value of four risk assessment instruments

Abstract

AIMS OF THE STUDY: Many countries have seen a decline in recidivism rates over the past decades. These base rates are pertinent information for assessing the recidivism risk of offenders. They provide a foundation for clinical assessment and an empirical basis for risk assessment instrument norms, which inform expected recidivism rates. The present study explored the extent to which base rates influence the validity of risk assessment instruments.

METHODS: We systematically reviewed the available evidence on the discrimination ability of four well-established risk assessment instruments used to estimate the probability of recidivism for general (Level of Service Inventory-Revised [LSI-R]), violent (Violence Risk Appraisal Guide [VRAG]), sexual (Static-99R), and intimate partner violent offences (Ontario Domestic Assault Risk Assessment [ODARA]). We conducted a bivariate logit-normal random effects meta-analysis of sensitivity and false positive rates and modelled the positive and negative predictive values. We used base rates as reported in (a) the construction samples of each risk assessment instrument and (b) recent official statistics and peer-reviewed articles for different offence categories and countries. To assess the risk of bias, we used the Joanna Briggs Institute Critical Appraisal Checklist for Diagnostic Test Accuracy Studies.

RESULTS: We screened 644 studies and subsequently analysed 102, of which 96 were included in the systematic review and 24 in the meta-analyses. Discrimination was comparable for all four instruments (median area under the curve = 0.68–0.71). The information needed to calculate summary statistics of sensitivity and false positive rate was often not reported, and a risk of bias may be present in up to half of the studies. The largest summary sensitivity and false positive rate were estimated for the ODARA, followed by the LSI-R, the VRAG, and the Static-99R. If base rates are low, positive predictive values tend to be relatively low, while negative predictive values are higher: positive predictive value = 0.032–0.133 and negative predictive value = 0.985–0.989 for sexual offences; positive predictive value = 188–0.281 and negative predictive value = 0.884–0.964 for intimate partner violence; positive predictive value = 0.218–0.241 and negative predictive value = 0.907–0.942 for violent offences; positive predictive value = 0.335–0.377 and negative predictive value = 0.809–0.810 for general offences.

CONCLUSIONS: When interpreting the results of individual risk assessments, it is not sufficient to provide the discrimination of the instrument; the risk statement must also address the positive predictive value and discuss its implications for the specific case. As recidivism rates are neither stable over time nor uniform across countries or samples, the primary interpretation of risk assessment instruments should rely on the percentile rank. Expected recidivism rates should be interpreted with caution. However, our results are drawn from a limited database, as studies not reporting sufficient information were excluded from analyses and it was only possible to identify current base rates for modelling positive and negative predictive values for certain countries. International standards for consistently collecting and reporting base rates are important to better identify crime trends. Future research on the validity of risk assessment instruments should follow rigorous reporting standards.