Authors: Keith L. Monson PhD, Erich D. Smith MSFS, Eugene M. Peters PhD
Journal: Journal of Forensic Sciences
Publication Date: July 2, 2023
Summary:
In this study, the authors investigate the repeatability and reproducibility of forensic firearms examinations conducted by qualified forensic firearms examiners. The examiners compared bullets and cartridge cases fired from different types of firearms and provided opinions based on the Association of Firearm & Tool Mark Examiners (AFTE) Range of Conclusions, which includes categories such as Identification, Inconclusive, Elimination, or Unsuitable.
The key findings of the study are as follows:
- Repeatability: When examiners were confronted with the same comparison a second time, they showed high repeatability. The repeatability for known matches was 78.3%, and for known nonmatches, it was 64.5% when averaged over both bullets and cartridge cases. This suggests that examiners tend to make consistent decisions when presented with the same material again.
- Reproducibility: Reproducibility was also assessed, which measures the agreement between different examiners evaluating the same material independently. Reproducibility was slightly lower than repeatability, with a rate of 67.3% for known matches and 36.5% for known nonmatches when averaged over bullets and cartridge cases. This indicates that while examiners’ decisions are generally consistent within themselves, there is more variability when different examiners are involved.
- Disagreements: Most observed disagreements occurred within the Inconclusive categories. Fully reversed conclusions, where an Identification became an Elimination or vice versa, were rare. False Identifications were very few, and none were repeated by a second examiner, demonstrating the effectiveness of blind verification as a quality control measure.
- Impact of Categorization: The study also explored the impact of categorization methods on agreement. Recategorizing conclusions, such as compressing Inconclusive categories, led to increased agreement, especially for nonmatching comparison sets.
- Reliability vs. Accuracy: The study emphasizes that reliability, which measures consistency in decision-making, should not be conflated with accuracy, which assesses correctness. Examiners’ decisions were found to be reliable, indicating that they tend to avoid making false Identifications when comparing non-matching items and false Eliminations when comparing matching items.
- Variation Among Examiners: Differences in training, experience, laboratory protocols, and individual judgment were identified as sources of variation in examiners’ decisions. These factors contributed to variations in their conclusions.
- Quality Control Measures: The study underscores the value of blind verification as a quality control measure in casework, as it helped ensure that a second examiner did not repeat false Identifications.
In conclusion, this study provides valuable insights into the repeatability and reproducibility of forensic firearms examinations. It highlights the consistency of examiners’ decisions and sheds light on factors influencing variations in conclusions. The findings contribute to ongoing discussions on the accuracy and quality of forensic examinations in the field of firearms analysis.
Journal Reference: Monson, K. L., Smith, E. D., & Peters, E. M. (2023). Repeatability and reproducibility of comparison decisions by firearms examiners. Journal of Forensic Sciences, 68(5), 1721-1740. https://doi.org/10.1111/1556-4029.15318
