Authors: Keith L. Monson PhD, Erich D. Smith MSFS, Eugene M. Peters PhD
Journal: Journal of Forensic Sciences
Publication Date: July 2, 2023
Summary:
In this study, the authors investigate the repeatability and reproducibility of forensic firearms examinations conducted by qualified forensic firearms examiners. The examiners compared bullets and cartridge cases fired from different types of firearms and provided opinions based on the Association of Firearm & Tool Mark Examiners (AFTE) Range of Conclusions, which includes categories such as Identification, Inconclusive, Elimination, or Unsuitable.
The key findings of the study are as follows:
- Repeatability: When examiners were confronted with the same comparison a second time, they showed high repeatability. The repeatability for known matches was 78.3%, and for known nonmatches, it was 64.5% when averaged over both bullets and cartridge cases. This suggests that examiners tend to make consistent decisions when presented with the same material again.
- Reproducibility: Reproducibility was also assessed, which measures the agreement between different examiners evaluating the same material independently. Reproducibility was slightly lower than repeatability, with a rate of 67.3% for known matches and 36.5% for known nonmatches when averaged over bullets and cartridge cases. This indicates that while examiners’ decisions are generally consistent within themselves, there is more variability when different examiners are involved.
- Disagreements: Most observed disagreements occurred within the Inconclusive categories. Fully reversed conclusions, where an Identification became an Elimination or vice versa, were rare. False Identifications were very few, and none were repeated by a second examiner, demonstrating the effectiveness of blind verification as a quality control measure.
- Impact of Categorization: The study also explored the impact of categorization methods on agreement. Recategorizing conclusions, such as compressing Inconclusive categories, led to increased agreement, especially for nonmatching comparison sets.
- Reliability vs. AccuracyIn scientific and measurement contexts, "accuracy" refers to the degree of proximity or closeness between a measured value and the true or actual value of the measured quantity. Accuracy indicates how well a measurement reflects the correct value. Here are key points about accuracy: • True Value: Accuracy assesses how closely a measurement or reading corresponds to the true, known, or accepted value of the quantity being measured. It is a measure of correctness.
• Error Measurement: The degree of accuracy is often expressed in terms of measurement error, which is the difference between the measured value and the true value. An accurate measurement has a minimal error.
• High Accuracy: A measurement or instrument is considered highly accurate when its readings are very close to the true value, with minimal or negligible error.
• Precision vs. Accuracy: Accuracy should not be confused with precision. Precision relates to the reproducibility and consistency of measurements. While accuracy addresses correctness, precision addresses how closely repeated measurements agree with each other.
• Example: If a laboratory balance measures the weight of a sample as 4.55 grams, and the actual weight of the sample is indeed 4.55 grams, the measurement is considered accurate.
• Error Sources: Errors in measurements can arise from various sources, including instrument calibration, environmental conditions, operator technique, and inherent limitations of the measurement device.
• Accuracy Assessment: To assess accuracy, calibration processes and standardization procedures are often employed to ensure that measurement instruments are correctly aligned with known reference standards.
• Quantitative Evaluation: Accuracy can be quantitatively evaluated by calculating the absolute or relative error, which expresses the difference between the measured value and the true value as a percentage or a fraction.
• Importance: In scientific research, quality control, manufacturing, and various fields, accuracy is essential for making informed decisions, ensuring product quality, and achieving reliable and credible results.
• Measurement Instruments: The accuracy of measurement instruments is a critical consideration in fields such as metrology, engineering, chemistry, and physics, where precise and accurate measurements are vital.
• Verification and Validation: To ensure the accuracy of measurements and instruments, verification and validation processes are often carried out, including testing and comparing results against reference standards.
In summary, accuracy in measurement refers to the degree of closeness between a measured value and the true or actual value of the quantity being measured. It is a fundamental concept in scientific research, quality control, and various industries where precise and reliable measurements are essential for making informed decisions and ensuring the quality and integrity of processes and products. More: The study emphasizes that reliability, which measures consistency in decision-making, should not be conflated with accuracy, which assesses correctness. Examiners’ decisions were found to be reliable, indicating that they tend to avoid making false Identifications when comparing non-matching items and false Eliminations when comparing matching items. - Variation Among Examiners: Differences in training, experience, laboratory protocols, and individual judgment were identified as sources of variation in examiners’ decisions. These factors contributed to variations in their conclusions.
- Quality Control Measures: The study underscores the value of blind verification as a quality control measure in casework, as it helped ensure that a second examiner did not repeat false Identifications.
In conclusion, this study provides valuable insights into the repeatability and reproducibility of forensic firearms examinations. It highlights the consistency of examiners’ decisions and sheds light on factors influencing variations in conclusions. The findings contribute to ongoing discussions on the accuracy and quality of forensic examinations in the field of firearms analysis.
Journal Reference: Monson, K. L., Smith, E. D., & Peters, E. M. (2023). Repeatability and reproducibility of comparison decisions by firearms examiners. Journal of Forensic Sciences, 68(5), 1721-1740. https://doi.org/10.1111/1556-4029.15318
