An error rate tells us something about a forensic method’s performance under defined conditions. It is not automatically the probability that a particular expert’s conclusion in a particular case is wrong. Confusion between those concepts can distort how scientific evidence
Evidence is any form of proof, such as objects, materials, or scientific findings, presented to establish or disprove a fact in a legal proceeding. It is used to reconstruct events and link or exclude individuals Read Full Definition is presented to courts and the public.
- What is an error in forensic comparison?
- Why the denominator matters
- Method performance and method conformance
- What are black-box and interlaboratory studies?
- Why a single headline accuracy figure is misleading
- How could error rates be reported clearly?
- Are error rates equal to the probability a suspect is innocent?
- What about cognitive bias?
- Case example: a questioned bullet
- Scientific references
Pattern comparison disciplines, including firearms, footwear and friction ridge examination, must account for examiner decisions, unsuitable materials, inconclusive outcomes and differences in study design. The most informative research describes what was tested, who was tested, how decisions were scored and which conditions influenced performance.
What is an error in forensic comparison?
A forensic feature comparison may ask whether two samples originate from the same source or different sources. Depending on a validated method and its reporting scheme, the examiner might conclude support for common source, support for different sources, an inconclusive result, or that the item is unsuitable for comparison.
- False positive: an outcome supporting a same-source association when the ground truth in the study is different source.
- False negative: an outcome supporting different sources when the ground truth is the same source.
- Inconclusive: the method or examiner does not make a definitive source decision under the applicable criteria.
- Unsuitable: the questioned material lacks sufficient detail for the examination under the stated procedure.
These definitions sound straightforward. The challenge is how a study counts inconclusive and unsuitable decisions, what conclusions are permitted, and whether the study accurately reflects the intended casework.
Why the denominator matters
Imagine a hypothetical test set with 100 known nonmatching comparisons. An examiner reports 1 false positive, 69 correct exclusions and 30 inconclusive outcomes. A simple ratio using all comparisons is 1/100. Restricting the calculation to definitive decisions gives 1/70. Neither is inherently the single universal error rate; each answers a different question. A transparent report states the denominators and reports the frequencyFrequency is a fundamental concept in physics and wave theory. It refers to the number of times a specific point on a wave, such as a crest or trough, passes a fixed reference point in Read Full Definition of inconclusives separately.
Forensic error-rate research increasingly recognizes that inconclusive outcomes are not an administrative nuisance to remove silently from the data
Information in analog or digital form that can be transmitted or processed. Read Full Definition.
Method performance and method conformance
Swofford and colleagues (2024) distinguished method performance from method conformance. Performance concerns how well a method discriminates between propositions under the study conditions. Conformance concerns whether a result arose from following the method as defined. Both are relevant to reliability.
An examiner may follow a method correctly yet reach an inconclusive conclusion because the sample is unsuitable. Conversely, a technically accurate decision does not establish that every procedural requirement was followed.
What are black-box and interlaboratory studies?
In a black-box study, qualified practitioners examine samples whose true status is known to the researchers but not to participating examiners. Researchers analyze decision accuracyIn scientific and measurement contexts, "accuracy" refers to the degree of proximity or closeness between a measured value and the true or actual value of the measured quantity. Accuracy indicates how well a measurement reflects Read Full Definition, repeatability and reproducibility. Other designs assess individual analytical stages, equipment performance or the effects of particular variables.
Black-box studies are valuable, but they do not automatically measure every scenario encountered in operational casework. Sample difficulty, case context, prevalence of same-source comparisons and who volunteers to participate may influence results.
Why a single headline accuracy figure is misleading
A 2024 bullet comparison study reported that decision rates varied with bullet characteristics, firearm types, examination conditions and individual participants. The authors specifically cautioned against summarizing the study using one overall decision rate. A 2025 NIST working-group report discussed how such data should inform training, quality assuranceQuality assurance (QA) is a comprehensive set of activities and processes to ensure the overall quality and reliability of the work conducted within a laboratory. QA is a proactive approach that focuses on improving scientific Read Full Definition, casework and testimony.
The lesson is not that forensic methods are universally unreliable or invariably accurate. Reliability needs to be characterized for particular tasks and conditions, and evidence statements should not imply greater certainty than the data support.
How could error rates be reported clearly?
| Question | What a transparent study reports |
|---|---|
| Who was tested? | Examiner experience, recruitment and relevant training |
| What was tested? | Material quality, same-source and different-source comparisons, range of difficulty |
| Which outcomes were allowed? | Decision categories, suitability and inconclusive rules |
| What is the denominator? | Number of eligible comparisons for each reported rate |
| How much uncertainty exists? | Confidence intervals or appropriate uncertainty estimates |
| What limits generalization? | Scope, casework representativeness and potential biases |
Are error rates equal to the probability a suspect is innocent?
No. An error rate estimated from a controlled comparison study describes decisions made across a specified test set. It does not incorporate all prior information, case circumstances or the legal questions needed to calculate the probability of innocence or guilt. Confusing a method’s error characteristics with a case-specific posterior probability is a form of statistical misinterpretation.
What about cognitive bias?
Task-irrelevant information can influence human decision-making. Laboratories should consider measures such as information management, sequential disclosure where appropriate, blind or independent testing, documented decision criteria and quality review. Any particular intervention should itself be assessed for feasibility and effectiveness.
Case example: a questioned bullet
An examiner compares a questioned bullet with a known test fire and reaches an inconclusive outcome. The correct account is not that the firearm matched or was excluded. The laboratory records the actual examination, the evidential limitations and the decision category. If an attorney cites a research error rate, the relevant study’s sample types, decision rules and uncertainties should be disclosed.
Read Simply Forensic’s guide to forensic propositions and trace evidence overview for related interpretation concepts.
Scientific references
- Swofford et al. (2024). Inconclusive Decisions and Error Rates in Forensic Science. Forensic Science International: Synergy.
- Hicklin et al. (2024). Accuracy and Reproducibility of Bullet Comparison Decisions by Forensic Examiners.
- Bullet Black Box Working Group (2025). Implications of the Bullet Black Box Study. NIST.
The numerical example is hypothetical and is not an observed result from the cited studies.
