A study showed that digital forensic examiner’s observations are biased by contextual information, and the consistency between the experts is low.
This is the first study designed to explore specifically the biasability and reliability of digital forensics (DF) decision-making, published in May 2021 Journal of FSI: Digital Investigation.
In the study, renowned cognitive bias expert Itiel Dror and co-author Nina Sunde illustrated that experts tended to find more or less evidence on a suspect’s computer hard drive to implicate or exonerate them depending on the contextual information about the investigation that they were given. Moreover, even those presented with the same information often reached different conclusions about the evidence.
The authors gave 53 digital forensics examiners from eight countries, including the UK, the same computer hard drive to analyze. Some of the examiners were provided with only basic contextual information about the case. In contrast, others were told the suspect had confessed to the crime, had a strong motive for committing it or that the police believed she had been framed.
Dror and Sunde selected 11 different “traces” within the evidence file that they could use to compare the observations of the digital forensic examiners. Some of the traces were easier to find, such as e-mail content, documents, and chats, while others required more in-depth examination; however, none were complex.
According to the study, none of the participants observed all 11 traces. Sixty-six percent of the participants identified 5 to 8 traces, 26% found 1 to 4 traces, and only 8% found 8 to 10 traces. When comparing the groups, higher numbers were observed in the guilt groups, followed by the control group.
“The innocence group observed the least number of traces, indicating that they were biased to find less evidence,” the authors write. “The guilt groups had the highest number of traces, indicating that they were biased to find more evidence. The control group was between the Innocence and guilt groups. However, there was very little difference between the strong guilt group and the control group in the proportion of observations, which indicates that the strong guilt context (the suspect had confessed) did not bias the participants to observe more traces than an examination without such context.”
The weak guilt group observed significantly more traces, suggesting that the ambiguous weak guilt context—wage conflict where the suspect had taken side with the workers—biased the group to find more traces.
Additionally, the researchers used a statistical measure to gauge the consistency of observations within each participant group that analyzed the evidence file based on the same contextual information. The team recorded an overall low/inadequate reliability of less than 0.667—for this specific co-efficient, values 0.80 or higher are considered strong, while those lower than 0.667 are considered inadequate. The highest reliability score was seen at the observation level for examiners receiving strong guilt context (0.51) and innocence context (0.44).
“Although high reliability between [digital forensic] examiners is anticipated, it is important to be aware that consistency does not imply accuracy or validity,” the authors write. “Consistency may arise from a variety of reasons. This entails that quality measures should not only be directed toward the tools and technology but also the human. It is not possible to calibrate a human the same way as a technical instrument. Still, measures such as blind proficiency testing through the use of fake test cases may provide knowledge about human performance.”
Still, the authors conclude there is a “serious and urgent need for quality assurance” in digital forensic examinations. To minimize bias, they suggest ensuring that all digital forensic examinations occur based on task-relevant contextual information only—exposure to “other” contextual information, such as whether a suspect has confessed or has prior arrests, should be minimized at all costs.
Jornal Reference: Nina Sunde, Itiel E. Dror, A hierarchy of expert performance (HEP) applied to digital forensics: Reliability and biasability in digital forensics decision making, Forensic ScienceForensic science is a method that applies a scientific process and technical approaches to study traces rooted in criminal activity or a litigious civil or administrative matter. Forensic science, also known as criminalistics, is a field... International: Digital Investigation, Volume 37, 2021, 301175, ISSN 2666-2817, https://doi.org/10.1016/j.fsidi.2021.301175.
