Opinion
No. 4:21-CR-00204
2023-04-26
UNITED STATES of America v. Hunter Ryan ANDERSON, Defendant.
Geoffrey W. MacArthur, Assistant U.S. Attorney, U.S. Attorney's Office, Williamsport, PA, for United States of America. Helen A. Stolinas, The Mazza Law Group, PC, State College, PA, for Defendant.
Geoffrey W. MacArthur, Assistant U.S. Attorney, U.S. Attorney's Office, Williamsport, PA, for United States of America. Helen A. Stolinas, The Mazza Law Group, PC, State College, PA, for Defendant.
MEMORANDUM OPINION
Matthew W. Brann, Chief United States District Judge
Defendant Hunter Anderson, charged with felon in possession of a firearm, challenges the admissibility of DNA identification evidence generated by TrueAllele, a computer software program that analyzes complex DNA mixtures using statistical modeling. Anderson asserts that TrueAllele and the methodology it employs are insufficiently reliable to pass muster under Rule 702 of the Federal Rules of Evidence and Daubert v. Merrell Dow Pharmaceuticals, Inc. The Court disagrees. TrueAllele has been tested and validated, subjected to peer review, and broadly accepted in the field of forensic science. Accordingly, Anderson's motion is denied.
I. BACKGROUND
A. The Case Against Anderson
On July 22, 2021, a federal grand jury returned a one-count indictment against Anderson: possession of a firearm by a felon, in violation of 18 U.S.C. § 922(g). According to the Government, on October 31, 2019, law enforcement officers arrested Anderson at a house on Hard Rock Lane in Centre County, Pennsylvania on a warrant "not directly related" to the instant matter. When the officers searched the home, they found in a closet near the room where Anderson slept "a backpack with a Smith & Wesson .40 caliber firearm and Anderson's ID."
Doc. 1 (Indictment).
Doc. 31 (Gov't Opp.) ¶ 2.
Id.
Further, the Government alleges that several weeks earlier—on October 10, 2019—Anderson fired a weapon at another person during a domestic dispute at 611 Mudlick Road in Centre County. Law enforcement recovered from the wall of the Mudlick Road residence a .40 caliber bullet. The Pennsylvania State Police Crime Laboratory in Harrisburg tested the .40 caliber bullet and found that it was a "ballistic match" to the firearm recovered from Hard Rock Lane. The firearm was then swabbed for DNA, but the State Police Crime Lab found multiple sources of DNA and "could not state within a degree of scientific certainty that there was a match with Anderson's DNA."
Id. ¶ 3.
Id. ¶ 4.
Id.
Id. ¶ 5.
As such, the State Police sent the DNA sample from the firearm to Cybergenetics, Inc., a private company founded by Dr. Mark W. Perlin that developed, owns, and operates TrueAllele, "a probabilistic genotyping computer system that interprets DNA evidence using a statistical model." According to Dr. Perlin, "in cases where human review might be less reliable or not possible"—for example, when a DNA sample contains DNA from multiple individuals or when its properties have been affected by external elements prior to collection—TrueAllele can be used to determine the probability that the sample matches the DNA of a given person. For this, TrueAllele employs a two-step identification process: the software first "infer[s]" genotypes contained in the sample using "evidence data," and then second, "match[es]" genotypes by "comparing evidence with a suspect relative to a population."
Doc. 33 (June 27, 2022, M. Perlin Decl. - SEALED) ¶ 4.
See id. ¶¶ 5-18.
Id. ¶ 10.
In the instant case, TrueAllele found that a match between the firearm sample and Anderson was 11.5 trillion times more likely than a coincidental match between the firearm and another Caucasian.
Doc. 31 (Gov't Opp.) ¶ 7.
B. The Daubert Motion
On June 13, 2022, Anderson filed a motion to determine the admissibility of DNA identification evidence generated by TrueAllele, asking the Court to exclude that DNA evidence as well as expert testimony by Dr. Perlin. The parties briefed the motion and requested a hearing on the matter.
See Doc. 28 (Anderson Mot. to Exclude).
See Doc. 29 (Anderson Br.) at 5-6; Doc. 31 (Gov't Opp.) at 1.
The Court held the hearing over two separate days—December 21, 2022, and January 26, 2023. On the first hearing date, the Government presented as a witness Dr. Perlin, who discussed the creation, function, and application of TrueAllele. On the second date, Anderson called two witnesses in rebuttal: Nathaniel D. Adams and Dr. Jeanna Neefe Matthews. Mr. Adams holds a bachelor of Science in Computer Science from Wright State University and has published three academic articles, all on the use of forensic DNA software, and testified in sixteen different court cases. Dr. Matthews has a bachelor's degree in Mathematics and Computer Science from The Ohio State University and a master's degree and Ph.D. in Computer science from the University of California at Berkeley; she is a professor at Clarkson University and has authored more than 150 academic publications and testified in six cases. Consistent with the declaration they prepared in advance of their testimony, Mr. Adams and Dr. Matthews asserted that TrueAllele "cannot be considered a reliable software program according to well-established principles of software engineering or part of an objective and validated analytical process."
See Doc. 42 (Dec. 21, 2022, Hearing Tr.); Doc. 53 (Jan. 26, 2023, Hearing Tr.).
See Doc. 42 (Dec. 21, 2022, Hearing Tr.).
See Doc. 53 (Jan. 26, 2023, Hearing Tr.).
DX 2.0 (N. Adams CV).
DX 1.0 (Dr. Matthews CV).
DX 10.0 (Adams-Matthews Declaration) ¶ 4.
Following the hearing, the parties submitted supplemental briefs on the motion. Accordingly, Anderson's motion to exclude the TrueAllele DNA identification evidence and corresponding expert testimony by Dr. Perlin is now ripe for disposition.
See Doc. 57 (Anderson Supp. Br.); Doc. 56 (Gov't Suppl. Br.).
II. LEGAL STANDARD
Federal Rules of Evidence 702 and 703 govern the admissibility of expert testimony and set forth certain criteria for admissibility. Expanding upon those rules, the Supreme Court of the United States explained the standard for admissibility of expert testimony in Daubert v. Merrell Dow Pharmaceuticals, Inc. In Daubert, the Supreme Court delegated to district courts a gatekeeping responsibility under Rule 702, requiring trial judges to determine at the outset whether an expert witness may "testify to (1) scientific knowledge that (2) will assist the trier of fact." That gatekeeping function demands an assessment of "whether the reasoning or methodology underlying the testimony is scientifically valid" as well as "whether that reasoning or methodology properly can be applied to the facts in issue." The Supreme Court explained that a district court "exercises more control over experts than over lay witnesses," since "[e]xpert evidence can be both powerful and quite misleading because of the difficulty in evaluating it."
509 U.S. 579, 113 S.Ct. 2786, 125 L.Ed.2d 469 (1993).
Id. at 592, 113 S.Ct. 2786.
Id. at 592-93, 113 S.Ct. 2786.
Id. at 595, 113 S.Ct. 2786 (internal quotation marks omitted).
Following Daubert, the United States Court of Appeals for the Third Circuit cast expert admissibility determinations in light of three basic requirements: (1) qualification; (2) reliability; and (3) fit. The qualification prong demands that the proffered expert possess sufficient "specialized knowledge" to testify as an expert. To satisfy the reliability prong, an expert's opinion "must be based on the 'methods and procedures of science' rather than on 'subjective belief or unsupported speculation.' " The Third Circuit has set forth eight non-exclusive factors that "a district court should take into account" when deciding the reliability of expert testimony:
In re Paoli R.R. Yard PCB Litigation, 35 F.3d 717, 741-43 (3d Cir. 1994).
Id. at 741.
Id. at 742 (quoting Daubert, 509 U.S. at 589, 113 S.Ct. 2786).
(1) whether the method consists of a testable hypothesis;Regarding the fit prong, the Third Circuit explained that admissibility depends on "the proffered connection between the scientific research or test result" and the "particular disputed factual issues." As such, "expert testimony based on assumptions lacking factual foundation in the record is properly excluded."
(2) whether the method has been subject to peer review;
(3) the known or potential rate of error;
(4) the existence and maintenance of standards controlling the technique's operation;
(5) whether the method is generally accepted;
(6) the relationship of the technique to methods which have been established to be reliable;
(7) the qualifications of the expert witness testifying based on the methodology; and
(8) the non-judicial uses to which the method has been put.
Id. at 742 n.8.
Id. at 743 (internal quotation marks omitted).
Meadows v. Anchor Longwall and Rebuild, Inc., 306 F. App'x 781, 790 (3d Cir. 2009).
The burden of proof for admissibility of expert testimony falls upon the party that seeks to introduce the evidence. However, as the Third Circuit has emphasized, "[t]he test of admissibility is not whether a particular [expert] opinion has the best foundation or whether it is demonstrably correct"; rather, "the test is whether the particular opinion is based on valid reasoning and reliable methodology." The Third Circuit further explained:
Oddi v. Ford Motor Co., 234 F.3d 136, 145 (3d Cir. 2000).
Id. at 145-46 (internal quotation marks omitted).
This standard is not intended to be a high one, nor is it to be applied in a manner that requires the plaintiffs to prove their case twice—they do not have to demonstrate to the judge by a preponderance of the evidence that the assessments of their experts are correct, they only have to demonstrate by a preponderance of evidence that their opinions are reliable.District courts must always be cognizant of the fact that "[t]he analysis of the conclusions themselves is for the trier of fact when the expert is subjected to cross-examination."
Id. at 145 (internal quotation marks omitted).
Id. at 146 (internal quotation marks omitted).
III. ANALYSIS
For the instant motion, Anderson predicates his objection to the DNA identification evidence generated by TrueAllele solely on the second Daubert factor: reliability. According to Anderson, the evidence "is not sufficiently testable," the "peer review that has been done has not been independent," there are "serious concerns about the calculation of the error rate," and "the program is not generally accepted in the relevant scientific community of software engineers." The Government, predictably, disagrees.
See Doc. 57 (Anderson Supp. Br.) at 29 ("The results of TrueAllele testing in this case, and the testimony of Dr. Mark Perlin, should not be introduced because the testimony has not been shown to be reliable pursuant to the Daubert standard, which is codified in Rule 702.").
Id. at 29-30.
To resolve this motion, the Court first surveys the relevant legal authority and then turns to the evidence at issue.
A. Relevant Legal Authority
Given the relatively recent emergence of probabilistic genotyping—at least as applied in the field of criminal law—the available legal authority concerning the admissibility of such evidence is limited. That said, this is not the first case in which a prosecuting authority has sought to introduce into evidence results generated by probabilistic genotyping software, nor is it the first time a defendant has attempted to exclude evidence of this nature. As the Court considers the instant motion, it finds particularly instructive three prior opinions.
The first, and most relevant, is a 2021 ruling by the United States Court of Appeals for the Sixth Circuit in United States v. Gissantaner. There, the Sixth Circuit assessed whether the district court erred in excluding DNA-sorting evidence—specifically, the results generated by a probabilistic genotyping software program named STRmix, which the Sixth Circuit recognized as one of TrueAllele's peer programs. The Sixth Circuit examined the DNA evidence against Rule 702 and a proper framing of the Daubert reliability factors and concluded that it "should be admitted."
990 F.3d 457 (6th Cir. 2021).
Id. at 460-62.
Id. at 463.
For the first factor (testability), the Sixth Circuit found that "STRmix can be tested," explaining that forensic scientists can determine whether the program generates either "false positive[s]" or "false negative[s]":
Suppose that one person, Aaron, contributed to a lab-created mixture, but another, Britney, did not. Forensic scientists can test STRmix to see whether it suggests that Aaron is a match for the mixture, but Britney is not. If STRmix suggests that Aaron is not a match for the mixture (by outputting a low likelihood ratio), that would be a false negative. If STRmix suggests that Britney is a match for the mixture (by outputting a high likelihood ration) that would be a false positive.According to the Sixth Circuit, "[e]ach possibility shows that STR mix is testable," as both affirm "that lab-created mixtures offer a way to assess the reliability of STRmix."
Id. at 464.
Id. (internal quotation marks, brackets, and citations omitted).
For the second factor (peer review), the Sixth Circuit emphasized that "[s]ubjecting a new technology to peer review and publication offers another measure of reliability," explaining that "[p]ublication in a peer-reviewed journal" demonstrates that the theory and procedures at issue "have been submitted to the scrutiny of the scientific community." Importantly, the Sixth Circuit noted that "this factor does not demand independent authorship," explaining that "[i]ndependent authorship may (or may not) represent the scientific ideal, but submission to peer review generally suffices under Daubert" because "[p]eer review contains its own independence": it involves "anonymously reviewing a given experimenter's methods, data, and conclusions on paper." Accordingly, the Sixth Circuit concluded that STRmix—which was the subject of "more than [fifty] published peer-reviewed articles," including at least two that were "done by individuals unconnected to the development of the software"—"clears this bar."
Id. at 464 (internal quotation marks and citations omitted).
Id. at 465, 468 (internal quotation marks and citations omitted).
Id. at 465.
For the third and fourth factors (error rate and standards for operation, respectively), the Sixth Circuit explained that the principal question is "[h]ow often . . . STRmix falsely suggest[s] a suspect matches a DNA sample." The answer, it found, was "[n]ot often":
Id.
When examining "false inclusions," one peer-reviewed study concluded, based on an analysis of the DNA of 300,000 people who were known not to be in a mixture, that STRmix had accurately excluded the non-contributors 99.1% of the time. Just 1% of the time, in other words, it gave a likelihood ratio suggesting that someone was included in the mixture who was not actually included in it.The Sixth Circuit deemed STRmix's low error rate unsurprising given the existence of, and adherence to, relevant industry standards. Specifically, the Sixth Circuit explained that "[t]he Scientific Working Group on DNA Analysis Methods, a national association of forensic laboratories sponsored by the FBI, has produced guidelines governing the use of this kind of software, guidelines that the Michigan State Police laboratory used in this case." The Sixth Circuit noted that "[t]he forensic scientist who ran the sample in this case began training with STRmix more than a year before analyzing the sample," and "[d]uring that time, the laboratory tested its copy of the software, using lab-created mixtures to establish 'internal validation' that STRmix reliably assisted the laboratory's work."
Id.
Id. at 466.
Id. at 467.
Finally, for the fifth factor (general acceptance), the Sixth Circuit concluded that STRmix "has garnered wide use in forensic laboratories across the country," as "[m]ore than 45 laboratories use it, including the FBI and many state law enforcement agencies." The Sixth Circuit acknowledged that probabilistic genotyping software "remains controversial" among computer scientists, but concluded that such criticism "does not mean that STRmix has fallen short of 'general' acceptance."
Id.
Id. at 469.
The second prior opinion instructive here is a 2012 ruling by the Superior Court of Pennsylvania in Commonwealth v. Foley. In that case, the Superior Court considered whether the trial court erred in admitting the DNA-related testimony of Dr. Perlin concerning a DNA probability analysis generated by TrueAllele—the same expert and software program at issue here. Although the Superior Court considered the admissibility of this evidence under the Frye test (as opposed to the Daubert analysis applied by federal courts), the holding is nevertheless relevant. Specifically, the Superior Court found "no legitimate dispute regarding the reliability of Dr. Perlin's testimony," noting "the extent of usage of Dr. Perlin's system" (it was "used by New York State for all their data banking," by "[t]he Allegheny County Crime Lab . . . for looking at mixtures in complex cases and DNA evidence," and by the "United Kingdom's Forensic Science Service . . . to analyze crime scene evidence") and that "TrueAllele has been tested and validated in peer-reviewed studies."
38 A.3d 882 (Pa. Super. 2012).
Id. at 888-89.
Id. at 889-90.
The final opinion of note is a 2022 ruling by the Court of Appeals of New York in People v. Wakefield. As in Foley, the case involved TrueAllele and Dr. Perlin. Specifically, the New York Court of Appeals assessed whether "the trial court abused its discretion in determining that TrueAllele is not novel but instead is 'generally accepted' under the Frye standard." The Court of Appeals held that "the relevant scientific community generally accepted TrueAllele's DNA interpretation process" and the "continuous probabilistic genotyping approach" TrueAllele employs, citing "developer and independent validation studies and laboratory internal validation studies, many published and peer-reviewed," as well as "its use in other jurisdictions" and its "approv[al] for use" by the FBI-sponsored Scientific Working Group, the American National Standards Institute ("ANSI"), and the National Institute of Standards and Technology ("NIST"). Although the defendant protested that general acceptance could not be established because Dr. Perlin and Cybergenetics had not disclosed the TrueAllele source code, the Court of Appeals disagreed.
38 N.Y.3d 367, 174 N.Y.S.3d 312, 195 N.E.3d 19 (2022).
Id. at 380, 174 N.Y.S.3d 312 (internal quotation marks and citation omitted).
Id. at 381-83, 174 N.Y.S.3d 312.
Id. at 383, 174 N.Y.S.3d 312 ("Disclosure of the TrueAllele source code was not needed in order to establish at the Frye hearing the acceptance of the methodology by the relevant scientific community.").
B. Reliability of TrueAllele
With the aid of the existing legal authority on the admissibility of DNA identification evidence generated by probabilistic genotyping software, the Court now turns to the evidence at issue in this motion—that is, the match statistic TrueAllele reported for Anderson's DNA and the DNA sample taken from the gun seized in October 2019. Consistent with the Sixth Circuit's analysis in Gissantaner, the Court's reliability inquiry focuses on the following five factors: (1) testability, (2) peer review, (3) error rate, (4) standards for operation, and (5) general acceptance.
1. Testability
For the first reliability factor, the Court considers "whether the premises on which" TrueAllele DNA generated the identification evidence "are testable—or, better yet, actually tested." The Third Circuit explains that " '[t]estability' has also been described as 'falsifiability,' " instructing that "[a] proposition is 'falsifiable' if it is 'capable of being proved false; defeasible.' " And "[p]roving a statement false typically requires demonstrating a counterexample empirically." Importantly, the question "is whether a method can be 'assessed for reliability,' not whether it always gets it right." As such, "[d]isputes about the adequacy of the theory's testing or about the accuracy of a theory's results, generally speaking, provide grist for adversarial examination, not grounds for exclusion."
United States v. Mitchell, 365 F.3d 215, 235 (3d Cir. 2004).
Id. (citing Webster's Third New International Dictionary 820 (unabridged ed. 1966)).
Id.
Gissantaner, 990 F.3d at 464 (citation omitted).
Id. (internal quotation marks, brackets, and citations omitted); see also Mitchell, 365 F.3d at 238 (holding that fingerprint identification analysis, although not necessarily subjected to "directed, specific actual testing," was "testable" such that it "assure[d] the opponent of proffered evidence the possibility of meaningful cross-examination (should he or someone else undertake the testing)").
As the Sixth Circuit held in Gissantaner, probabilistic genotyping software programs like TrueAllele "can be tested." Specifically, the Sixth Circuit explained that forensic scientists can use "lab-created mixtures" to assess the likelihood of a program generating a "false negative" (i.e., incorrectly suggesting that an individual who contributed to a lab-created mixture is not a match for the mixture) or "false positive" (i.e., incorrectly suggesting that an individual who did not contribute to the mixture is a match). Indeed, TrueAllele has be subjected to that type of testing. As the Government notes, Dr. Perlin testified about "multiple studies in which TrueAllele has been tested to determine whether its results are accurate and reproducible," including tests "against known DNA samples."
Id.
Doc. 56 (Gov't Supp. Br.) at 6 (citing Doc. 42 (Dec. 21, 2022, Hearing Tr. - Dr. Perlin Testimony) at 65:8-25, 67:7-12, 79:14-20).
Accordingly, the Court finds that the TrueAllele program can be, and has been, tested. This factor weighs in favor of admissibility.
2. Peer Review
The second factor asks whether the TrueAllele program has been subject to peer review. On this, there is no dispute: there have been multiple peer-review studies on TrueAllele. Indeed, the Superior Court of Pennsylvania in Foley concluded the same—and that was in 2014.
In re Paoli R.R. Yard PCB Litigation, 35 F.3d at 742 n.8.
See Doc. 56 (Gov't Supp. Br.) at 6-7 ("TrueAllele has also been subjected to peer review."); Doc. 57 (Anderson Supp. Br.) at 10 (acknowledging the "eight published validation studies in peer-review journals regarding TrueAllele").
38 A.3d at 889-90 ("TrueAllele has been tested and validated in peer-reviewed studies").
Here, the Government has identified eight separate peer-review studies published in journals of note, including five in the Journal of Forensic Sciences. One of these studies was prepared by Virginia's Department of Forensic Science, independent of Cybergenetics. And at least one other contained independent analysis—the paper produced with the Cuyahoga County Regional Crime Laboratory for Forensic Science contains results generated by Cybergenetics when it analyzed the data on its computers at its facility in Pittsburgh as well as results generated by the Cuyahoga County Crime Lab at its facility in Cleveland; they produced "the same match statistics," the "same answer."
See GX 16.0 (TrueAllele Validation Slide Deck) at 6.
Id. (noting the following peer-reviewed validation study: "Greenspoon SA, Schiermeier-Wood L, Jenkins BC. Establishing the limits of TrueAllele Casework: a validation study. Journal of Forensic Sciences. 2015;60(5):1263-1276"); see also Doc. 42 (Dec. 21, 2022, Hearing Tr. - Dr. Perlin Testimony) at 65:21-25 ("For example, the fourth paper down, Greenspoon, Schiermeier-Wood and Jenkins was done independently of Cybergenetics. That went out to four-person mixtures. That was done entirely by the Commonwealth of Virginia's Department of Forensic Science.").
Doc. 42 (Dec. 21, 2022, Hearing Tr. - Dr. Perlin Testimony) at 83:13-85:7.
Anderson attempts to downplay the significance of those peer-review studies, asserting that "peer review and publication . . . does not necessarily correlate with reliability." He emphasizes that Dr. Perlin was the principal or supporting author of all but one of the peer-review publications and that the journal reviewers did not have access to TrueAllele or its source code. But neither fact affects the outcome of the Court's analysis here. First, as the Sixth Circuit noted in Gissantaner, "this factor does not demand independent authorship," as "[p]eer review contains its own independence." Second, the validity of a peer-review study, for purposes of the Daubert reliability analysis, does not depend on whether the journal reviewers had access to the source code for the program at issue. The availability and analysis of source code had no bearing on the Sixth Circuit's analysis of this factor in Gissantaner, and was likewise deemed immaterial by the New York Court of Appeals in Wakefield.
Doc. 57 (Anderson Supp. Br.) at 9.
Id. at 10.
See id. at 465, 468.
Moreover, in this case, one of Anderson's experts, Dr. Matthews, indicated that it is not altogether surprising that the journal reviewers for the peer-review articles on TrueAllele were not provided "software artifacts" or "the source code to TrueAllele" because the reviewers were "not undertaking rigorous verification and validation activities." To be clear, Dr. Matthews gave that statement when explaining why, in her opinion, peer review is no substitute for software verification and validation. Dr. Matthews emphasized that peer reviewers "are not answering the question [of whether] this software [is] reliable enough to be introduced in . . . court in any particular case." That may be true, but it is of no moment. Peer review and publication does not, by itself, establish reliability; it is instead simply a factor in the reliability analysis. And for that factor, publication in a peer-review journal is typically sufficient.
Doc. 53 (Jan. 26, 2023, Hearing Tr. - Dr. Matthews Testimony) at 117:11-118:6.
Id.
Id. at 118:7-9.
In re Paoli R.R. Yard PCB Litigation, 35 F.3d at 742 n.8.
Gissantaner, 990 F.3d at 464 ("Publication in a peer-reviewed journal typically satisfies this consideration.").
Because TrueAllele has been the subject of multiple published peer-review studies, the factor weighs in favor of admissibility.
3. Error Rate
As the Sixth Circuit held in Gissantaner, when assessing the error rate for probabilistic genotyping software, the principal question is how often the software falsely suggests a suspect matches a DNA sample. Put differently, the error rate is the probability that the "likelihood ratio" (i.e., the match statistic) "suggest[s] that someone was included in the mixture who was not actually included in it."
Id.
During the evidentiary hearing, the Court put this question to Dr. Perlin directly, asking whether it was "possible to determine how often TrueAllele will falsely suggest a suspect matches a DNA sample." Dr. Perlin responded, "Yes. That's an error rate that has been calculated both in validation studies and on evidence by making comparisons with thousands of reference profiles and with newer methods by considering all possible samples." But, as reflected throughout Dr. Perlin's testimony, that answer contains two different conceptions of "error rate" as it pertains to TrueAllele. Resolving this factor requires disentangling those distinct concepts.
Doc. 42 (Dec. 21, 2022, Hearing Tr. - Dr. Perlin Testimony) at 161:1-3.
Id. at 161:4-8.
First, Dr. Perlin detailed an error rate for this case that he defined as "the chance that somebody whose DNA isn't on the gun might have a match statistic as strong as the one reported against [Anderson]." He explained that TrueAllele allows the operator to make this determination "by looking at the noncontributors": the program generates match statistics for the genotypes of random people with no connection to the case at issue (i.e., known noncontributors) and then compares those match statistics against the match statistic TrueAllele reported for the target individual (i.e., the defendant). Dr. Perlin stated that in this case, TrueAllele calculated match statistics for "all people on earth and all possible genotypes" and compared that with the match statistic reported for Anderson, resulting in an "error rate" of "one in 146 trillion people."
Id. at 56:18-20 (cleaned up).
Id. at 56:14-59:22.
Id. at 59:9-17.
Second, Dr. Perlin described an error rate generated through a "specificity" analysis, the component of validation studies designed to answer two questions: (1) "[i]f someone is not in the DNA, are you showing that they're not there"; and (2) "to the extent you're finding people who are not there, as a false inclusion, can you use that to establish error rates." Dr. Perlin testified that TrueAllele has been subjected to this type of specificity analysis. In particular, he highlighted a 2014 paper published jointly with the Commonwealth of Virginia's Department of Forensic Science in which he and his co-authors "compared 100 of the matching genotypes with 10,000 random references for a million nonmatching comparisons." For the "bulk of the comparisons," TrueAllele produced "[n]o information," meaning it was simply "excluding people who [were] not there." And for the known noncontributors whose match statistics nevertheless got "past an exclusionary result," Dr. Perlin and his co-authors were able to "calculate an error rate." They found that for "match statistics over 100, but less than 10,000, the error rate was one in a million, or point 00001 percent." Dr. Perlin described this as markedly superior to human review, which had an "error rate of false inclusion" ranging between two and six percent.
Id. at 73:20-74:1.
Id. at 74:2-3.
Id. at 74:3-14.
Id. at 74:15-24.
Id. at 75:8-76:15.
Id. at 76:11-77:9.
Although Dr. Perlin characterizes both concepts as error rates, they are fundamentally distinct. The former tests the reliability of the match statistic reported for a particular defendant in a particular case; it does not provide a general rate of error broadly applicable to the TrueAllele program (i.e., the likelihood that TrueAllele, in any case, will falsely report a high match statistic for an individual whose DNA was not in the sample tested). As Anderson correctly describes it, this purported "error rate" is not "provided for the method itself," but is instead offered "for each result the report generates, using the same math and operation of the same program being queried." Accordingly, it is not an error rate—at least, not as defined by the Sixth Circuit in Gissantaner and not as demanded by this factor of the reliability analysis.
Doc. 57 (Anderson Supp. Br.) at 14.
The latter, however, aligns more closely with the definition of "error rate" as used in this context. By assessing in a neutral study the match statistics for known non-contributors, Dr. Perlin and his co-authors at the Virginia Department of Forensic Science were able to determine a "rate of false inclusion"—that is, how many known noncontributors produced match statistics "past an exclusionary result" (i.e., at rates over 100 or 1,000 or 10,000). That peer-reviewed validation study is similar to the study in Gissantaner—"an analysis of the DNA of 300,000 people who were known not to be in a mixture," which revealed that the probabilistic genotyping software at issue, STRmix, "accurately excluded the non-contributors 99.1% of the time"—that the Sixth Circuit cited when concluding that STRmix had a viable error rate.
Doc. 42 (Dec. 21, 2022, Hearing Tr. - Dr. Perlin Testimony) at 74:2-77:9.
The discussion in this case of error rates for TrueAllele has been less than clear—lost in translation, it seems, as the parties endeavored to import concepts from the fields of forensic and computer science into the legal arena. That said, Dr. Perlin and the Government have presented evidence of validation studies showing how often TrueAllele falsely suggests a known non-contributor matches a DNA sample. In other words, experts at Cybergenetics and in independent crime labs have tested the reliability of TrueAllele's results by calculating an error rate. This factor therefore weighs in favor of admissibility.
4. Standards for Operation
Next, Anderson asserts that TrueAllele is not subject to the relevant industry standards and controls, characterizing Dr. Perlin as particularly hostile to such oversight and guidance: "Not only is there an absence of standards and controls employed with respect to the use of TrueAllele in this case, Dr. Perlin eschews the need for such standards or controls." The Government disputes this, arguing that "TrueAllele complies with industry standards"—most notably, the standards promulgated by the FBI-sponsored Scientific Working Group, ANSI, and the American Academy of Forensic Sciences Standards Board ("ASB"). The Court agrees with the Government.
Doc. 57 (Anderson Supp. Br.) at 16.
Doc. 56 (Gov't Supp. Br.) at 7; see also Doc. 42 (Dec. 21, 2022, Hearing Tr. - Dr. Perlin Testimony) at 115:11-15 ("Q. Does—does TrueAllele follow standards such as SWGDAM [i.e., the standards issued by the Scientific Working Group] and ANSI? A. Yes. It follows the same standards that anyone else in [the] forensic science community would follow for testing the reliability of their—of their software systems.").
For the Scientific Working Group standards, Dr. Perlin testified that after "initial guidelines . . . came out in 2010," the Scientific Working Group "issued guidelines for validating probabilistic genotyping systems like TrueAllele." Dr. Perlin prepared a document detailing how TrueAllele complies with these guidelines, separating the guidelines into its component provisions and identifying which TrueAllele validation studies establish compliance for each provision.
Doc. 42 (Dec. 21, 2022, Hearing Tr. - Dr. Perlin Testimony) at 105:18-24.
GX 7.0 (TrueAllele Standards Compliance) at Appendix 1 (TrueAllele Validation Summary for the Scientific Working Group guidelines).
Anderson does not dispute the applicability of the Scientific Working Group guidelines or Dr. Perlin's proffer about the validation studies and applicable guidelines provisions; instead, Anderson emphasizes that "TrueAllele had not been the subject of an internal validation study by [the Pennsylvania State Police]." According to Anderson, this distinguishes the instant case from Gissantaner, where the Michigan State Police laboratory operated the probabilistic genotyping software at issue (STRmix) in accordance with the Scientific Working Group's standards and performed internal validation studies to ensure the software functioned properly. But the Court finds this distinction both unsurprising and immaterial.
Doc. 57 (Anderson Supp. Br.) at 16-17 (citing Doc. 53 (Jan. 26, 2023, Hearing Tr. - N. Adams Testimony) at 56:4-7).
Id. at 17 (citing Gissantaner, 990 F.3d at 467).
In Gissantaner, the Michigan State Police obtained a DNA sample from a gun recovered from the defendant's home, and then "an analyst with the Michigan State Police laboratory took information about the DNA present in the mixture and entered it into STRmix to estimate how much of the DNA came from each person." Because the State Police lab operated STRmix and used the program to generate the disputed likelihood ratio for the defendant, the Sixth Circuit appropriately noted the significance of the State Police lab's compliance "with the guidelines promulgated by the Scientific Working Group, as confirmed through an audit performed by the FBI."
Id. at 467.
Here, however, the Pennsylvania State Police did not operate TrueAllele or use the program to generate the disputed match statistic. Instead, Cybergenetics performed that work from its facility in Pittsburgh: the Pennsylvania State Police collected the DNA samples (from the gun and from Anderson, respectively) and then sent a case packet containing the relevant data to Cybergenetics for analysis on TrueAllele. Accordingly, the question here is whether Cybergenetics and its employees at the facility in Pittsburgh operated TrueAllele in accordance with the Scientific Working Group's standards and performed internal validation studies to ensure the software functioned properly. By all accounts, they did.
Doc. 42 (Dec. 21, 2022, Hearing Tr. - Dr. Perlin Testimony) at 124:14-23 (noting that Cybergenetics "ran the DNA" after receiving it from the Pennsylvania State Police, "processing" it on TrueAllele).
Id. at 119:21-124:23 (describing how "the initial analysis of the DNA was done by the Pennsylvania State Police," which then provided Cybergenetics "a case packet requesting that you provide a free analysis of the evidence in this case"; after Cybergenetics "provided the initial match statistic," the State Police "agreed to pay for the report," which Cybergenetics prepared after it "ran the DNA").
See GX 7.0 (TrueAllele Standards Compliance) at Appendix 1 (TrueAllele Validation Summary for the Scientific Working Group guidelines).
For the guidelines ANSI and ASB issued regarding the validation of probabilistic genotyping systems (referred to as "ANSI/ASB Standard 018"), Dr. Perlin testified that Cybergenetics "produce[d] a document that shows paragraph by paragraph throughout the document how TrueAllele complies with the standard." The Government introduced that document into evidence at the evidentiary hearing, and it does indeed detail TrueAllele's compliance with the various provisions of ANSI/ASB Standard 018.
Doc. 42 (Dec. 21, 2022, Hearing Tr. - Dr. Perlin Testimony) at 106:8-18.
See GX 7.0 (True Allele Standards Compliance) at 14-22.
In response, Anderson homes in on a particular provision of ANSI/ASB Standard 018—Guideline 4.7—and argues that Dr. Perlin's testimony establishes that TrueAllele is not in compliance with that provision. Specifically, Anderson contrasts Dr. Perlin's testimony that TrueAllele "doesn't need calibration," with Guideline 4.7, which, Anderson asserts, provides that "probabilistic genotyping software systems are calibrated using historical data ideally from the same laboratory in which that system is employed."
Doc. 57 (Anderson Supp. Br.) at 17.
Doc. 42 (Dec. 21, 2022, Hearing Tr. - Dr. Perlin Testimony) at 84:13-14.
Doc. 57 (Anderson Supp. Br.) at 17.
But that argument is flawed for two reasons. First, the language Anderson points to is not in Guideline 4.7 itself; rather, the language comes from "Annex A" of ANSI/ASB Standard 018, which provides "supporting information" for the Standard's various provisions. Guideline 4.7 provides only that "[p]rior to implementation, the laboratory shall verify the functionality of its defined software settings and parameters utilizing different data sets than what were originally used to establish those settings and parameters." And to that end, Cybergenetics explains that "[s]ufficient testing is done on a variety of data sets before new software is distributed and used in routine processing. The testing is documented, and any new software features are documented prior to release." That statement (which Anderson has not refuted) seemingly satisfies the requirements of Guideline 4.7.
See ANSI/ASB Standard 018, 1st Ed. 2020, https://www.aafs.org/sites/default/files/media/documents/018_Std_e1.pdf.
GX 7.0 (True Allele Standards Compliance) at 8.
Id. at 22.
Second, a full reading of Annex A's supporting information on Guideline 4.7 undercuts Anderson's claim that Cybergenetics' operation of TrueAllele, as described by Dr. Perlin, is at odds with the Guideline's requirements. The relevant portion of Annex A contains the following explanation:
[Guideline 4.7] serves to further verify the established software settings and parameters. Probabilistic genotyping software systems are calibrated using historical data ideally from the same laboratory in which that system is employed. It is therefore important to test the system by exposing it to data that it has not seen in the past. This, in turn, will provide the laboratory with a more realistic assessment of the readiness of the system for casework. The new data should be comprised of samples that represent the variety of casework handled within the validating laboratory.This guidance presumes that probabilistic genotyping software systems like TrueAllele "are calibrated using historical data"; it does not require calibration.
ANSI/ASB Standard 018, 1st Ed. 2020, https://www.aafs.org/sites/default/files/media/documents/018_Std_e1.pdf.
Id.
When asked about calibration for TrueAllele, Dr. Perlin gave the following testimony:
Calibration is a different concept [than internal validation]. There's some software programs that don't have all the variables needed to derive the information from the data. There, there are extra variables, maybe thresholds or parameters or statistical values. And in order to provide them when a user runs the program, a calibration has to be done ahead of time to—so the user or the software can supply all those extra variables to the software so it can solve the problem. TrueAllele doesn't have any of those extra variables. It doesn't need calibration. It runs everything directly from the data while it's solving the problem. So it may take a day to solve the problem instead of five minutes, but it's learning all of those parameters directly from the evidence data instead of using a calibration that may have been done a year before on some other data.In effect, Dr. Perlin asserted that although certain probabilistic genotyping software programs like STRmix use "external information from a calibration study"—such as "stutter values, dropout values, [and] threshold values"—TrueAllele does not rely on that type of external information to generate its results, as it instead "analyzes the data directly." Dr. Perlin's testimony therefore illuminates certain operational and design distinctions between TrueAllele and other probabilistic genotyping software programs. At most, the testimony raises questions about whether Guideline 4.7 is specifically tailored for programs like TrueAllele; it is by no means "contradicted" by Paragraph 4.7.
Doc. 42 (Dec. 21, 2022, Hearing Tr. - Dr. Perlin Testimony) at 108:2-16.
Id. at 108:17-109:7.
Doc. 57 (Anderson's Supp. Br.) at 17.
More to the point, Dr. Perlin's testimony does not refute Cybergenetics' claim that "testing is done on a variety of data sets before new software is distributed." Consistent with the text of Guideline 4.7, the relevant portion of Annex A demands only that labs using probabilistic genotyping software validate the software before operation by testing it against new data. Cybergenetics represents that it performs this type of testing, and Anderson has presented no evidence to the contrary.
GX 7.0 (True Allele Standards Compliance) at 22.
Based on the evidence presented at the evidentiary hearing, the Government has established that TrueAllele complies with the relevant standards issued by the Scientific Working Group and ANSI/ASB. Accordingly, this factor weighs in favor of admissibility.
5. General Acceptance
Finally, the Government argues that "TrueAllele and other probabilistic genotyping systems are widely accepted in the scientific and legal community." As just discussed, the Government has established that TrueAllele complies with the relevant standards in the field of forensic science. Moreover, in the criminal context, the Government notes that TrueAllele is currently used by ten independent crime laboratories across the United States [Redacted]. It has also been used by defense attorneys and has helped exonerate multiple defendants. And outside the criminal arena, TrueAllele was even used to help identify human remains at Ground Zero following the 9/11 terrorist attacks.
Doc. 56 (Gov't Supp. Br.) at 7 (citing GX 9.0 (Nov. 24, 2022, Agencies Assisted by Cybergenetics and TrueAllele Technology); GX 22.0 (Dec. 2022 List of TrueAllele Cases)).
See Doc. 42 (Dec. 21, 2022, Hearing Tr. - Dr. Perlin Testimony) at 115:11-15.
[redacted].
GX 22.0 (Dec. 2022 List of TrueAllele Cases) at 5-6.
See Doc. 42 (Dec. 21, 2022, Hearing Tr. - Dr. Perlin Testimony) at 103:12-104:4.
In response, Anderson asserts that "among the community of computer scientists and software engineers, TrueAllele is not considered reliable software due to the lack of adherence to accepted practices in the field." Indeed, both of Anderson's experts are computer scientists, and they contend that probabilistic genotyping software programs like TrueAllele should be deemed unreliable because, among other things, they do not comply with the standards promulgated by the Institute of Electrical and Electronics Engineers ("IEEE").
Doc. 57 (Anderson Supp. Br.) at 26.
See Doc. 53 (Jan. 26, 2023, Hearing Tr. - N. Adams Testimony) at 38:11-43:17 (discussing the history of the IEEE standards and explaining why TrueAllele does not meet them); Doc. 53 (Jan. 26, 2023, Hearing Tr. - Dr. Matthews Testimony) at 107:15-112:18 (same).
But Anderson's position that the TrueAllele results should be excluded from evidence because the program does not comply with IEEE standards is at odds with the Sixth Circuit's ruling in Gissantaner. There, the Sixth Circuit found that the relevant standards for probabilistic genotyping software programs like TrueAllele are those promulgated by the FBI-sponsored Scientific Working Group. Indeed, the Sixth Circuit acknowledged that probabilistic genotyping software "remains controversial among a subset of the scientific community (computer scientists)," but concluded that the "existence of criticism . . . does not mean that [a probabilistic genotyping software program] has fallen short of 'general' acceptance."
990 F.3d at 466, accord Wakefield, 38 N.Y.3d at 375-76, 174 N.Y.S.3d 312, 195 N.E.3d 19 (describing the Scientific Working Group, ANSI, and NIST as "the relevant scientific community" when assessing "the fully continuous probabilistic genotyping approach used by TrueAllele").
Gissantaner, 990 F.3d at 469.
Given the forensic science community's broad acceptance of TrueAllele and other probabilistic genotyping software programs and the widespread use of such programs in the legal context, the Court finds that TrueAllele and the methods it employs are "generally accepted" by the relevant scientific community. This finding accords with the rulings by the Sixth Circuit in Gissantaner, the Pennsylvania Superior Court in Foley, and the New York Court of Appeals in Wakefield. Therefore, the final reliability factor considered here—as with the four previous factors discussed—weighs in favor of admissibility.
In re Paoli R.R. Yard PCB Litigation, 35 F.3d at 742 n.8.
Gissantaner, 990 F.3d at 466 (finding that "STRmix satisfies [the] consideration" of general acceptance, noting that "numerous courts have admitted STRmix over challenges to its general acceptance in the relevant scientific community"); Foley, 38 A.3d at 888-89 (finding "no legitimate dispute regarding the reliability of Dr. Perlin's testimony," emphasizing that TrueAllele technology is used "by New York State for all of their data banking," the "Allegheny County Crime Lab," the "United Kingdom's Forensic Science Service" to "analyze crime scene evidence," and the "World Trade Center" to "reanalyze all of the data" on unidentified victim remains); Wakefield, 38 N.Y.3d at 375-76, 174 N.Y.S.3d 312, 195 N.E.3d 19 ("As to general acceptance of the continuous probabilistic genotyping system, the testimony of the People's witnesses established that probabilistic genotyping methods have been recognized by the relevant scientific community.").
IV. CONCLUSION
Weighing the evidence presented against the relevant Daubert reliability factors, the Court finds TrueAllele's DNA identification methodology and the match statistic reported in this case sufficiently reliable to warrant admission. Anderson will be permitted to contest this evidence at trial through cross examination and the testimony of rebuttal witnesses; he cannot, however, keep the evidence from the jury. The Daubert motion is denied.
An appropriate Order follows.