Opinion
2:21-cv-00750-RSL
05-19-2023
KAELI GARNER, et al., Plaintiffs, v. AMAZON.COM, INC., et al., Defendant.
ORDER REGARDING PLAINTIFFS' MOTION TO PROHIBIT USE OF TECHNOLOGY-ASSISTED REVIEW PROCEDURES
ROBERT S. LASNIK UNITED STATES DISTRICT JUDGE
This matter comes before the Court on “Plaintiffs' Motion to Compel Defendants to Comply with the Court's Prior Discovery Orders.” Dkt. # 143. In October 2022, the Court granted plaintiffs' motion to compel searches of the electronically-stored information (“ESI”) of 36 Amazon employees using 38 search strings, as proposed by plaintiffs. Dkt. # 136. Using the approved search terms, defendants identified approximately 2 million potentially responsive documents.Within two weeks of the Court's order, defendants notified plaintiffs that Amazon intended to “use technology-assisted review [“TAR”] tools to prepare its production,” promising that they would “discuss these tools with Plaintiffs before beginning review.” Dkt. # 144-4 at 2.Two days later, plaintiffs objected, asserting that “[a]pplying TAR at this stage - after the parties have agreed to search terms as a basis for culling documents, and after the Court issued an order on search terms and custodians, is improper and impermissible.” Dkt. # 144-2 at 3. Plaintiffs argued that it was too late to alter the protocol for the production of ESI in this case. Defendants disagreed, insisting that the large number of retrieved documents necessitated the use of TAR to identify responsive, non-privileged documents. Dkt. # 144-1. Plaintiff again objected to any effort “to use TAR to further review/filter the documents retrieved by the Court-ordered search terms.” Dkt. # 144-3 at 3.
In the context of the motion to compel, defendants had estimated that the search terms would return between 3.1 and 4.4 million documents.
TAR is also known as computer-assisted review or predictive coding. Reviewers typically code a set of documents as responsive or unresponsive to “train” the TAR software, which then uses the coded documents to generate algorithms for reviewing all other documents in the selected universe for responsiveness. Progressive Cas. Ins. Co. v. Delaney, No. 2:11-cv-00678-LRH, 2014 WL 3563467, at *10 (D. Nev. July 18, 2014); https://legal.thomsonreuters.com/en/insights/articles/myths-and-facts-about-technology-assisted-review.
At approximately the same time, the parties negotiated an extension of the discovery deadline to August 16, 2023.
Having reviewed the memoranda, declarations, and exhibits submitted by the parties and having heard the arguments of counsel, the Court finds as follows:
Plaintiffs make clear in reply that they are not trying to prevent Amazon from conducting a responsiveness and privilege review of the universe of documents the search terms retrieved. Rather, they seek to prevent the unilateral application of TAR, apparently preferring that Amazon conduct its review by hand. In this district, it is presumed that the use of technology-assisted review is a reasonable option for locating or filtering ESI. https://view.officeapps.live.com/op/view.aspx?src=https%3A%2F%2Fwww.wawd.uscourt s.gov%2Fsites%2Fwawd%2Ffiles%2FModelESIAgreementCLEAN2i1223idocx&wdOri gin=BROWSELINK at 3. Litigants are instructed to confer and attempt to reach agreement on “appropriate computer- or technology-aided methodologies[] before any such effort is undertaken.” Id. at 3-4. That requirement is incorporated into the April 2022 ESI Agreement and Order entered in this matter, which also recognized “that searching for relevant ESI is an iterative process.” Dkt. # 89 at 2-4. Case law discussing the use of computer-assisted methodologies to review documents for production emphasizes the need for cooperation and transparency. In Youngevity Int'l, Corp. v. Smith, No. No. 16-cv-00704-BTM (JLB), 2019 WL 1542300, at *12 (S.D. Cal. Apr. 9, 2019), for example, the court noted that:
Technology-assisted review of ESI does require an ‘unprecedented degree of transparency and cooperation among counsel' in the review and production of ESI responsive to discovery requests. Progressive Cas. Ins. Co., 2014 WL 3563467, at *10. In this regard, courts typically ‘have required the producing party to provide the requesting party with full disclosure about the technology used, the process, and the methodology, including the documents used to ‘train' the computer.” Id.
While defendants' discovery conduct has not been beyond criticism, the nub of this particular discovery dispute is that when Amazon raised the issue of using TAR, plaintiffs refused to discuss it. This was not a valid option given the circumstances. The parties had already agreed that an extension of the discovery deadline was needed: there was therefore time to exchange proposals regarding the use of predictive coding to improve the accuracy, speed, and efficiency of the document review Amazon was then undertaking. Courts generally agree that the producing party is best situated to evaluate the various options for reviewing and producing its own ESI and places the burden on that party to make an initial, detailed proposal about the technology and methodologies it intends to use. Id. at 11; Hyles v. New York City, No. 10-cv-3119(AT)(AJP), 2016 WL 4077114, at *2 (S.D.N.Y. Aug. 1, 2016); Progressive Cas. Ins. Co., 2014 WL 3563467, at *10. The requesting party then has an opportunity to consult with its own ESI discovery experts so that it can intelligently respond and the parties can move toward an agreed protocol for the use of TAR. By refusing to discuss the use of TAR at all, plaintiffs improperly short-circuited this process.
The Court acknowledges the similarities between the facts of this case and those of Progressive Cas. Ins. Co. v. Delaney. In both cases, search terms had been used to identify potentially responsive ESI before the producing party indicated that TAR should be used to further refine the production. In contrast with this case, however, the stipulated ESI protocol in Progressive gave the producing party two options once the universe of potentially responsive documents was identified: to produce all of the “hit” documents or to manually review the retrieved documents and produce only those that were, in fact, responsive. 2014 WL 3563467, at *11. Progressive chose a third, unauthorized, path by conducting a TAR review. It also withheld documents that had already been found to be responsive through an aborted manual review process and it refused to provide details regarding its TAR proposal to the requesting party. In those circumstances, the court declined to adopt Progressive's predictive coding proposal because it would only result in more discovery disputes and delay. The court instead ordered the production of all non-privileged documents retrieved using the search terms.
Plaintiffs suggest that the use of TAR on a pool of documents already retrieved through search terms is somehow improper because it “will only reduce the document pool further and will certainly not reveal documents that the application of search terms has precluded.” Dkt. # 143 at 10 (quoting In re Allergan Biocell Textured Breast Implant Prod. Liab. Litig., No. 2:19-md-2921(BRM)(ESK), 2022 WL 16630821, at *3 (D.N.J. Oct. 25, 2022)). But, as discussed above, the Model ESI Agreement in this district clearly contemplates using TAR to filter, not just locate, documents, and the ESI Agreement entered in April 2022 simply directs the parties “to confer to attempt to reach agreement on . . . appropriate computer- or technology-aided methodologies[] before any such effort is undertaken.” The Court finds that the use of search terms is not, standing alone, a bar to using technology to further refine the production.
It is worth repeating at this juncture that plaintiffs do not dispute that Amazon has the right to conduct a responsiveness and privilege review of the documents retrieved through application of the search terms to the custodial records, they just want the review done by hand rather than through predictive coding because (a) November 2022 was “too late” to raise the issue and (b) the TAR process has identified only 2,564 responsive documents out of an initial universe of 2,036,172. For the reasons discussed above, the Court finds that raising the possibly of using TAR once the search terms were run and the universe of hits was known was not unreasonable and did not excuse plaintiffs from engaging in a discussion regarding the methodology and protocols to be used. Plaintiffs' timeliness objection is overruled.
Plaintiffs' concern regarding the incredibly low production rate (less than 0.13% of the documents returned using the search terms) may be valid, but it does not appear to be related to Amazon's use of TAR. Amazon's initial proposal regarding how their TAR would work utilized an unacceptably low recall target of 75% But as the process unfolded, Amazon continued its review after that rate was achieved, ultimately conducting a human review of over 1.8 million documents. When a sample of the remaining pool of unreviewed documents was analyzed, the human reviewers found no responsive documents. Thus, humans reviewed over 89% of the documents identified using the agreed-upon search terms and found that only 2,564 of them were actually relevant, responsive, and non-privileged. The 224,924 unreviewed documents are, according to the TAR methodology, the least likely to be responsive and, in fact, a human review of 1,527 of them revealed no responsive materials. Because humans reviewed the vast majority of the universe of documents and the statistical estimate of responsive documents remaining in the unreviewed documents is 0%, the estimated recall rate approaches 100% There is no reason to suspect that the low percentage of production is, as plaintiffs argue, the result of Amazon's use of TAR versus human review.
Mr. Johnson states that recall requirements between 70-80% are standard because “[a]ttempting to achieve higher recall rates, for example 85%, can result in dramatically higher costs for the review, and likely eliminate any potential costs savings and efficiencies that parties would hope to realize using TAR.” Dkt. # 149 at ¶ 17. The goal of discovery is the production of all responsive, relevant, non-privileged documents. If TAR is able to assist in reaching that goal while saving money, it is a useful tool. But if TAR is cost-effective only if 25% of responsive, relevant, nonprivileged documents remain hidden, it may costs less, but it is no longer effective.
For all of the foregoing reasons, plaintiffs' motion to compel is DENIED. Defendants shall, if they have not already done so, immediately produce the 2,564 documents tagged as responsive through the human review process.