Opinion
18 Civ. 02468 (VB)(JCM)
09-28-2021
CARLOS FLORES, LAWRENCE BARTLEY, DEMETRIUS BENNETT, L'MANI DELFMA, EDGARDO LEBRON, ANTONIO ROMAN, DONTAE QUINONES and SHAROD LOGAN, on behalf of themselves and all others similarly situated, Plaintiffs, v. TINA M. STANFORD, as Chairwoman of the New York State Board of Parole; WALTER W. SMITH, As Commissioner of the New York State Board of Parole; SALLY THOMPSON, as Commissioner of the New York State Board of Parole; JOSEPH P. CRANGLE, as Commissioner of the New York State Board of Parole; ELLEN E. ALEXANDER, as Commissioner of the New York State Board of Parole; MARC COPPOLA, as Commissioner of the New York State Board of Parole; EDWARD SHAKEY, as Commissioner of the New York State Board of Parole; TANA AGOSTINI, as Commissioner of the New York State Board of Parole; CHARLES DAVIS, as Commissioner of the New York State Board of Parole; CAROL SHAPIRO, As Commissioner of the New York State Board of Parole; ERIK BERLINER, as Commissioner of the New York State Board of Parole; OTIS CRUSE, as Commissioner of the New York State Board of Parole; TYCEE DRAKE, as Commissioner of the New York State Board of Parole; and CARYNE DEMOSTHENES, as Commissioner of the New York State Board of Parole, Defendants.
OPINION & ORDER
JUDITH C. McCARTHY United States Magistrate Judge
Before the Court is an application from non-party Northpointe, Inc. ("Northpointe") seeking an order preventing disclosure of certain materials produced by Northpointe to Plaintiffs Carols Flores, Lawrence Bartley, Demetrius Bennett, L'Mani Delima, Edgardo Lebron, Antonio Roman, Dontae Quinones and Sharod Logan's (collectively, "Plaintiffs") expert, Dr. Cynthia Rudin ("Dr. Rudin"). (Docket No. 211). Plaintiffs opposed the request. (Docket No. 207). The Court heard oral argument on September 16, 2021. (Docket No. 219). For the reasons that follow, Northpointe's request is denied.
I. BACKGROUND
This dispute stems from the Court's previous order, dated February 12, 2021 (the "February 12, 2021 Order"), compelling Northpointe to produce to Plaintiffs certain proprietary information pertaining to Northpointe's Correctional Offender Management Profiling for Alternative Sanctions ("COMPAS") tool. (See Docket Nos. 181, 186, 209-3; February 12, 2021 Minute Entry).
As alleged in the Second Amended Complaint (the "SAC"), COMPAS is a commercial product sold by Northpointe containing "secret algorithms" that is used as a risk assessment tool by defendants Tina Stanford, as Chairwoman of the New York State Board of Parole (the "BOP"), and the individual BOP Commissioners (the "Commissioners") (collectively, the "Defendants"), when deciding whether to grant offenders sentenced to life in prison discretionary parole. (Docket No. 110 ¶¶ 10, 209, 211, 213, 215). The SAC further alleges that Defendants rely on COMPAS "without knowing how or whether COMPAS considers the diminished culpability of juveniles and the hallmark features of youth" because Northpointe "considers COMPAS a proprietary instrument and a trade secret." (Id. ¶¶ 213, 215). Moreover, COMPAS "sometimes treats youth as an aggravating factor" in predicting risk of felony violence. (Id. ¶¶ 211-12). For example, Plaintiff L'Mani Delima ("Mr. Delima"), is twenty-six years old, has been denied parole three times, and has a high "Risk of Felony Violence" COMPAS score even though he was convicted when he was only thirteen. (See Id. ¶¶ 47, 49-53, 212). According to Plaintiffs, Defendants' reliance on this algorithm of which they lack knowledge or understanding deprives juvenile offenders who received life sentences of the individualized parole assessments to which they are entitled under the Eighth and Fourteenth Amendments. (See Id. ¶¶ 2-10, 136-46, 222-35).
The Court's February 12, 2021 Order required Northpointe to produce to Plaintiffs (1) the normative dataset used to create and normalize COMPAS (the "Norm Group Data"); and (2) the regression models for two COMPAS "scales": (a) the General Recidivism Risk Scale, and (b) the Violent Recidivism Risk Scale (the "Regression Models" and collectively, the "Compelled Materials"). (See February 12, 2021 Minute Entry; Docket No. 209-3; see also Docket Nos. 181, 186). The Norm Group Data is a repository of offender information from several jurisdictions that Northpointe uses to generate the Regression Models, which are sets of inputs used to predict the likelihood of new offenses and new violent offenses after an offender's COMPAS assessment date. (Docket Nos. 181 at 2; 209-3 at 7:9-8:9; 219 at 20:6-12). Northpointe also uses the Norm Group Data to translate recidivism risk scores into data presented to individual Defendants before parole hearings. (Docket No. 209-3 at 7:22-8:20; see also Docket No. 181 at 2). To address Northpointe's concerns that the Compelled Materials constitute trade secrets and/or other proprietary information, (see Docket No. 186), the Court ordered that the Compelled Materials be produced pursuant to a protective order and be designated "attorney's and expert's eyes only." (See February 12, 2021 Minute Entry; Docket No. 209-3 at 37:5-15).
Northpointe and Plaintiffs entered into a Second Supplemental Stipulation of Confidentiality and Proposed Protective Order regarding the Compelled Materials on February 26, 2021 (the "Second Protective Order"). (Docket No. 198). The Second Protective Order requires that the Compelled Materials be designated attorneys' eyes only and "Highly
Confidential" as described in the parties' first Stipulation of Confidentiality and Protective Order (the "First Protective Order") and the supplement thereto concerning Northpointe (the "Supplemental Protective Order") (collectively, the "Protective Orders"). (Docket No. 198 ¶¶ 2-3; see also Docket Nos. 151, 154). Under these earlier Protective Orders, materials designated as such cannot be "provide[d, ] show[n] [or] . . . discuss[ed] . . . with Plaintiffs" without Northpointe's written consent or a court order. (Docket No. 154 ¶ 4). They "shall not be released or disclosed in any manner to any other person," with some limited exceptions, and "shall be used by the receiving party solely for the purposes of [the instant] action and solely to the extent necessary for [its] litigation . . ., including any appeals thereof." (Docket No. 151 ¶¶ 6, 21). At the conclusion of this action, all such materials and any copies thereof must either be destroyed or returned to adverse counsel. (Id. ¶ 22). The Second Protective Order further provides that the Compelled Materials "may not be shared with, shown to, or discussed with a party or a party's representative;" "can be reviewed by" only eight specific attorneys for Plaintiffs and Defendants; and cannot be disclosed to any expert unless Northpointe provides "express written consent," or any objections to such disclosure are overruled. (Docket No. 198 ¶ 3). Furthermore, neither the Compelled Materials nor their contents can be filed publicly. (Id. ¶¶ 4-5).
Notwithstanding these protections and Plaintiffs' efforts to meet and confer, (see Docket No. 207 at 3), Northpointe will not consent to allow Dr. Rudin to access the Compelled Materials. (See generally Docket No. 211). According to Northpointe, these documents contain Northpointe's "most proprietary information" and "permitting Dr. Rudin access to [it] . . . would threaten [Northpointe's] very existence." (Id. at 1). This is because Dr. Rudin, a professor of computer science, electrical and computer engineering, and statistical science at Duke University, is an outspoken critic of Northpointe and the COMPAS tool. (See Docket No. 211 at 1-2; see also Docket Nos. 207 at 3; 211-1). Indeed, Dr. Rudin has published several articles and given lectures questioning the government's reliance on COMPAS to evaluate recidivism risk, when neither the public nor the government have access to the data or formulas on which COMPAS is based. According to Dr. Rudin, this lack of transparency is particularly unfair when it is possible to use publicly available algorithms to predict recidivism risk with the same accuracy as COMPAS. In making this argument, Dr. Rudin has indicated that she and her coauthors have attempted, without success, to uncover the COMPAS algorithm(s). See, e.g., NC State Research, DSI RED Talk with Cynthia Rudin Secrecy, Criminal Justice and Variable Importance, YouTube (Nov. 6, 2019) https://www.voutube.com/watch?v=QWcH1U-5Ftw, at 27:03 to 27:45 ("[W]e've been working on this analysis that tries to uncover some of COMPAS' secret sauce"; "we've been trying to figure out what's actually inside COMPAS"; "[I]f COMPAS is gonna sell people this model and say that they're better than my interpretable machine learning models I want to know what the heck's in there, right?"; "Our goal is to identify bounds for how much COMPAS relies on different covariate subsets . . . ."). Dr. Rudin has also opined that Northpointe should be "put. . . out of business" and "there should not be a business model" for the COMPAS tool. Nat'l Academy of Sciences, supra n.l at 1:48-1:52.
https://staticl.squarespace.eom/static/5852eccbebbdlaa62603da84/t/5el387el64e84e62el2fc52e/1578338273912/H arvard+Data+Science+on+Explainability+2019.pdf; Cynthia Rudin et al., The age of secrecy and unfairness in recidivism prediction, Harv. Data Sci. Rev., 2(1), Nov. 2018, https://arxiv.org/abs/1811.00731; Nat'l Academy of Sciences, Cynthia Rudin - Interpretable ML for Recidivism Prediction - The Frontiers of Machine Learning, YouTube (Feb. 14, 2017), https://www.youtube.com/watch?v=MjxcwKN2dXs; (see also
Docket No. 211-1 at 2-3).
See, e.g., Cynthia Rudin et al., Broader Issues Surrounding Model Transparency in Criminal Justice Risk Scoring, Harv. Data Sci. Rev., 2(1), Mar. 31, 2020, https://doi.org/10.1162/99608f92.038c43fe; Aaron Fisher et al., All Models are Wrong, but Many are Useful: Learning a Variable's Importance by Studying an Entire Class of Prediction Models Simultaneously, Journal of Machine Learning Res., 20, Dec. 2019, https://imlr.org/papers/volume20/18-760/18-760.pdf; Cynthia Rudin et al., Why Are We Using Black Box Models in Al When We Don 'tNeed To? A Lesson From An Explainable Al Competition,Harv. Data Sci. Rev., Nov. 1, 2019, 1.2,
Apart from the above projects, Dr. Rudin focuses on the socially responsible use of machine learning to solve real-world problems in a variety of contexts, such as healthcare, energy use and criminology. (See Docket Nos. 207 at 2; 209-1 at 2). For example, she has used her expertise to help Con Edison establish an underground electrical distribution network in New York City; create optimal scoring systems to predict seizures in ICU patients; and develop a code for detecting crime in cities, which was adapted by the New York City Police Department ("NYPD") in 2016. (Docket No. 209-1 at 2). Dr. Rudin also collaborated with the Cambridge Police Department related to the clustering of break-ins from 2012 to 2016. (Docket No. 217 ¶ 10). Furthermore, she has participated in the Department of Justice's (the "DOJ") Bureau of Justice Assistance Criminal Justice Technology Forecasting Group, and served on the faculty advisory board of the Department of Defense's (the "DOD") Advanced Research Projects Agency. (Docket Nos. 207 at 2; 209-1 at 18).
Dr. Rudin submitted a declaration attesting that she has never been affiliated with a Northpointe competitor, and does not have plans to consult or work for one in the future. (Docket No. 217 ¶ 14). Her only current work in the criminal justice field from which she receives any benefit is membership of the National Academies of Science Committee on Law and Justice (the "NASCLJ"), which reimburses her for travel to and from Washington, D.C. for meetings.(Docket No. 217 ¶ 5). Dr. Rudin has not received grants or any other benefit for any of her publications regarding criminal justice issues, with the exception of two academic articles. (Id. ¶¶ 11-12). Dr. Rudin's current and future consulting engagements for which she expects compensation relate to antitrust and online advertising, healthcare analytics, and the use of artificial intelligence in loan decisions. (Id. ¶¶ 7, 13).
The NASCLJ "is a committee of national experts in the fields of criminal justice, criminology, public policy, law, sociology, economics and behavioral sciences." (Docket No. 217 If 5); see also About the Committee on Law and Justice, National Academies of Sciences, Engineering, Medicine, https://www.nationalacademies.org/claj/about (last visited September 16, 2021). At oral argument, the Court inquired whether Dr. Rudin's work for the NASCLJ entails engaging with lobbyists, government officials or Northpointe competitors. (Docket No. 219 at 23:16-22). Plaintiffs asserted that to their knowledge, the NASCLJ is a nonprofit organization with no connections to any lobbyist, government or government officials. (Id. at 23:23-25:14). Further, Dr. Rudin's committee on the NASCLJ is exclusively made up of academics, a "director of a justice center," and one appellate judge. (Id. at 25:21-25).
Dr. Rudin further asserts that she has already agreed to be bound by the First and Supplemental Protective Orders, and is prepared to be bound by the Second Protective Order. (Id. ¶ 15). She further agrees that by being bound by the Protective Orders, she would be subject to the Court's jurisdiction for the purpose of enforcing or otherwise providing relief relating thereto. (Id.). If permitted to review the Compelled Materials, Dr. Rudin attests that her obligation to maintain their confidentiality would not interfere with her current or anticipated work or any ethical obligations, nor would any such work require disclosure of the information they contain. (Id. ¶ 16). In order to prevent the possibility of inadvertent disclosure, Dr. Rudin would store the Compelled Materials only on her password-protected computer; refrain from copying the files; and permanently delete them from her computer at the conclusion of the litigation. (Id. ¶ 17). At oral argument, Plaintiffs' counsel represented that they would be happy to agree to further restrictions regarding the sharing or reproducing of the information in these materials. (Docket No. 219 at 26:7-9).
II. LEGAL STANDARD
Because Northpointe asks this Court to prohibit Plaintiffs from sharing the Compelled Materials with Dr. Rudin, (Docket No. 211 at 1), the Court treats its request as a motion for a protective order. Rule 26(c) of the Federal Rule of Civil Procedure provides that a "court may, for good cause, issue an order to protect a party or person [from whom discovery is sought] from annoyance, embarrassment, oppression, or undue burden or expense." Fed.R.Civ.P. 26(c)(1). Such protection may include "requiring that a trade secret or other confidential research, development, or commercial information not be revealed or be revealed only in a specified way . . . ." Fed.R.Civ.P. 26(c)(1)(G). The district court has "broad discretion ... to decide when a protective order is appropriate and what degree of protection is required." Seattle Times Co. v. Rhinehart, 467 U.S. 20, 36 (1984); see also Dove v. Ail. Cap. Corp., 963 F.2d 15, 20 (2d Cir. 1992).
"The party seeking a protective order bears the burden of establishing that good cause for the order exists." Duling v. Gristede 's Operating Corp., 266 F.R.D. 66, 71 (S.D.N.Y. 2010). "Good cause is established by 'demonstrating a particular need for protection.'" Id. (quoting Cipollone v. Liggett Group, Inc., 785 F.2d 1108, 1121 (3d Cir. 1986)). "Ordinarily, good cause exists 'when a party shows that disclosure will result in a clearly defined, specific and serious injury.'" In re Terrorist Attacks on Sept. 11, 2001, 454 F.Supp.2d 220, 222 (S.D.N.Y.2006) (quoting Shingara v. Skiles, 420 F.3d 301, 306 (3d Cir. 2005)). "Although the burden is on the movant to establish good cause for the entry of a protective order, the court ultimately weighs the interests of both sides in fashioning an order." Duling, 266 F.R.D. at 71.
"[T]here is no absolute privilege [against disclosure] for trade secrets and similar confidential information." Federal Open Mkt. Comm. of Fed. Reserve Sys. v. Merrill, 443 U.S. 340, 362 (1979) (quoting 8 Charles Alan Wright & Arthur R. Miller, Federal Practice and Procedure § 2043 (1st ed. 1970)). "If a court finds that commercially sensitive materials are relevant to a party's claim, the court balances that party's need to have its chosen expert review the materials against the producing party's interest in protecting the materials from a potential competitor." Grand River Enters. Six Nations, Ltd. v. King, No. 02 Civ. 5068(JFK), 2009 WL 222160, at *5 (S.D.N.Y. Jan. 30, 2009); see also Uniroyal Chem. Co. Inc. v. Syngenta Crop Prot, 224 F.R.D. 53, 57 (D. Conn. 2004). "Relevant considerations in striking this balance include: 1) whether the person receiving the confidential information is involved in competitive decision making or scientific research relating to the subject matter of the [materials], 2) the risk of inadvertent disclosure of proprietary information, 3) the hardship imposed by the restriction, 4) the timing of the remedy and, 5) the scope of the remedy." See Uniroyal Chem. Corp., 224 F.R.D. at 57. The court must also consider "the specific expertise of th[e] [chosen] consultant and whether other consultants possess similar expertise." Bridgeport Music Inc. v. UMG Recordings, Inc., No. 05 Civ. 6430(VM)(JCF), 2008 WL 577284, at *1 (S.D.N.Y. Feb. 27, 2008) (quoting BASF Corp. v. United States, 321 F.Supp.2d 1373, 1378 (Ct. Int'l Trade 2004)).
III. DISCUSSION
Northpointe argues that to prevent competitive injury to Northpointe, the Court should (a) prohibit disclosure of the Compelled Materials to Dr. Rudin; or (b) in the alternative, delay any ruling regarding such disclosure until after certification of the putative class. (Docket No. 211 at 1-4). Plaintiffs respond that disclosure of the Compelled Materials to Dr. Rudin is necessary to the effective prosecution of their case, and will not create any risk of competitive injury because Dr. Rudin has never worked for a Northpointe competitor and will be bound by the Protective Orders. (Docket No. 207 at 1-3). They further assert that Northpointe's alternative request would unnecessarily delay the litigation and is inappropriate because the Court already declined to postpone disclosure of the Compelled Materials in its February 12, 2021 Order. (Id. at 3). The Court agrees that Dr. Rudin's review of the Compelled Materials is paramount to Plaintiffs' prosecution of this case. In addition, disclosure to Dr. Rudin poses minimal risk of competitive injury in light of the Protective Orders in place and the further restrictions the Court imposes as set forth below. Because these materials are relevant to this action whether a class is certified or not, it declines to postpone their disclosure to Dr. Rudin any longer.
A. Relevance
Although the Court addressed the relevance of the Compelled Materials in the February 12, 2021 hearing, (see Docket No. 209-3 at 41:18-22), for completeness, the Court will briefly address this issue as it pertains to Dr. Rudin's role in this case. See Grand River Enters. Six Nations, 2009 WL 222160, at *5. Northpointe argues that "[t]he design and function of COMPAS are irrelevant to class certification" and are only minimally relevant to this lawsuit as a whole because all but one of the named Plaintiffs, Mr. Delima, had favorable COMPAS risk scores when they were denied parole. (Docket No. 211 at 3; see also Docket Nos. 110 ¶¶ 20, 29, 42, 52, 64, 72, 85, 97; 219 at 43:19-45:3). Plaintiffs assert that the Compelled Materials are highly relevant to class certification as well as the merits, because they will demonstrate how COMPAS factors youth into recidivism risk scores both across the proposed class and for the individual Plaintiffs. (See Docket No. 219 at 20:16-21:14, 45:24-46:19; see also Docket No. 181 at 2).
Ordinarily, "[i]n order to obtain pre-certification discovery concerning class issues, the plaintiff must show that 'such discovery would be relevant to [her] future motion for class certification.'" See Van Elzen v. Revimedia, Inc., 17-CV-2131, 2018 U.S. Dist. LEXIS 132007, at *11 (S.D.N.Y.Aug. 3, 2018) (quoting Bais Yaakov of Spring Valley v. Houghton Mifflin Harcourt Publishers, Inc., 36 F.Supp.3d 417, 421 (S.D.N.Y. 2014)). "Any class discovery must also be proportional to the needs of the case and 'bear some relationship to the claims of the named Plaintiffs].'" Dupres v. Houslanger & Assocs., PLLC, 19-CV-6691-RPK-SJB, 2021 WL 2373737, at *2 (E.D.N.Y. June 9, 2021) (quoting Chow v. SentosaCare, LLC, 19-CV-3451-FB-SJB, 2020 WL 559704, at *3 (E.D.N.Y. Jan. 23, 2020)) (alteration in original).
Class certification is governed by Federal Rule of Civil Procedure 23 ("Rule 23"). Under Rule 23(a), the party seeking certification must demonstrate that "(1) the class is so numerous that joinder of all members is impracticable; (2) there are questions of law or fact common to the class; (3) the claims or defenses of the representative parties are typical of the claims or defenses of the class; and (4) the representative parties will fairly and adequately protect the interests of the class." See Fed. R. Civ. P. 23(a). In addition to these requirements, the proposed class must satisfy at least one of the three requirements listed in Rule 23(b). See Fed. R. Civ. P. 23(b)(1)-(3).
However, it is well-established within this Circuit that "[discovery relating to class issues may overlap substantially with merits discovery." Bodner v. Paribas, 202 F.R.D. 370, 373 (E.D.N.Y. 2000) (quoting Manual for Complex Litigation (Third) § 30.12 (1995)) (collecting cases); see also Dupres, 2021 WL 2373737, at *2-3; Thompson v. Glob. Contact Servs., LLC, 20-CV-651, 2021 WL 1103029, at *2 (E.D.N.Y. Feb. 16, 2021). This is because "[a] key question in class certification may be the similarity or dissimilarity between the claims of the representative parties and those of the class members-an inquiry that may require discovery on the merits and development of basic issues." See Bodner, 202 F.R.D. at 373 (quoting Manual for Complex Litigation (Third) § 30.12). For this reason, "courts are reluctant to bifurcate class-related discovery from discovery on the merits." Chen-Oster v. Goldman, Sachs & Co., 285 F.R.D. 294, 299 (S.D.N.Y. 2012) (citing Wal-Mart Stores, Inc. v. Dukes, 564 U.S. 338 (2011)); see also Hines v. Overstock.com, Inc., No. 09-CV-991 (SJ), 2010 WL 2775921, at *1 (E.D.N.Y. July 13, 2010) ("[W]here discovery relating to class issues overlaps substantially with merits discovery, bifurcation will result in duplication of efforts and needless line-drawing disputes."). Moreover, "[t]he fact that discovery may identify potential class members does not render otherwise proper discovery premature because it precedes class certification." Ulit4less, Inc. v. FedEx Corp., 11 Civ. 1713(CS)(PED), 2014 WL 12779557, at *2 (S.D.N.Y. Oct. 31, 2014); see also Charvat v. Plymouth Rock Energy, LLC, No. 15-CV-4106 (JMA)(SJL), 2016 WL 207677, at *2 (E.D.N.Y.Jan. 12, 2016) (declining to bifurcate discovery that "appear[ed] relevant to both the class and individual claims").
Here, irrespective of whether the Compelled Materials are specifically relevant to class certification, they are relevant to the merits of Plaintiffs' constitutional claims. Relevance is "construed broadly to encompass any matter that bears on, or that reasonably could lead to other matter that could bear on' any party's claim or defense." Doe v. Sarah Lawrence Coll., 19 Civ. 10028 (PMH)(JCM), 2021 WL 197132, at *3 (S.D.N.Y. Jan. 20, 2021) (quoting State Farm Mut. Auto. Ins. Co. v. Fayda, 14 Civ. 9792 (WHP) (JCF), 2015 WL 7871037, at *2 (S.D.N.Y. Dec. 3, 2015), aff'd, 2016 WL 4530890 (S.D.N.Y. Mar. 24, 2016)). When determining whether to release an offender, the Parole Board is expressly required to consider, among other things, the offender's "risk and needs scores" generated by COMPAS. See NY. Comp. Codes R. & Regs, tit. 9, § 8002.2(a); (Docket No. 209-3 at 8:18-12:14). The SAC alleges that when generating risk scores, COMPAS "sometimes" impermissibly treats youth as an aggravating factor and does not account for juvenile rehabilitative capacities. (Docket No. 110 ¶¶ 1-10, 210-12). The SAC further alleges that Defendants' reliance on this discriminatory tool for parole decisions denies juvenile offenders sentenced to life in prison with the possibility of parole - like the named Plaintiffs - a meaningful opportunity for release based on demonstrated maturity and rehabilitation, in violation of the Eighth and Fourteenth Amendments. (Id. ¶¶ 1-10, 211-12, 224- 35). Therefore, a central component of Plaintiffs' argument is that COMPAS's algorithm weighs youth unfavorably when generating risk scores.
If the Board's decision denying release "departs" from these scores, it must "specify any [such] scale . . . from which it departed and provide an individualized reason for such departure." See NY. Comp. Codes R. & Regs. tit. 9, § 8002.2(a) (2021).
As Plaintiffs explain, the Supreme Court has held that a sentence to life without parole for a juvenile offender violates the Eighth Amendment unless the juvenile is the "rare . . . offender whose crime reflects irreparable corruption." (Docket No. 110 ¶¶[ 3-5) (quoting Montgomery v. Louisiana, 136 S.Ct. 718, 734 (2016)). Further, juvenile offenders sentenced to life must be afforded a "meaningful and realistic opportunity for release based upon demonstrated maturity and rehabilitation." Miller v. Alabama, 567 U.S. 460, 479-80 (2012)) (quoting Graham v. Florida, 560 U.S. 48, 75 (2010)). According to Plaintiffs, COMPAS violates that requirement by counting juvenile offenders' youth against them when assessing risk scores. (Docket No. 110 ¶¶ 224-28). In addition, "[a] fundamental requirement of due process [under the Fourteenth Amendment] is 'the opportunity to be heard' ... at a meaningful time and in a meaningful manner." See Armstrong v. Manzo, 380 U.S. 545, 552 (1965) (quoting Grannis v. Ordean, 234 U.S. 385, 394 (1914)). Thus, Plaintiffs argue, Defendants' use of COMPAS violates the Due Process Clause because Defendants do not know how the tool treats age, and "the scores are based on statistical facts about membership in a supposed statistical group" that may be inaccurate. (See Docket No. 110 ¶¶ 208-16, 234-35).
Northpointe's other productions confirm that age is one datapoint on which COMPAS's risk scores are based, but do not reveal how COMPAS weighs youth when measuring risk. (Docket Nos. 209-3 at 14:8-15:22; 219 at 21:7-9). As a result, more information is necessary to support Plaintiffs' claim that COMPAS's risk models violate young offenders' constitutional rights. (Docket Nos. 209-3 at 14:9-16:20; 219 at 20:16-21:14). According to Plaintiffs, the Regression Models would help support their claim by indicating the combination of inputs used to calculate COMPAS scores, and thus, the relationship between youth and a high score. (See Docket No. 209-3 at 7:1-22; see also Docket No. 181 at 2). Similarly, the Norm Group Data would reveal whether the raw data used to generate risk scores is statistically appropriate for juvenile offenders such as Plaintiffs. (See Docket Nos. 181 at 2; 219 at 20:22-21:14). Although certain of Plaintiffs' attorneys have access to the Compelled Materials, (Docket No. 198 ¶ 3), they cannot understand these highly technical documents without input from Dr. Rudin. (Docket Nos. 207 at 3; 209-3 at 26:3-6; 219 at 20:16-21). In other words, Dr. Rudin's analysis of these documents may help Plaintiffs substantiate their allegations that COMPAS punishes juvenile offenders for their youth, such that Defendants' reliance on this tool is constitutionally problematic.
The Court is not persuaded that the Compelled Materials are irrelevant simply because only Mr. Delima received a high COMPAS score for the risk scales at issue. (See Docket No. 211 at 3). Even if the Court does not ultimately certify the class, the weight COMPAS affords to youth in determining recidivism risk will be highly relevant to the underlying constitutional claims. See generally Ebbert v. Nassau Cty., No. CV 05-5445(FB)(AKT), 2007 WL 674725, at *8-10 (E.D.N.Y. Mar. 5, 2007); (Docket No. 110 ¶ 212). Even though only Mr. Delima received a high COMPAS score, all of the named Plaintiffs maintain that Defendants' reliance on COMPAS is constitutionally inappropriate because COMPAS's algorithm inherently discriminates against young offenders and is based on inaccurate statistics. (See Docket No. 110 ¶¶ 208-16, 224-28, 234-35); see also supra n.5. That argument does not necessarily require a finding that each named Plaintiffs actual risk score, which depends on a multitude of factors besides age, was high.
In addition, this information is relevant to class certification because Plaintiffs allege that by using COMPAS, Defendants systematically punish putative class members like Mr. Delima for being convicted when young. (Docket No. 110 ¶¶ 212, 222-23.ii.n). The raw data and formulas on which COMPAS is based are therefore central to Plaintiffs' assertions that the unconstitutional "practices employed against [them] were common to other class members" who were convicted when young. See Dupres, 2021 WL 2373737, at *3 (noting that class member files and documents that would reveal whether defendants used inappropriate practices to collect on civil court judgments from putative class members in consumer action were relevant to class certification); (Docket No. 110 ¶ 212, 222-23.ii.n). The information also bears directly on Defendants' conduct across the proposed class, and thus, the presence of common issues of fact, whether the named Plaintiffs' claims are typical of the class, and whether they can adequately represent the class's interests. See Dupres, 2021 WL 2373737, at *3; (Docket No. 110 ¶ 212, 222-23.ii-iv). In light of the overlapping purposes of the Compelled Materials, and the fact that this case has been pending for over three years, the Court finds that further delay of this discovery would be unnecessarily wasteful and duplicative. See Hines, 2010 WL 2775921, at *1. Therefore, it declines Northpointe's request to consider this issue only after a class is certified.
The Court is dubious, however, that Plaintiffs need to understand the Compelled Materials in order to depose the individual Commissioners or any other Department of Corrections and Community Supervision ("DOCCS") or Division of Criminal Justice Services ("DOS") personnel, as Plaintiffs assert. (Docket No. 207 at 3). Any argument that the nuts and bolts of COMPAS are relevant to these individuals' subjective knowledge or use of the tool is belied by Plaintiffs' own allegation that this information is not known to those outside of Northpointe. (Docket No. 110 ¶¶ 213, 215-16).
B. Risk of Competitive Injury
The Court next considers Northpointe's interest in preventing competitive injury by disclosure of the Compelled Materials. See Bridgeport Music, 2008 WL 577284, at *1. The Court is sensitive to Northpointe's concerns regarding the potential ramifications of disseminating this proprietary information, and is disquieted by Dr. Rudin's public comments. However, the record before the Court indicates that the risk of intentional or inadvertent disclosure of this information to competitors is not high.
Plaintiffs' papers do not dispute Northpointe's assertion that the Compelled Materials contain highly confidential trade secrets. (Compare Docket No. 207 at 2, with Docket No. 211 at 1-2); (see also Docket No. 181 at 1, 3) (noting that during meet-and-confer discussions, "Plaintiffs were willing to assume the [Compelled Materials] contain trade secrets"). For the first time at oral argument, however, Plaintiffs suggested that the information in the Compelled Materials may not fit this definition because Northpointe has already published the "input[s] to the risk . .. scale[s]" and the Norm Core Data are not proprietary. (Docket No. 219 at 20:6-15, 34:7-21; see also Id. at 27:15-20). District courts "generally reject arguments raised for the first time at oral argument," out of both fairness and a concern for efficiency. See N-N v. Mayorkas, 19-CV-5295(EK), 2021 WL 1997033, at *8 (E.D.N.Y. May 18, 2021); see also In re AMR Corp., 598 B.R. 365, 384 (Bankr. S.D.N.Y. 2019); Goldberg v. UBS AG, 690 F.Supp.2d 92, 98 (E.D.N.Y.2010). Because Plaintiffs failed to raise this argument in any of their written submissions regarding this dispute, Northpointe has not had a fair opportunity to respond. See TV-TV, 2021 WL 1997033, at *8. Therefore, the argument is rejected.
As a general matter, "[t]he disclosure of confidential information on an 'attorneys' eyes only' basis is a routine feature of civil litigation involving trade secrets." In re The City of New York, 607 F.3d 923, 935 (2d Cir. 2010). The Second Circuit has explained that the "purpose" of "attorneys' eyes only" protection is "to prevent aparty from viewing the sensitive information while nevertheless allowing theparty's lawyers to litigate on the basis of th[e] information" disclosed. Id. at 935-36 (emphases in original). Thus, "[p]rotective orders that limit access to certain documents to [both] counsel and experts are commonly entered in litigation involving trade secrets." Vesta Corset Co. v. Carmen Foundations, Inc., No. 97 Civ. 5139(WHP), 1999 WL 13257, at *3 (S.D.N.Y.Jan. 13, 1999); see also Quotron Sys., Inc. v. Automatic Data Processing, Inc., 141 F.R.D. 37, 40 (S.D.N.Y. 1992). Furthermore, "[notwithstanding the issuance of any protective order[s], the court possesses the power to modify [them]" to adequately protect the confidentiality of sensitive commercial information. See Duffy v. Illinois Tool Works Inc., No. 15-CV-7407(JFB)(SIL), 2018 WL 1335357, at *5 (E.D.N.Y. Mar. 15, 2018) (citing Gambale v. Deutsche Bank AG, 311 F.3d 133, 141 (2d Cir. 2004)).
To begin, the Protective Orders already in force prohibit disclosure of the Compelled Materials except to select individuals, and restrict their use to the litigation of this case. (Docket Nos. 198, 151, 154). Dr. Rudin has agreed under oath to be bound by these Orders and subject herself to this Court's jurisdiction for their enforcement and any other relief relating thereto. (Docket No. 217 ¶¶ 15-17). She has further promised to take additional measures to prevent inadvertent disclosure. (Id. ¶ 17). Dr. Rudin has also handled confidential information in the past when consulting on criminal justice and other policy matters. (Docket Nos. 207 at 2; 209-1 at 18). There is no information before the Court calling into question Dr. Rudin's integrity in agreeing to keep the Compelled Materials confidential. Moreover, "[a] [c]ourt should not lightly assume that parties will violate their obligations imposed on them by a duly entered [protective] order on the [c]ourt's docket." See CSL Silicones Inc. v. Midsun Grp. Inc., 3:14-CV-01897 (CSH), 2016 WL 3568173, at *5 (D. Conn. June 27, 2016). In addition, "[although [the potential] . . . injury [to Northpointe] is serious, it involves money, not public safety, and it can usually be remedied by an injunction or money damages" as sanctions for violating such an order. See In re The City of New York, 607 F.3d at 936; see also CSL Silicones Inc., 2016 WL 3568173, at *5.
Numerous courts have permitted disclosure of sensitive commercial information to an expert under that rationale where, as here, the expert has expressly agreed to be bound by a protective order and/or confidentiality agreement imposing sanctions for any violation thereof. See, e.g., CSL Silicones Inc., 2016 WL 3568173, at *4-5 (holding that protective order "obviate[d] . . . concerns" of competitive injury by "limit[ing] experts' use and disclosure of the material, and allowing] Defendant to recover for any damages caused by a breach of that provision"); In re Am. Express Anti-Steering Rules Antitrust Litig. (Noll), 10-CV-04496 (NGG) (RER), 2012 WL 13098456, at *2 (E.D.N.Y.Nov. 16, 2012) (noting that "protective orders guard against such risk [of disclosure] and provide sanctions for any misuse of protected information"); Santella v. Grizzly Indus., Inc., No. 3:12-mc-00131-SI, 2012 WL 5399970, at *5 (D. Or. Nov. 5, 2012) ("[T]he Court concludes that Mr. Domeny will abide by the protective order, which negates the risk of Mr. Domeny intentionally disclosing SawStop's confidential information."); NellsonN. Operating, Inc. v. Elan Nutrition, LLC, 238 F.R.D. 544, 547 (D. Vt. 2006) (finding no "reason to doubt the integrity with which [expert] undercook] to keep [requesting party's] information confidential" where expert had previous experience handling confidential information and agreed to segregate the materials to be disclosed). In light of Dr. Rudin's assurance under oath that she will comply with the Protective Orders, as well as her demonstrated ability to keep similarly sensitive information confidential, (see Docket Nos. 207 at 2; 209-1 at 18; 217 ¶¶ 15-17), the Court finds that the Protective Orders and additional restrictions listed below sufficiently protect against the threat of intentional disclosure to a competitor. See Santella, 2012 WL 5399970, at *5.
Contrary to Northpointe's assertions, Dr. Rudin's negative opinion of Northpointe and COMPAS do not constitute good cause to prevent her review of the Compelled Materials. (See Docket No. 211 at 1-3). Dr. Rudin's criticisms boil down to a belief that as a policy matter, the data and formulas behind COMPAS should not be kept secret from the public or the government officials who use it. However, "[i]t is not uncommon for experts and consultants to be retained over the course of their careers by parties holding opposing points of view and to receive sensitive confidential information in the process." Persian Gulf Inc. v. BP W. Coast Prod. LLC, No. 3:15-cv-01749-L-AGS, 2019 WL 4727938, at *3 (S.D. Cal. Sept. 27, 2019). Dr. Rudin's views alone do not constitute evidence that she is lying or "will somehow use the information for [a competitor's] benefit." See Barry Fiala, Inc. v. Stored Value Sys., Inc., No. 02-2248 Ma/An, 2005 WL 8157420, at *3 (W.D. Tenn. Apr. 13, 2005). Thus, Dr. Rudin's comments shed no light on her credibility, and do not amount to evidence that her "agreement to comply with the Protective Order[s] lacks good faith." See Persian Gulf, 2019 WL 4727938, at *3. Indeed, courts have soundly rejected the proposition that an expert's negative opinion of the disclosing party necessarily means the expert will surreptitiously violate a protective order restricting use of that party's documents. See, e.g., Id. (holding that expert's "ties with consumer groups seeking greater transparency and more government oversight of Defendants' operations" did not demonstrate he was "more likely than other independent experts to violate" the protective order in force); Santella, 2012 WL 5399970, at *5 (finding minimal risk of intentional disclosure where defendant's expert consulted for organization that opposed regulatory mandates of technology used by disclosing party, but agreed to be bound by protective order). Therefore, any argument that Dr. Rudin will purposely violate the Protective Orders in this case simply because she believes no one should be using COMPAS lacks merit.
Northpointe maintains that regardless of her agreement to comply with the Protective Orders, Dr. Rudin's "vendetta against Northpointe is so pervasive that no level of restriction could protect Northpointe if she is given access to the Compelled Materials." (Docket No. 211 at 3). That argument fails, too. Such a risk of inadvertent disclosure is typically triggered when the proposed recipient of sensitive information has an active relationship with one or more of the disclosing party's competitors, and is involved in competitive decision-making. See, e.g., Santella, 2012 WL 5399970, at *5-6; Infosint S.A. v. H. Lundbeck A.S., No. 06 Civ. 2869 (LAK)(RLE), 2007 WL 1467784, at *3 (S.D.N.Y. May 16, 2007); BASF Corp., 321 F.Supp.2d at 1380-81; cf. AG A Med. Corp. v. W.L. Gore & Assocs., Inc., Civ. No. 10-3734 (JNE/JSM), 2011 WL 13135783, at * 10-13 (D. Minn. Dec. 13, 2011); Nellson, 238 F.R.D. at 547. In such situations, the recipient's ethical obligations to the competing entity, combined with the human mind's natural difficulty suppressing information once learned, put the recipient in too difficult of a position to keep the disclosed information confidential, despite his or her best intentions. See NorbrookLab'ysLtd. v. G.C. HanfordMfg. Co., No. 5:03-CV-165(HGM/GLS), 2003 WL 1956214, at *4-5 (N.D.N.Y.Apr. 24, 2003); BASF Corp., 321 F.Supp.2d at 1380-81.
Here, however, Dr. Rudin has never been affiliated with a Northpointe competitor, and does not have plans to consult or work for one in the future. (Docket No. 217 ¶ 14). Dr. Rudin is a member of the NASCLJ, a nonprofit entity mainly comprised of academics whose mission is to improve public policy related to law and justice. (Id. ¶ 5); see also supra n.2. She is also a fierce critic of governments' use of COMPAS to make parole decisions without providing government officials or the public any insight into the factors or data on which COMPAS is based. See supra n. 1. However, neither this criticism, nor Dr. Rudin's advocacy of transparent models other than COMPAS, make her just "as threatening as any direct competitor," as Northpointe contends. (Docket No. 211 at 2). This is because her position as an outside advocate gives her no control over a competitor's decisions to use one particular algorithm or dataset over another. See Santella, 2012 WL 5399970, at *5-6; see also Infosint S.A., 2007 WL 1467784, at *6 (noting that whether a potential recipient of information is involved in competitive decision-making "is arguably the determinative factor in th[e] analysis") (quoting Amgen, Inc. v. Elanex Pharmaceuticals, Inc., 160 F.R.D. 134, 139 (W.D.Wa. 1994)). Thus, her role in the machine learning industry does not implicate the ethical concerns or cognitive difficulties explained above. See Santella, 2012 WL 5399970, at *6 (explaining that involvement in competitive decision-making would put expert in a "position inadvertently to use th[e sensitive] information in reaching a decision because it would be irrevocably present in his mind, but as an outside advocate [expert could] only communicate, or in any other way use, that information through an affirmative act," which did not support a risk of inadvertent disclosure).
Indeed, absent any evidence that Dr. Rudin is involved in competitive decision-making, Northpointe's argument amounts to a fear that Dr. Rudin will communicate Northpointe's proprietary information to the public or a competitor simply because she has the information and believes it should be publicly available. See Id. That behavior would violate the Protective Orders, which Dr. Rudin has promised to respect. See id.; (Docket No. 217 ¶¶ 15-17). Indeed, because the data and inputs at issue amount to "specific values," it is difficult to imagine a scenario in which Dr. Rudin would disclose them by accident. See Santella, 2012 WL 5399970, at *6. Moreover, fear that an expert could innocently misuse sensitive information because it is "in his [or her] head," without more, is too theoretical to warrant preclusion of that expert's receipt of relevant information. See Advanced Semiconductor Materials Am. Inc. v. Applied Materials Inc., No. 95-20169 RMW (EAI), 1996 WL 908654, at *3 (N.D. Cal. Oct. 28, 1996); see also AGA Med. Corp., 2011 WL 13135783, at *12-13.
The Court is also satisfied that the nature of Dr. Rudin's work does not pose any other concerns increasing the risk of inadvertent disclosure. See Nellson, 238 F.R.D. at 547; (Docket No. 217 ¶¶ 16-17). Any benefit Dr. Rudin receives from her advocacy regarding recidivism prediction and other criminal justice issues is limited. (Docket No. 217 ¶¶ 5, 10-11). Therefore, there is minimal financial incentive for her to disclose the Compelled Materials or their contents. She would also face enormous professional and reputational consequences if she were to disclose this information in violation of a court order. Although Dr. Rudin's comments are troubling, the Court disagrees that she will be unable to continue criticizing COMPAS while keeping the Compelled Materials confidential and without letting them influence her ideas. (See Docket No. 211 at 3). The specific data and inputs on which COMPAS is based are unrelated to her argument that such information should be public. Therefore, Dr. Rudin's argument for more transparency would not draw from her knowledge of the Compelled Materials. Northpointe has failed to articulate any instances of inadvertent disclosure that could arise in this context that do not fall within the Protective Orders. The Court, however, cautions Dr. Rudin against engaging in any behavior that creates the impression that she has used or disclosed the contents of the Compelled Materials beyond the scope of this litigation, whether implicitly or explicitly. To ensure further safeguards of Northpointe's information, the Court also directs the parties and Northpointe to jointly propose an additional protective order governing disclosure of the Compelled Materials to Dr. Rudin, and reflecting several additional restrictions set forth below.
C. Risk of Prejudice to Plaintiffs
Plaintiffs argue that Dr. Rudin's expertise is uniquely suited to their prosecution of this case and that granting Northpointe's request would deprive them of their right to retain an expert of their choosing. (Docket No. 207 at 1-2 n.l). They also assert that an understanding of the information in the Compelled Materials is essential to substantiate their claims that Defendants' use of COMPAS violates young offenders' constitutional rights. (See Docket Nos. 181 at 2; 219 at 20:16-21:14, 37:17-38:12). Northpointe counters that the information is unnecessary to support Plaintiffs' claims. (Docket No. 211 at 3). Moreover, Northpointe argues that Dr. Rudin is not uniquely qualified, nor is she "the only expert available to assist in prosecution of [Plaintiffs'] case." (Id.). Given Dr. Rudin's specific expertise and the relevance of the Compelled Materials, the Court agrees that Plaintiffs would be highly prejudiced if it precluded Dr. Rudin's review of the Compelled Materials.
Relying on an opinion from the Northern District of California, Northpointe asserts that the applicable legal standard requires that Dr. Rudin "only be permitted to review the Compelled Materials if 'she has unique experience [sic].'" (Docket No. 211 at 2-3) (quoting/Intel/ Corp. v. Tela Innovations, Inc., No. 3:18-cv-02848-WHO, 2019 WL 2476620, at *13 (N.D. Cal. June 13, 2019)). Although the Court will consider the extent of Dr. Rudin's expertise and qualifications, see Bridgeport, 2008 WL 577284, at *1, the Court declines to rely on this non-binding out-of-Circuit decision for the standard applicable to this dispute. Furthermore, a review of Intel reveals that the standard cited by Northpointe applies "when an individual's past or current work in the field create a 'substantial risk' that information will be misused," which typically occurs when the "individual ha[s] [one or more] active and ongoing relationships with competitors." See 2019 WL 2476620, at *13; (Docket No. 211 at 2-3). Here, where Dr. Rudin has never worked for a Northpointe competitor and her activities do not otherwise pose a "substantial risk" that she will misuse the Compelled Materials, (see supra Section III.B), this standard is not appropriate. See Intel, 2019 WL 2476620, at* 13.
First, Dr. Rudin is extensively qualified in the field of machine learning and has robust experience exploring its practical applications to the social problems at the heart of this case. (See generally Docket No. 209-1). She has devoted her career to developing algorithms that comport with responsible policy initiatives on a variety of topics, including criminology and criminal justice, and is therefore familiar with the competing concerns for safety and personal liberty at stake in parole determinations. (See Id. at 1-3). Dr. Rudin's focus makes her skills particularly apt for Plaintiffs' case, which will ultimately require a technical analysis of the data, inputs and formulas that drive COMPAS and whether that blueprint comports with the Constitution. See Nellson, 238 F.R.D. at 547 ("Expert testimony will be essential to understanding the parties' claims in this case and Elan must have latitude to choose the experts that it believes have the necessary expertise and ability to convey highly technical information to the Court."). It is not clear whether other data scientists possess this specific combination of interests and experience that would help Plaintiffs pursue their case. See In re Am. Express Anti-Steering Rules Antitrust Litig. (Noll), 2012 WL 13098456, at *3.
The Court also finds that at this stage of the proceedings, it would be prejudicial to prevent Dr. Rudin from reviewing the Compelled Materials any longer. See Persian Gulf, 2019 WL 4727938, at *4. Dr. Rudin has confirmed that Northpointe's productions to which she has access do not shed light on the amount of risk COMPAS assigns to particular ages, which is key to Plaintiffs' claims. See Duffy, 2018 WL 1335357, at *5 (permitting disclosure of product formula to expert based on expert's representation that information already available was "insufficient to conduct the analysis with the requisite level of scientific certainty"); Advanced Semiconducter Materials Am. Inc., 1996 WL 908654, at *3 ("Because of the highly technical nature of this case, expert testimony will be crucial to an understanding of ASMA's claims."); (Docket No. 209-3 at 26:2-27:2). The Court already determined that the Compelled Materials are highly relevant to Plaintiffs' core constitutional allegations, and would fill in these informational gaps. (See supra Section III.A). The Court also accepts Plaintiffs' representations that their counsel's review of the Compelled Materials is useless absent assistance from a person with highly technical expertise. (Docket Nos. 207 at 3; 209-3 at 26:3-6; 219 at 20:16-21). Finally, the record indicates that Plaintiffs have been consulting with Dr. Rudin at least for several months, and this case has been pending since 2018. (See Docket No. 209-3 at 26:2-27:2). Even if they could locate another expert with the requisite knowledge to conduct the analysis they need, "forcing Plaintiff[s] to search for a comparable replacement" and educate that person regarding their claims "would unduly impair the prosecution of [their] case." See Persian Gulf, 2019 WL 4727938, at *4. Accordingly, Dr. Rudin's review of the Compelled Materials is crucial to Plaintiffs' efficient pursuit of this litigation, and the Court overrules Northpointe's objection.
IV. CONCLUSION
For the foregoing reasons, Northpointe's motion for a protective order is GRANTED in part, and DENIED in part. Consistent with this Opinion & Order, the parties and Northpointe are directed to submit for the Court's review a supplemental protective order specifically governing Dr. Rudin's review of the Compelled Materials. The supplemental protective order should provide, at minimum, that:
(a) the terms of the Protective Orders entered in this action apply to Dr. Rudin in full force;
(b) Dr. Rudin cannot share or discuss the Compelled Materials or their contents with anyone except for the individuals specifically permitted to review them in those Orders;
(c) the Compelled Materials must be produced to Dr. Rudin in hard-copy form and Plaintiffs' counsel must return them to Northpointe at the conclusion of this case;
(d) Dr. Rudin may not copy, scan or otherwise duplicate or manipulate the Compelled Materials;
(e) at the conclusion of this case, Dr. Rudin must destroy any other records and/or notes relating to the Compelled Materials in her possession and/or control, and Plaintiffs' counsel must certify to Northpointe that she has done so; and
(f) Dr. Rudin cannot use the information in the Compelled Materials for her other projects or for any purpose beyond this litigation.
The Clerk is respectfully requested to terminate the pending motion (Docket No. 211).
SO ORDERED.