REINHARDT, Circuit Judge, with whom Circuit Judges SCHROEDER and THOMAS join, concurring in part and dissenting in part:

The result we reach is most unfortunate. Not only is it wrong as a matter of law, but it may lead some to conclude that we are insensitive to the needs and problems of minority educators and minority students. While that may not be so, one thing is certain. The loser in the case before us is clearly the already beleaguered public education system of the state of California.

Although I concur in Section IA and Parts II, III, and IV of the majority opinion, I cannot agree with the majority's holding that the CBEST was properly validated under Title VII. As a result of this ruling, qualified minority educators — teachers, administrators, librarians, and other officials — will be denied the opportunity to work in California's severely under-staffed public schools, simply because they failed to pass a test that concededly has a disparate impact on minority group members. Despite Title VII's clear requirement that a test that has a disparate impact on minorities must be validated by reference to a particular job, the majority accepts validation studies that fail to differentiate among different school jobs, ranging from bilingual education teachers to mathematics teachers to physical education teachers. Further, even though Title VII demands that the defendants demonstrate "by professionally acceptable methods" that the test items are predictive of or significantly correlated to job-related skills, the majority finds that the CBEST met this requirement through a study that the district court found to be "unscientific" and "not particularly helpful."

A sampling of the plaintiff class illustrates the disservice the majority decision does not only to minority educators but also to the students in the California public school system. The plaintiff class includes a Cambodian-born applicant with a postgraduate teacher preparation certificate and a Bachelor's degree in Liberal Studies from a California State University, who is seeking to teach bilingual elementary school classes to address the needs of over 24,000 Khmer-speaking students in California public schools. He is qualified but for the fact that he is four points short of obtaining the required score on the reading section of the CBEST. The class also includes a Latino resident of Los Angeles County, in which Latinos comprise the largest ethnic group. The man seeks a position as a counselor in the public schools, and has an undergraduate degree from University of California at Irvine and a Master's degree in Counseling from California State University at Los Angeles, but has failed the reading section of the CBEST. He too is qualified, but for the test. Another class member is an African-American Assistant Principal and physical education teacher with a Bachelor's degree and a Master's degree in Physical Education from Stanford, who was promoted from his position as a teacher to Assistant Principal, and who received excellent performance reviews as an Assistant Principal. He has been denied the opportunity to return to his post as Assistant Principal because he is six points short of passing the math portion of the CBEST. The result of the failure to validate the CBEST in a lawful manner is to perpetuate discrimination against minority teachers who could be of particular assistance to minority students and serve as role models for them. It is significant in this regard that minority students outnumber white students in many of California's largest school districts, in some cases by substantial amounts.

As the majority notes, we follow the three-step approach set forth in Craig v. County of Los Angeles, 626 F.2d 659, 662 (9th Cir. 1980) to determine whether a test is job related. I believe that the district court erred in applying the second and third steps of the Craig test, because (1) the studies relied on by the CTC (Commission on Teacher Credentialing) do not differentiate among different school jobs in determining the skills to be tested; and (2) the studies do not demonstrate by professionally acceptable methods that the test items actually measure the job-related skills identified by the CTC. Either of these errors in the method of validation would be sufficient to render the CBEST invalid. Finally, although I believe that the district court's determination of job-relatedness should be reversed even under the clear error standard adopted by the majority, the proper standard for reviewing the determinations at issue in this case is de novo.

I. Failure to validate for separate jobs

The use of a test that has a disparate impact under Title VII is impermissible unless it is proven that the test is "job related for the position in question." 42 U.S.C. § 2000e-2(k)(1)(A)(i) (emphasis added). The Uniform Guidelines provide that "[a]ny validity study should be based upon a review of information about the job for which the selection procedure is to be used." 29 C.F.R. § 1607.14(A). The CBEST is invalid because it was not validated with specific reference to the various particular jobs for which the CBEST is used.

The Supreme Court clarified the job-specific validation requirement in Albemarle Paper Co. v. Moody, holding that "[a] test may be used in jobs other than those for which it has been professionally validated only if there are no significant differences between the studied and unstudied jobs." 422 U.S. 405, 432, 95 S.Ct. 2362, 45 L.Ed.2d 280 (1975). In Albemarle, employment tests designed to measure general intelligence were administered to ten different job groups, which were primarily grouped together on the basis of their proximity to each other in the line of progression, rather than in terms of the specific skills that the jobs required. Id. at 429-30, 95 S.Ct. 2362. The Court held the tests invalid in part because the validation study focused on higher-skilled positions on the production lines without adequately examining whether the tests were valid for workers entering lower level jobs. Id. at 433-34, 95 S.Ct. 2362. Further, the Court noted that one of the tests required for the skilled lines of progression correlated to job performance in only three of the eight lines at issue. Id. at 431, 95 S.Ct. 2362. Because the study had not shown that there were "no significant differences" between the various jobs this test purported to evaluate, the test was held invalid. Id. at 432, 95 S.Ct. 2362.

We required a similar level of job specificity in Craig, which involved, in relevant part, a height requirement for prospective sheriffs. Craig, 626 F.2d at 666-68. Although the height requirement was justified by reference to the need for officers to "command respect," we rejected the generalization that "all police officers expend a significant portion of their time and effort in confrontation and control conditions" because "[n]o study proved the generalization to be a fact." Craig, 626 F.2d at 667. We therefore required the defendants to validate the requirement by reference to more than a general description of a police officer's functions, because different police officers have different duties. Other circuits have also rejected attempts to use a single test to measure skills for jobs that were generally similar but had appreciably different specific duties.

The CBEST was validated without regard to the important differences among the various jobs at issue, which range from twelfth grade mathematics teachers to seventh grade bilingual education teachers to guidance counselors, librarians, and administrators. The 1982 Wheeler and Elias study encompassed skills relevant to the job of "teaching at any grade level;" the 1985 Practioner's Review study divided the "particular jobs" into four groups, specifically (1) elementary school teachers, (2) secondary school teachers, (3) pupil personnel and librarians, and (4) administrators; and the Lundquist study divided the pool into teachers and administrators. Thus, even the study that provided the most specific analysis of job categories did not differentiate among the job duties of bilingual education teachers, mathematics teachers, and physical education teachers, for example.

The majority states that the CBEST establishes only "a minimum level of competence" in skills that are job related for all the positions in question. While it may be possible to construct a test that measures skills relevant for all teachers, Albemarle holds that such a test may only be used if it has been validated for each "particular job" involved or if there are "no significant differences" among the jobs at issue. Albemarle, 422 U.S. at 432, 95 S.Ct. 2362. The positions at issue in AMAE bear less resemblance to each other than those in Craig, Albemarle, or the cases from other circuits. If a validation study must differentiate between different officers in the sheriff's department or workers on different lines at the paper production factory, it is certainly improper for the CBEST to be validated by reference to undifferentiated but significantly different teaching jobs. A validation study that focuses on the "particular job" of "a teacher" or even "a secondary school teacher" fails to satisfy the legal requirement that the examination predict job performance for each particular job position in question. Teachers' jobs are too diverse to be lumped into one or two categories for validation purposes. There is no legal or logical basis to support the majority's abandonment of the job-specific validation requirement employed in Albemarle and Craig. In doing so, it clearly erred.

II. Failure to match test items to skills important to the job

As the majority notes, the third step of the Craig test requires employers to "demonstrate by `professionally acceptable methods' that the selection device is `predictive of or significantly correlated' with the element of work behavior identified [by the job analysis]." Craig, 626 F.2d at 662 (quoting Albemarle Paper, 422 U.S. at 431, 95 S.Ct. 2362). In other words, the employer must demonstrate not only that the skills identified in the job analysis are job related, but also that the test items actually measure those skills. In Contreras v. City of Los Angeles, for example, this requirement was satisfied by an independent "examination review phase" in which a new group of job experts reviewed each test question and decided whether it tested one of the critical elements identified in the preceding "job analysis" phase. 656 F.2d 1267, 1282 (9th Cir. 1981); cf. Craig, 626 F.2d at 667 (finding that testimony from the sheriff and a physical training instructor that height was important in commanding respect and applying restraint was insufficient to show by professionally acceptable methods that the height requirements were predictive of the ability to carry out the "control functions" of the job). As the majority acknowledges, "a key requirement of [the] third step, a requirement essential to the proof of job relatedness, is that the validation method be professionally acceptable." Contreras, 656 F.2d at 1282. Here, even assuming that the job analysis was sufficiently specific to generate skills important for all the positions in question, the CBEST is invalid because the CTC failed to demonstrate in a professionally acceptable manner that the test actually measures the skills identified in the job analysis.

The majority appears to concede that neither the Practitioner's Review study nor the Lundquist study attempted to match the test items to the job skills as required under Craig. Maj. op. at 589. To support the district court's finding of validity, the majority cites evidence describing the process of test development employed by ETS. However, this evidence merely purports to show that the test items were expertly developed; it does not demonstrate, as the third step of the Craig test requires, that the final version of the CBEST administered to job applicants is "predictive of or significantly correlated" with the skills identified in the job analysis as important for the job. Where an employer seeks to use a test that has a disparate impact on minorities, it must present evidence that the test actually measures the skills that have been identified as important for the job, and may not rely solely on the fact that it used a "well known and respected as a developer of standardized tests." Maj. op. at 588 (quoting the report of Dr. William A. Mehrens).

As evidence that the test questions were "matched" to the skills they were intended to measure, the majority points only to the Curriculum Matching Project, in which two ETS employees matched CBEST test specifications to material found in textbooks purportedly used in California public schools. The district court criticized this study as "unscientific" and "not particularly helpful," but nevertheless upheld the test. AMAE v. State of California, 937 F.Supp. 1397, 1412 n. 21 (N.D.Cal. 1996). We have explicitly held that it is "essential" that the item validation method be professionally acceptable. Contreras, 656 F.2d at 1282. "Unscientific" and "not particularly helpful" tests clearly do not meet that standard. Thus, even under a clear error standard, the district court's determination that step three of the Craig test was satisfied requires reversal.

III. Standard of review

It is plain from a reading of the majority's opinion that it is heavily influenced by the fact that we are reviewing district court determinations that it believes to be of some complexity. Indeed, in oral argument, the state relied almost exclusively on the fact that the district court had considered the matter carefully and reached its conclusions following a long trial. This may have caused the majority to fail to examine sufficiently the legal inadequacies of the validation studies pointed out in the preceding sections of this dissent.

At each step in its analysis of job-relatedness, the majority emphasizes that the district court's determination was not clearly erroneous. However, as the majority recognizes, courts have not discussed the appropriate standard of review for a district court's ruling on test validation issues. In fact, the standard of review here should be the same as it was in Albemarle and Craig, where the Supreme Court and this court reversed a district court's determination that a test was valid based on errors in the application of the job-relatedness test similar to those committed by the district court here. At no point did either court suggest that the clearly erroneous test applied.

Although I believe that the district court's determination of job relatedness should be reversed regardless of the applicable standard of review, the proper standard is de novo. The majority fails to separate errors regarding the application of the various prongs of the Craig test, which involve primarily legal issues, from errors regarding the underlying factual evaluations of expert testimony and statistical significance. The errors here fall in the former category and are therefore subject to de novo review. See United States v. McConney, 728 F.2d 1195, 1201 (9th Cir. 1984) (en banc).

IV. Conclusion

The defendants failed to meet their burden of demonstrating that the CBEST, with its discriminatory impact, is job related. The studies relied on by the CTC to validate the test did not differentiate among different teaching jobs, and the CTC did not demonstrate by professionally acceptable methods that the test items actually measured the skills identified in the job analysis. Title VII's requirement that tests be validated properly must be vigilantly enforced to ensure that tests that have a disparate impact on minorities are used only if they are job related for the position in question and if no alternative nondiscriminatory selection device exists. Plaintiffs' challenge to the CBEST implicates one of the most important objectives of Title VII — preventing employers from erecting arbitrary barriers which interfere with minority access to employment opportunities. Griggs v. Duke Power Co., 401 U.S. 424, 432, 91 S.Ct. 849, 28 L.Ed.2d 158 (1971). By erroneously affirming the district court's decision, we allow the State of California to perpetuate discrimination against qualified minority teachers, who are already seriously underrepresented in the California public school system, and, derivatively, against minority students as well. In the end, all Californians are the victims of the discriminatory tests the majority upholds. I respectfully dissent.

FERNANDEZ, Circuit Judge, with whom RYMER, Circuit Judge, joins, Concurring in part and Dissenting[fn1] in part: [fn1] Judges O'Scannlain and Kleinfeld concur in the portion of this opinion which dissents from part IV of the majority opinion.

I concur in the majority opinion with the exception of part IV, where the majority determines that a district court can refuse to award costs to the prevailing party simply because the court decides that the losing parties do not have much money, and are worthy because they have presented an important issue which is close and has some merit. As to that, I dissent.

To my mind, reasoning along that line bespeaks an improper bias in favor of plaintiffs. If the issue is important, both sides have a hand in seeing to it that the true public interest is vindicated, and, as it turns out, a defendant may well be the party correctly representing that public interest. Here we deal with the public's interest in having competent people running its schools and teaching its children, and Defendants showed that it is Plaintiffs who erred when they incorrectly thought that a competency test was overly burdensome and improper. That is to say, Plaintiffs' claim that they and those they represent should not have to meet the minimal competency requirements of the CBEST was wrong. The public interest has been vindicated, but not by them. Rather, Defendants properly represented the public's desires and aspirations for California's children. They were right, and should not be denied their costs.

Thus, I support the reasoning found in National Information Services, Inc. v. TRW, Inc., 51 F.3d 1470, 1471 (9th Cir. 1995), which recognized, as does the majority opinion, that "Rule 54(d)(1) creates a presumption in favor of awarding costs to the prevailing party." Because of that strong presumption, "[a] district court therefore generally must award costs unless the prevailing party is guilty of some fault, misconduct, or default worthy of punishment." Id. at 1472; see also Zenith Ins. Co. v. Breslaw, 108 F.3d 205, 207 (9th Cir. 1997). As we noted in National Information Services, 51 F.3d at 1471, it does not matter that the plaintiff sued in good faith and with no vexatious purpose, and even raised "`important and intricate' legal questions and included claims that were `not meritless.'" That is no reason to punish the defendant.

When a prevailing party is denied costs, that party is punished, but the punishment should not be exacted unless "the party has done something to deserve it." Id. at 1472. Of course, the district court should have broad equitable discretion to decide whether the prevailing party "deserves it," but that is where the discretion should generally stop. Therefore, all of the other reasons cited by the district court and the majority to justify denying costs to the prevailing parties in this case are improper. They, in fact, represent an abuse of discretion by the district court.

The majority opinion correctly notes that some of our prior decisions have upheld district courts' decisions to deny costs to the prevailing parties on the basis of the losing parties' limited financial conditions. See, e.g., Wrighten v. Metropolitan Hospitals, Inc., 726 F.2d 1346, 1358 (9th Cir. 1984); Moore v. Hughes Helicopters, Inc., 708 F.2d 475, 486 (9th Cir. 1983); National Organization for Women v. Bank of California, 680 F.2d 1291, 1294 (9th Cir. 1982). However, they did so without much explanation, and, unlike the majority, I would overrule those decisions instead of National Information Services.

The fact, if it is a fact, that Plaintiffs do not have access to much money is hardly a good reason to allow them to impose enormous costs upon Defendants, which have shown themselves to have been correct in the first place. I see nothing especially extraordinary about plaintiffs who are of modest means but still sufficiently well funded to carry out lengthy litigation. I see no reason to encourage them to cause great expense to their adversaries, safe in the knowledge that there is little downside to their efforts. Again, that is why this court has properly "recognized that denial of Rule 54(d) costs operates to punish the prevailing party for some impropriety during the course of litigation," and should otherwise be eschewed. Zenith, 108 F.3d at 207; see also Cherry v. Champion Intern. Corp., 186 F.3d 442, 447-48 (4th Cir. 1999) (holding that there should not be an exception to the presumption in favor of awarding costs to prevailing party because losing plaintiff is of "modest means," and disparity of resources is likewise an improper basis); Smith v. Southeastern Pennsylvania Transp. Authority, 47 F.3d 97, 100 (3d Cir. 1995) (holding that disparity in financial resources between the parties is not a valid basis to exempt losing party from paying costs).

Nor should it matter that Defendants are public entities. See Smith, 47 F.3d at 99 (denying costs would be unfair to taxpayers who subsidize prevailing defendant); Burroughs v. Hills, 741 F.2d 1525, 1538 (7th Cir. 1984) (Posner, J., concurring) (The plaintiffs' limited means claim "is entitled to little weight in a case such as this where the defendants are . . . public officials whose expenses of suit were borne by the United States, which is to say by the federal taxpayer. Most taxpayers are persons of limited means; most government revenue is raised from such persons, and not from the wealthy few. The denial of costs to the defendants in this case will not contribute, however slightly, to a more egalitarian distribution of the nation's wealth.").

I find it regrettable that the majority is using this case for the purpose of overruling National Information Services and enshrining a kind of plaintiffs-are-good-defendants-are-not test in its place. While the majority does not specifically say that it is taking that position, the denial of costs to Defendants surely has that effect. That is most unfortunate. See Mitchell v. City of Moore, 218 F.3d 1190, 1204 (10th Cir. 2000) (rejecting argument that "presumption in favor of costs should be relaxed when the prevailing party is the defendant in a civil rights case"); Cherry, 186 F.3d at 448 (holding that presumptive award of costs to prevailing party cannot be overcome because of "public interest" nature of the litigation); Contreras, 119 F.3d at 1295 (holding that civil rights plaintiffs do not warrant special treatment with respect to payment of costs to prevailing party); Jones, 789 F.2d at 1233 (same). In many cases, it is inequitable to require prevailing parties to bear their own attorney's fees. National Information Services' holding that unless the prevailing parties did something wrong, we should not impose the further inequity of depriving them of costs may not be all encompassingly perfect. However, it is much closer to perfect than the position we now adopt.

Thus, while I agree with most of the fine majority opinion, I must respectfully dissent from its decision regarding costs.

KLEINFELD, Circuit Judge, with whom Circuit Judge O'SCANNLAIN joins, concurring in part and dissenting in part.

I join in parts I(B), II, and III of Judge Graber's opinion, but not parts I(A) or IV; in part II of Judge Gould's opinion but not part I; that part of Judge Fernandez's opinion dissenting on costs but not his concurrence in part I(A) of Judge Graber's opinion. That is to say, I agree with Judge Graber that the test was adequately validated and did not violate the Civil Rights Act. And I agree that the judgment does not need to be vacated because of the district court's use of an in-chambers expert (though Judge Tashima's concerns strike me as valid, and I might join in his view were I not confident that the result has to be the same regardless of the ex parte advice from the judge-selected expert). I agree with Judge Gould's dissent on whether Title VII applies. And I agree with Judge Fernandez that we should reverse the denial of costs.

I write separately for two related reasons. First, without at least a paragraph like the one above (and a scorecard to keep track of such paragraphs), readers of the opinion cannot figure out who stands where, and what the count is. So I have to write something. Second, our decision on whether Title VII applies, which generates most of our division (and the need for so incomprehensible a sentence as the one with which I begin this separate opinion), is entirely unnecessary and could better be avoided. We are not bound to avoid deciding, as we would be with a constitutional question, but it would be preferable for us to avoid deciding.

The way to avoid deciding the issue of whether Title VII applies is the obvious and traditional formula: "Assuming without deciding whether Title VII applies, we conclude that it was not violated, because this test does not discriminate in violation of that law." I would have preferred that we had used that formula.

The reason why I advocate such avoidance is not the difficulty of the question. Judge Gould's analysis is compelling and clearly correct. Congress prohibited discrimination by employers with respect to employees. The defendants do not employ the plaintiffs or those whom the plaintiffs represent. So the Title VII prohibition does not speak to the situation.

Congress adopted the Civil Rights Act of 1964 after one of the great legislative battles of our time. It did not choose to impose the expense of justifying tests and other actions that affect employability on third parties other than employers. The country suffered from massive direct and intentional race discrimination at that time. Considering the political challenge that the Civil Rights Act of 1964 posed for its advocates, and the skill and consumption of political capital it required of such great men as President Lyndon Johnson, Senator Hubert Humphrey, and Senator Everett Dirksen, we cannot assume that Congress would have gone any further than it did. Reading statutes as if they said what they do not say, in order to go further than the legislature did, vitiates careful legislative compromises. If we do not respect the compromises legislators make, how shall they be induced to make them? "If we were to `fully effectuate' what we take to be the underlying policy of the legislation, without careful attention to the qualifying words in the statute, then we would be overturning the nuanced compromise in the legislation, and substituting our own cruder, less responsive mandate for the law that was actually passed." Those who only got half a loaf from Congress frequently come to the federal courts for the other half, but their mail ought to be forwarded to Capitol Hill.

There are several reasons why it would be better for us to avoid deciding whether Title VII applies. One is that deciding the issue reduces the collegiality of our decision. By "collegiality" I do not mean civility or good feeling among colleagues, meanings sometimes inferred from the word. I mean the dictionary definition, "shared authority among colleagues," so that we meld our individual voices into the voice of the court. An appellate court ought to speak collectively as nearly as possible, particularly when it speaks with the force of an en banc decision. In my experience, the way votes are accumulated for a collegial decision is mainly by deletion of matters not essential to the result, and on which there is disagreement.

In this case, the decision on whether Title VII applies is not essential to the result. Of the eleven judges on this en banc panel, nine agree that, whether Title VII applies or not, the test was adequately validated and would not violate Title VII. We could simply stop there. If the test is appropriate even if administered by an "employer," as the district court found and nine of us would affirm, then we need not decide whether a non-employer state board should be treated as if it were an employer for purposes of Title VII.

There are quite a few reasons why such collegiality is desirable. One is that the law is hard to figure out when a reader needs a scorecard to count votes on an issue by issue basis. It is much easier to know what the law is from Chief Justice Marshall's unanimous opinions, than from the old English Reports with individual judges' opinions reported seriatim. Careful jurists can fall into disagreement on how to keep score. When we do decide on an issue, despite the decision being inessential to the result, there is necessarily a question of whether our decision is mere dictum. The traditional definition of obiter dictum is a "remark by the way" or as "[s]tatements and comments in an opinion concerning some rule of law or legal proposition not necessarily involved nor essential to determination of the case in hand . . . and lack the force of an adjudication." Our resolution of inessential issues invites more litigation.

In this case, our reaching out to hold that Title VII applies to non-employers opens the door to more litigation, instead of shutting it. We have now created a circuit split on a national issue of great importance. As Judge Gould points out, all the other circuits to have ruled on whether Title VII (or the analogously construed ADEA provision) applies to non-employers such as state licensing boards have gone the other way. In a schools case in the Seventh Circuit, a veterinary board case out of the Third, and most closely analogous, a school teacher state certification examination case out of the Fifth, our sister circuits have gone the other way. (So have we, for that matter, in a dental board case.) Though the majority opinion struggles to distinguish these cases, it is indeed a struggle, and the distinction is not solid enough for any state licensing board to rest on. A decision like today's, where we set up an inter-circuit conflict on a matter of great national importance, is colloquially known among those of us who do appellate law for a living as "cert bait." That is to say, we have furnished the Supreme Court with a reason to grant a petition for certiorari. As soon as we file our opinion, the highly capable lawyers involved will start calling around to arrange meetings relating to the certiorari petition and amicus briefs.

The majority opinion today says that we ought to decide the Title VII applicability issue because it is logical, but there is nothing illogical about assuming without deciding. In the study of logic, assuming without deciding is termed "conditional," "if p then q." Just as it is perfectly logical to say "if today is Thursday then tomorrow is Friday," without saying whether today is Thursday, it is perfectly logical to say that "if Title VII applies, it is satisfied," without saying whether Title VII applies.

The majority also says we should decide the Title VII issue even though we do not have to reach it, because the parties probably want to know the answer, particularly the State of California. Somehow I doubt that the State of California really wants to know that now it may have to spend hundreds of thousands, maybe millions, of dollars, and years of litigation, validating examinations and possibly all the other screening it does of all licensed professions. More likely, both sides just wanted to win this case and establish either that the CBEST examination was or was not permissible. These are school people. They probably care about schools, and want a final answer about the CBEST examination. We have undercut the finality of the answer by reaching out for "cert bait."

There is another, and to my mind, especially important reason why we ought to avoid deciding this question on which we disagree. We may be wrong. On anything, we can be wrong. I am fairly confident that Judge Gould is right about Title VII, but I could be wrong. The scope and applicability of Title VII is an exceedingly important, and always controversial, question. Because we are not elected and always in every circumstance run a substantial risk of being wrong, the less we govern the better. That is not to say we should dodge the controversial. If the law is established and the facts show a violation, we are obligated to apply the law without regard to whether it is controversial. But we should restrain ourselves from deciding important public policy issues, when they are unnecessary to decision in the matter before us, we lack a consensus, all the other circuits to have decided them have gone the other way, and the law we establish is not clearly compelled. The people can govern themselves.

All that said, I am not going to make the score even harder to keep by withholding my judgment. We have reached the Title VII issue whether I like it or not. I cannot prevent us from reaching it. So I join Judge Gould in concluding that it does not apply because the lawsuit is not against the relevant employer.

GOULD, Circuit Judge, concurring in part and dissenting[fn1] in part: [fn1] Judges O'Scannlain and Kleinfeld join in part II, agreeing that Title VII does not apply to the CBEST.

I

I concur in Parts II, III, and IV of the majority's opinion, and in the judgment. I write separately, however, to dissent from Part I and to express my view to the contrary that Title VII does not apply to the CBEST. The majority extends the reach of Title VII far beyond what Congress intended and in so doing creates potential mischief for all of our states.

II

Section 703(a)(1) of Title VII provides, in relevant part:

(a) It shall be an unlawful employment practice for an employer —

(1) to fail or refuse to hire or to discharge any individual, or otherwise to discriminate against any individual with respect to his compensation, terms, conditions, or privileges of employment. . . .

42 U.S.C. § 2000e-2(a)(1).

A natural reading of this section suggests that the "individual" it references is a potential, current, or past employee of the employer. That is the typical Title VII case. See, e.g., Oncale v. Sundowner Offshore Servs., Inc., 523 U.S. 75, 118 S.Ct. 998, 140 L.Ed.2d 201 (1998); Harris v. Forklift Sys., Inc., 510 U.S. 17, 114 S.Ct. 367, 126 L.Ed.2d 295 (1993); Ansonia Bd. of Educ. v. Philbrook, 479 U.S. 60, 107 S.Ct. 367, 93 L.Ed.2d 305 (1986); Albemarle Paper Co. v. Moody, 422 U.S. 405, 95 S.Ct. 2362, 45 L.Ed.2d 280 (1975); Griggs v. Duke Power Co., 401 U.S. 424, 91 S.Ct. 849, 28 L.Ed.2d 158 (1971).

However, in a line of cases beginning with Sibley Memorial Hospital v. Wilson, 488 F.2d 1338 (D.C. Cir. 1973), many courts — including this one — have extended Title VII liability to employers who discriminatorily interfere with an individual's employment relationship with a third party. See id. at 1341; see also Gomez v. Alexian Bros. Hosp., 698 F.2d 1019, 1021 (9th Cir. 1983). To extend Title VII's reach in this way requires not only an unnatural reading of the statute, but also a dubious inference about Congress' intent in enacting it. If Congress had intended to create liability for interference with third-party employment relationships, there is no discernable reason why it would have limited that liability to "employers," as defined in Title VII. See, e.g., Ehret v. Louisiana, 862 F.Supp. 1546, 1550 (E.D.La. 1992) (voluntary association of river pilots that interfered with third party employment relationship not subject to Sibley liability because it was not an "employer"). As the Seventh Circuit has noted:

It might be a good idea to impose liability on those who aid or abet violations of [anti-discrimination laws such as Title VII], but what sense would it make to confine that liability to persons or firms that happen to be employers? Since it would make very little sense that we can see, . . . we find it implausible to impute to Congress an intention to create, by language not at all suggestive of any such intention, aider and abettor liability of one employer to the employees of another employer.

EEOC v. Illinois, 69 F.3d 167, 169 (7th Cir. 1995).

This interpretation of the statute under Sibley and Gomez when applied to private employers at least advances the purpose of Title VII to eradicate discrimination in the workplace, and is consistent with the general directive that courts construe the statute broadly to effectuate that purpose. See Duffield v. Robertson Stephens Co., 144 F.3d 1182, 1192 (9th Cir. 1998).

III

The same cannot be said, however, of the majority's further extension of Title VII to impose indirect liability on the State of California. The Supreme Court repeatedly has admonished that, absent a clear expression of congressional intent, courts must not interpret federal statutes as creating "in-roads by implication on state authority" to exercise what have historically been state police powers. Palmer v. Massachusetts, 308 U.S. 79, 84, 60 S.Ct. 34, 84 L.Ed. 93 (1939); see also Will v. Michigan Dep't of State Police, 491 U.S. 58, 65, 109 S.Ct. 2304, 105 L.Ed.2d 45 (1989). Numerous courts have recognized that Sibley's liberal reading of the statute cannot extend Title VII liability to states that, in the course of exercising their police powers in good faith, may indirectly affect employment prospects. In Haddock v. Board of Dental Examiners, 777 F.2d 462 (9th Cir. 1985), for example, this court correctly observed that there is no evidence that when Congress amended Title VII to apply to state governments, it "intended to benefit anyone other than those actually employed by state governments or their subdivisions." Id. at 464 (emphasis added). We thus held that Title VII did not apply to the licensing activities of the California State Board of Dental Examiners. See id. Virtually all other courts to have considered Title VII's possible application to state licensing activities and to other exercises of traditional police powers have reached the same conclusion. See, e.g., Fields v. Hallsville Indep. Sch. Dist., 906 F.2d 1017, 1020 (5th Cir. 1990); George v. New Jersey Bd. of Veterinary Med. Exam'rs, 794 F.2d 113, 114 (3d Cir. 1986); United States v. New York State Dep't of Motor Vehicles, 82 F.Supp.2d 42, 51 (E.D.N.Y. 2000); National Org. of Women v. Waterfront Comm'n of New York Harbor, 468 F.Supp. 317, 320 (S.D.N.Y. 1979); see also EEOC v. Illinois, 69 F.3d at 171 (because state regulation of teaching, even though pervasive, is exercise of police powers, Sibley liability does not apply).

The majority's holding in Part I.A strays from these cases and is inconsistent with the Supreme Court's controlling law and mandate on which they rest. It also creates a striking conflict among the circuits on an issue of importance to our states, is discordant with basic principles of federalism, and is unworkable in practice.

A

The Supreme Court has consistently held that the establishment and regulation of a public school system lies at the core of a state's historic police powers; it is quintessentially a governmental, not a proprietary function. See Wisconsin v. Yoder, 406 U.S. 205, 213, 92 S.Ct. 1526, 32 L.Ed.2d 15 (1972) ("Providing public schools ranks at the very apex of the function of a State."); Hadley v. Junior College Dist., 397 U.S. 50, 56, 90 S.Ct. 791, 25 L.Ed.2d 45 (1970) ("Education has traditionally been a vital governmental function."); Epperson v. Arkansas, 393 U.S. 97, 104, 89 S.Ct. 266, 21 L.Ed.2d 228 (1968) ("[P]ublic education in our Nation is committed to the control of state and local authorities.").

Because California regulates teaching pursuant to its police power, we should read Title VII as imposing liability on such regulation only if congressional intent to do so is clear. See United States v. Lopez, 514 U.S. 549, 580-81, 115 S.Ct. 1624, 131 L.Ed.2d 626 (1995) ("[E]ducation is a traditional concern of the States. . . . In these circumstances, we have a particular duty to ensure that the federal-state balance is not destroyed. We start with the assumption that the historic police powers of the States are not displaced by a federal statute unless that was the clear and manifest purpose of Congress.") (Kennedy, J., concurring; citations and quotations omitted). As noted above, evidence of such an intent is far from clear. In fact, as this court recognized in Haddock, it is entirely absent. Courts have no business imputing to Congress an intent that Title VII's specific terms reach the sort of regulatory activities at issue in this case.

The majority turns away from this necessary conclusion by taking two routes to error. First, it argues that "[t]here is no overarching `licensing' exception to Title VII," and "the state's high level of involvement in all areas of local public schools distinguishes this case" from the licencing cases cited above. This argument misperceives the issue and misapprehends applicable law. The rationale underlying the licensing cases has little to do with the extent of the state's involvement in the regulated activity, as the majority asserts, and everything to do with the principle that courts should not lightly infer a congressional intent to regulate in areas of peculiarly state concern. See, e.g., New York State Dep't of Motor Vehicles, 82 F.Supp.2d. at 51.

Both the United States Supreme Court and the California Supreme Court recognize that the establishment and control of public schools is a peculiarly governmental function. See Lopez, 514 U.S. at 580-81, 115 S.Ct. 1624; Butt v. State, 4 Cal.4th 668, 685, 689, 15 Cal.Rptr.2d 480, 842 P.2d 1240, 1248, 1250 (1992). That California's regulation of public schools is pervasive does not convert that activity from a governmental exercise of police power into a proprietary activity. Rather, California's pervasive regulation of public schools is the result of the state constitution's requirement that the state ensure equality of educational opportunity. See generally Butt, supra. The degree of regulation that California exerts over public schools shows that it is properly performing a governmental function in light of the state constitutional mandate that underlies that function.

Second, on another erroneous path, the majority asserts that, because the CBEST applies only to public school employees, in imposing the test requirement California is acting in a proprietary capacity. This assertion fails to see the full picture of California's educational system. California's regulation of teacher qualifications is not confined to the public school system, but extends to all teachers. Private tutors and parents intent on home schooling must possess valid teaching credentials. Cal. Educ. Code § 48222. Similarly, private school teachers must be "capable of teaching," and to permit the state to monitor that requirement private schools are required to file annually with the Superintendent of Public Instruction a statement listing each of their faculty members and their qualifications. Cal. Educ. Code §§ 48222, 33190. Students who attend private schools that do not meet these standards are not excused from the state's compulsory education laws that require public school attendance. See id.; see also Cal. Educ. Code § 48200.

That the state has chosen to impose different qualification requirements on teachers in different settings does not mean that the CBEST is an exercise of proprietary, rather than governmental, powers. Instead, it reflects the differing circumstances and differing state interests that are implicated. For example, the California Constitution requires the state to "ensure basic educational equality" in its public schools. Butt, 4 Cal.4th at 681, 15 Cal.Rptr.2d 480, 842 P.2d at 1249. At the same time, under the United States Constitution, California may not require all children to attend those public schools, because "[t]he fundamental theory of liberty upon which all governments in this Union repose excludes any general power of the state to standardize its children by forcing them to accept instruction from public teachers only." In re Shinn, 195 Cal.App.2d 683, 693, 16 Cal.Rptr. 165, 172 (1961) (quoting Pierce v. Society of Sisters, 268 U.S. 510, 535, 45 S.Ct. 571, 69 L.Ed. 1070 (1925)). While the state may have a strong interest in ensuring uniformity of basic skills among public school teachers, it does not have the same interest in ensuring uniformity among private school teachers or private tutors. For this and perhaps other reasons, in exercising its police powers California might rationally impose different qualification requirements on teachers in these vastly different settings. See, e.g., People v. Turner, 121 Cal.App.2d Supp. 861, 866-67,

263 P.2d 685, 688 (Dep't Super.Ct. 1953) (describing basis for distinction in qualification requirements for private tutors, home school teachers, and teachers in private schools). Further, even if a lack of basic skills were perceived to be a problem in all educational settings, it is well-established in precedents assessing equal protection that a state may address such problems in a piecemeal fashion. See, e.g., City of New Orleans v. Dukes, 427 U.S. 297, 303-04, 96 S.Ct. 2513, 49 L.Ed.2d 511 (1976).

B

Not only does the majority's holding miss the clear mark set by Supreme Court precedent, it also squarely conflicts with the decisions of the only two other circuits to have considered a similar issue. In Fields v. Hallsville Independent School District, 906 F.2d 1017 (5th Cir. 1990), the Fifth Circuit held that a Texas teacher skills test, much like the CBEST and applicable only to public schools, was merely a licensing exam and not subject to Title VII. Id. at 1020. The majority attempts to distinguish Fields by urging that in that case there was little evidence of the "control" over schools that is present here. The evidence that the Fields court was looking for, but did not find, was evidence of, for example, whether the state "played any role in the general hiring and firing of teachers" or was involved in their "daily supervision." Id. No such evidence existed there, and none exists here either. Hiring and firing of California's school teachers is performed by local school boards, as the majority concedes. And those boards are charged by statute with controlling the daily activities that teachers perform, and with evaluating their performance. See, e.g., Cal. Educ. Code §§ 35020, 44660.

Similarly, in EEOC v. Illinois, the Seventh Circuit rejected the theory that Sibley liability applied to render the State of Illinois liable for discrimination in school districts' hiring and firing of teachers, despite the pervasive control that the state exerted over various aspects of the districts and their teachers, not unlike the control exerted by California upon which the majority's theory relies:

The State of Illinois exerts more control over public school teachers than over any private employees in the state and probably over any other persons formally employed by local governments in the state. The state fixes not only a minimum salary for all teachers — which is after all not much different from fixing a minimum wage for private as well as public employees, and no one supposes that the federal government is the indirect employer of all the workers covered by the federal minimum-wage law — but also the number of days a teacher must work, what holidays he gets off, the amount of sick leave he is entitled to, his eligibility for and length of sabbatical leave, the minimum lunch period, the terms of teachers' tenure, the rights of recalled teachers, and much else besides. 105 ILCS 5/24-1 et seq. But we do not think this makes the state the "real" employer of these teachers. So far as discrimination in hiring and firing on the basis of age or other forbidden characteristics is concerned, the key powers are, naturally, those of hiring and firing. Those powers are in the hands of the local school district, though constrained of course by the tenure provision of the state's school code. 105 ILCS 5/24-12. Fields v. Hallsville Independent School District, 906 F.2d 1017, 1020 (5th Cir. 1990) (per curiam), holds that a state's role in licensing public school teachers does not make them the state's employees. To similar effect, see Haddock v. Board of Dental Examiners, 777 F.2d 462, 464 (9th Cir. 1985). We think the present case is closer to Fields and Haddock than to the cases that classify the defendant as an indirect employer.

69 F.3d at 171.

C

The majority's theory that a state that extensively regulates an activity is subject to Sibley liability also is unworkable. Where and how are we to draw a line, other than arbitrarily? States regulate a wide range of activities in ways that potentially interfere with individuals' employment opportunities with third parties. For example, California extensively regulates farm labor contractors, requiring them to have a state license, and regulating their relations both with farm laborers and the farms with whom they contract. See Cal. Labor Code § 1682 et seq. It heavily regulates the pharmacy industry, including licensing pharmacists, and regulating their authority, duties, and work hours. See Cal. Bus. Prof. Code § 4051 et seq.; Cal. Labor Code § 850. It regulates school busing, including driver certification, whom buses may carry, and the features that buses must have. See Cal. Educ. Code § 39830 et seq. It regulates nursing and nursing education, requiring licenses for all nurses, and special licenses for public health nurses, including those employed by local, in addition to state, agencies. See Cal. Bus. Prof. Code §§ 675-79, 2700 et seq. It regulates harbors and navigation, including navigation, Cal. Harb. Nav. Code § 240 et seq., vessels, id. § 399 et seq., masters and crews, id. § 790 et seq., and requires special licenses for pilots in the San Francisco Bay Area, id. § 1105 et seq.

Under the majority's theory, depending on the extent of the regulation in each of these areas, the State of California potentially could be liable under Title VII for the good faith exercise of its police powers to protect migrant farm workers, the public health, the safety of school children, and the environment and safety in San Francisco, San Pablo, and Suisin Bays. This result would impair federalism and is not a choice made by Congress.

The majority's opinion also calls into question this court's own decision in Haddock. The majority contends that Haddock is not inconsistent with its holding in Part I.A, arguing that in that case "licensing was the entire connection between the plaintiffs and the defendants." The State of California, however, is heavily involved in regulating the dental profession, including such diverse areas as setting licensing requirements, employment and compensation for acupuncturists in dental offices, the use of general anesthesia, and the facilities and equipment that dentists may use. See Cal. Bus. Prof. Code § 1600 et seq. On the majority's theory, this regulation seems sufficient to render the State of California, if not the Board of Dental Examiners, liable under Title VII for any discriminatory impact in the dental licensing exam. Was Mr. Haddock's only problem that he sued the Board of Dental Examiners rather than the state?

These are only examples but illustrate that the majority's reliance on a "pervasive involvement" trigger for state Title VII liability either will extend Title VII into every aspect of a state's exercise of its police power, or will permit arbitrary application of Title VII depending on each court's interpretation and assessment of the scope of state regulatory control in varied fields. Neither result could be what Congress had in mind when it made it unlawful for "employers" "to fail or refuse to hire or to discharge any individual, or otherwise to discriminate against any individual with respect to his compensation, terms, conditions, or privileges of employment. . . ." 42 U.S.C. § 2000e-2(a)(1).

CONCLUSION

The majority's holding in Part I.A ignores the mandate of the Supreme Court that, absent clear evidence of a congressional intent, courts must refrain from construing federal statutes as regulating in areas where states traditionally have been sovereign in exercising their police powers. In addition, it conflicts with the only two other circuits to have considered the issue whether a state can be held liable for its school district's hiring and firing actions. The holding in Part I.A also impairs respected values of federalism because the majority applies Title VII to states' good faith exercise of their police powers. For these reasons, I respectfully dissent from that part of the majority opinion.

TASHIMA, Circuit Judge, dissenting:

Among the issues raised by plaintiffs on appeal are the procedural errors ("procedural taint") which "result[ed in] the unreported influence of Dr. Klein." Plaintiffs explain their contention as follows:

We do not know the extent of Dr. Klein's ex parte communications with the court, nor the influence these communications had with the court, nor the extent to which [his] views and unfamiliarity with employment testing standards could have been tempered by the heat of cross-examination. We know none of this because the district court did not require any report of the "advice" he received from Dr. Klein and, most importantly, denied Plaintiffs their right to cross-examine the court's expert.

Appellants' Opening Brief at 51. Because I cannot agree with the majority's cursory and mistaken treatment of the technical advisor issue in Part III, I respectfully dissent.

I.

Courts and jurists have long recognized that the "law should in some way effectively use expert knowledge wherever it will aid in settling disputes." Learned Hand, Historical and Practical Considerations Regarding Expert Testimony, 15 Harv. L.Rev. 40 (1901). "The only question is as to how it can do so best." Id. I agree with the majority that the district court retains the inherent authority to appoint a technical advisor in especially complex cases. Maj. op. at 591. As Justice Brandeis, speaking for the Court, noted more than 80 years ago:

Courts have (at least in the absence of legislation to the contrary) inherent power to provide themselves with appropriate instruments required for the performance of their duties. This power includes authority to appoint persons unconnected with the court to aid judges in the performance of specific judicial duties, as they may arise in the progress of a cause.

Ex Parte Peterson, 253 U.S. 300, 40 S.Ct. 543, 64 L.Ed. 919 (1920) (approving appointment of an auditor and report to be made by him). In exercising that power, however, the district court must observe certain, minimal, procedural safeguards in order to insure that the advisor is unbiased and impartial, and that his participation is properly confined. Here, at the district court's invitation, a question of the advisor's bias and partiality was raised, but not resolved. The record is also completely devoid of any evidence of what role Dr. Klein played in the proceedings, the scope and basis of his influence, what materials or knowledge he may have called on in bringing that influence to bear, and what part his knowledge, expertise, and advice played in the district court's ultimate finding that CBEST is a valid test under Title VII. I would therefore vacate the judgment and remand to the district court for it to make findings on the bias and partiality issue, and to explain Dr. Klein's role, in accordance with the guidelines set forth below.

Even though we have never held explicitly that district courts may appoint technical advisors outside the confines of Federal Rule of Evidence 706, courts and commentators agree that a trial court's inherent authority to appoint a technical advisor has not been displaced by the federal rules. See Reilly v. United States, 863 F.2d 149, 158 (1st Cir. 1988); Notè, Improving Judicial Gatekeeping: Technical Advisors and Scientific Evidence, 110 Harv. L.Rev. 941, 949-50 (1997) (hereinafter " Technical Advisors"); cf. Hall v. Baxter Healthcare Corp., 947 F.Supp. 1387, 1392 n. 8 (D.Or. 1996) (appointing technical advisor under "inherent authority" of Fed.R.Evid. 104). Although the use of a technical advisor should be "the exception and not the rule," expert advice can be quite useful in a trial that involves unusually complex issues in fields "beyond the regular questions of fact and law with which judges must routinely grapple." Reilly, 863 F.2d at 156-57. Moreover, because the district court is in a better position to decide if the appointment of a technical advisor is warranted, we should defer to its discretion with respect to whether or not to appoint one. See id. (reviewing appointment of technical advisor for an abuse of discretion).

The majority correctly concludes that this is one of those "rare cases" in which the appointment of a technical advisor under the inherent authority of the court is justified. Maj. op. at 590; see Reilly, 863 F.2d at 156. As the original panel opinion pointed out, this case "involves the highly technical field of psychometrics, and presents problems of unusual complexity beyond the normal questions . . . with which judges routinely grapple." Association of Mexican-American Educators v. California, 195 F.3d 465, 492-93 (9th Cir. 1999), reh'g en banc granted, 208 F.3d 786 (9th Cir. 2000).

II.

After three years of discovery, the district court found that it needed guidance on a host of complex, factual issues. As the district court noted in its order appointing technical advisor, "the Court comes face to face with many prickly problems requiring expertise in the esoteric fields of education and psychometrics including knowledge of theories about educational measurement and testing, cognitive psychology, statistics, and other fields pertaining to the CBEST and other cases." It then appointed Dr. Stephen P. Klein, a highly-respected test validation expert, as its "Technical Advisor." Although I agree with the majority that the district court did not abuse its discretion in deciding to appoint a technical advisor, that is only the beginning of our inquiry, and not its end, as the majority seems to think.

The order appointing the technical advisor provides that he is to review all of the "direct testimony of the experts heretofore furnished to the Court," briefs, selected pleadings, and depositions. Dr. Klein was also to be present in court during expert testimony, "take such notes as he deems proper and confer ex parte with the Court from time to time." It also noted that "[h]is role will be similar to that of a judicial clerk and, therefore, he will not be available for communication with or questioning by the parties." The order nowhere stated, however, what Dr. Klein was to do with all of this evidence and other information he was directed to absorb, except to "confer ex parte with the Court from time to time."

The use of a technical advisor is not without risks. First, whenever a court appoints a technical advisor, there is a danger that the court will rely too heavily on the expert's advice, thus compromising its role as an independent decisionmaker and the requirement that its findings be based only on evidence in the record. This risk is especially salient if the contents of the communications between the trial judge and the advisor is hidden from the parties (and appellate review), and where the parties have no opportunity to respond to the advisor's statements. See Technical Advisors, 110 Harv. L.Rev. at 957. Second, experts in the relevant field, particularly if it is a narrow and highly-specialized one, may be aligned with one of the parties; therefore, the district court must make every effort to ensure the technical advisor's neutrality, lest the advisor develop into, or give the appearance of being, an advocate for one side. Without some safeguards, the parties' confidence in the fairness of the trial will erode.

Thus, once the district court has appointed a technical advisor, its discretion concerning the advisor's role in the trial should not be unfettered. We have never had occasion to give definitive guidance to district courts on the use of technical advisors and this case clearly demonstrates why we should. Simply put, we are utterly unable, on this record, to review the propriety of Dr. Klein's advice to the district court because we have no idea what role he played in the district court's factfinding process. As indicated above, the district court laid out Dr. Klein's evidence-gathering duties in its appointing order, but that order does not state what Dr. Klein was to do with that store of knowledge. There is no evidence in the record that would provide any hint about the nature, scope, or substance of Dr. Klein's advice. We therefore do not know whether and, if so, to what extent, Dr. Klein may have impermissibly influenced the court's ultimate finding that CBEST is a valid test under Title VII: Did he draw upon his own expertise and knowledge of test validation to discount or contradict the opinion of a testifying expert on either side? Did he explain to the court why CBEST was valid, based either on his own opinion and/or on information (such as the scientific literature in the field, including his own writings) not in the record? Or was he asked only to review the court's proposed findings for technical accuracy? Moreover, plaintiffs have alleged that Dr. Klein was biased in favor of defendants and the district court did little to allay plaintiffs' concerns.

III.

Although we need not require strict adherence to any specific procedures, I would hold that a district court minimally must: (1) utilize a fair and open procedure for appointing a neutral technical advisor; (2) address any allegations of bias, partiality, or lack of qualification; (3) clearly define and limit the technical advisor's duties; (4) make clear to the technical advisor that any advice he gives to the court cannot be based on any extra-record information; and (5) make explicit, either through an expert's report or a record of ex parte communications, the nature and content of the technical advisor's advice. By adopting these minimal safeguards, the parties can be assured that the technical advisor appointment process is fair and that the technical advisor does not exercise undue influence on the district court. These procedures will also enable meaningful appellate review.

A. Neutrality of the Technical Advisor

The district court, of course, should have wide latitude when choosing a technical advisor, who, after all, is retained to aid the court's understanding of the difficult and complex technical issues presented in the case. The parties, however, should be given an opportunity to object to any proposed appointment based on bias, partiality, or lack of qualification. See Reilly, 863 F.2d at 159; Technical Advisors, 110 Harv. L.Rev. at 954-55; cf. Daubert v. Merrell Dow Pharm., Inc., 509 U.S. 579, 592 n. 10, 113 S.Ct. 2786, 125 L.Ed.2d 469 (1993) (noting that, with respect to proffered scientific evidence, the trial judge must determine "the qualification of a person to be a witness") (quoting Fed.R.Evid. 104(a)). While the district court is not required to reject the appointment of a technical advisor merely because one side or the other objects, it must adequately address any concerns of bias, partiality, or lack of qualification that the parties raise.

Here, the district court properly allowed both sides an opportunity to review Dr. Klein's curriculum vitae and to submit written interrogatories to him concerning his experience and background. Plaintiffs objected to Dr. Klein's appointment on the grounds that: (1) he had worked closely with the California Commission on Teacher Credentialing, one of the defendants in this case; (2) his private practice focuses on supporting certification exams; (3) he has taken positions on issues raised in the case; and (4) he had worked closely with one of defendants' key experts, Dr. Mehrens.

We are unable to discern from the record whether any of these allegations are well-founded, but it is clear that the district court did not adequately address these objections. After plaintiffs objected to the appointment, the district court overruled the objection, stating that plaintiffs would have the "opportunity to examine him fully with respect to any views that he may have." This response to the objection might have been adequate, had the district court actually given plaintiffs the promised opportunity. As it turned out, however, the court's response to plaintiffs concerns was wholly inadequate because the district court later decided that it would not have Dr. Klein testify. There is nothing in the record to indicate that the district court seriously considered plaintiffs' objections, or that it took any other measure to insure Dr. Klein's neutrality, such as making its own independent inquiry. This omission is all the more troubling because the objections, if well-founded, might well provide valid bases for disqualifying Dr. Klein from serving as a neutral technical advisor.

To avoid this situation in the future, I would require, absent mutual agreement by the parties on an acceptable technical advisor, that the district court meaningfully address, on the record, any colorable objection to the appointment of a specific technical advisor. Because the district court failed adequately to address plaintiffs' objections to Dr. Klein's appointment, I would remand to the district court for this determination to be made on the record.

B. The Technical Advisor's Role

The role of a technical advisor also should be carefully defined and limited to reduce the risk that the advisor will usurp the role of the court as factfinder, see Technical Advisors, 110 Harv. L.Rev. at 955, or of even giving that appearance. To this end, the district court should lay out the specific duties of the technical advisor, keeping in mind that the advisor's role should be that of "sounding board" and tutor who aids the court in understanding the "jargon and theory" relevant to the technical aspects of the evidence. Reilly, 863 F.2d at 158; see Technical Advisors, 110 Harv. L.Rev. at 955-56. This "job description" should make explicit that the technical advisor is not to contribute evidence (unless called as a witness) or to be an advocate on behalf of either party. Id., 110 Harv. L.Rev. at 955; see Reilly, 863 F.2d at 157. If the parties, the court, and the advisor all are clear on the advisor's role, there will be less likelihood that the advisor will overstep his bounds or that the parties may be led to believe that he has. Moreover, a "job description" that highlights those areas in which the district court needs aid or clarification may help to focus the parties' attention on issues that are important to the court and, therefore, aid the parties in addressing those concerns directly. See Technical Advisors, 110 Harv. L.Rev. at 955.

When the district court appointed Dr. Klein, as noted above, it ordered that he was "to review all the direct testimony of the experts, . . . the parties' briefs and proposed findings of fact and conclusions of law, and other such documents . . . as seems appropriate." It also explained that Dr. Klein should listen to the expert testimony, take notes, and confer with the court ex parte when necessary. The district court, however, failed to complete the list of Dr. Klein's critical duties. It did not describe the nature or scope of its consultations with Dr. Klein, e.g., whether his role was limited to reviewing the court's proposed findings for technical accuracy, whether he was to evaluate the opinions of the testifying experts, or whether he was to give the court his own, independent, expert opinion.

In the future, district courts should be more specific concerning the technical advisor's precise role. For example, the court could explain which technical concepts it finds troubling, and limit the expert's role to helping it comprehend those technical issues. See Hall v. Baxter Healthcare Corp., 947 F.Supp. at 1393-94 (listing specific questions submitted to the technical advisors). The district court may even find it helpful to invite the parties to participate in defining the expert's duties to reduce the risk of perceived unfairness and to help focus the issues. See Technical Advisors, 110 Harv. L.Rev. at 955.

C. Making a Record

Finally, the nature and scope of the advice provided to the district court by the technical advisor should be recorded in some manner. While I would not prescribe a mandatory procedure for making that record, such as a reporter's transcript of all ex parte conferences between the court and the technical advisor, obviously, some documentation is required — such as a report by the advisor, a summary of the advice given, or the court's statement on the record — of the court's interaction with the technical advisor.

The district court, as well as other courts, likened the duties of a technical advisor to those of a law clerk and, thus, reasoned that the communications between the court and advisor may be kept confidential, i.e., concealed from the parties and the appellate court. See Reilly, 863 F.2d at 158. While there are some similarities between a law clerk and a technical advisor, that analogy is limited.

In some important respects, a technical advisor is quite unlike a law clerk. A law clerk's function is to aid the judge in researching legal issues in cases pending before the court. Because the judge is an expert in the law and fully understands legal theory and analyses, it is unlikely, to say the least, that a law clerk will impermissibly usurp the judicial function. On the other hand, a technical advisor is brought in precisely because the judge is not familiar with the complex, technical issues presented in the case. See id. at 157 ("appointment of a technical advisor must arise out of some cognizable judicial need for specialized skills"). There is therefore an understandable concern that the technical advisor's opinion will carry undue weight with the judge. Cf. Daubert, 509 U.S. at 595, 113 S.Ct. 2786 (noting that "[e]xpert evidence can be both powerful and quite misleading because of the difficulty in evaluating it") (internal quotations and citation omitted). In short, a judge can filter out "bad" legal advice or research from a law clerk; he or she is ill-equipped, however, to do the same with "bad" technical advice. Moreover, resolution of legal issues is committed to the judge qua judge and is subject to de novo review. On the other hand, factual issues, no matter how technical, are committed to the factfinder and, to be reviewed properly, must be based on the record made in the trial court.

Whatever method the court employs in making a record of the technical advisor's advice and counsel, there is no reason why a neutral technical advisor's advice should be shrouded in absolute secrecy. Concealing the nature of that expert advice can only erode confidence in the court's role as a neutral and independent decisionmaker.

IV.

These procedural safeguards are not especially burdensome and should not be difficult to apply. They are, however, essential in order to insure that the parties have confidence that the technical advisor is unbiased, impartial, and qualified, and that he has not unduly influenced the proceedings. Such minimal procedural safeguards are also required so that any challenge to the appointment of a technical advisor or to the degree of his participation in the case can meaningfully be reviewed on appeal. Here, other than a short order appointing Dr. Klein, the district court provided no explanation of his role. It also failed adequately to address the parties' written objections to Dr. Klein's neutrality.

For the foregoing reasons, I would vacate the judgment and remand this case to the district court for its resolution of these issues and for such further proceedings as may thereafter be necessary.