A famous 2000 article praised by Malcolm Gladwell and cited in a dissent by Justice Ruth Bader Ginsburg seemed to find that when orchestras used blind auditions, with the musician performing behind a screen, the number of women hired went up.

Many orchestras became more democratic in the 1960s and ‘70s, and quite a few began to use blind auditions to promote this trend during this time. Economists Claudia Goldin of Harvard and Cecilia Rouse of Princeton observed that the number of female musicians hired by orchestras was growing. Could blind auditions be the reason?

Goldin and Rouse produced a study “Orchestrating Impartiality: The Impact of ‘Blind’ Auditions on Female Musicians,” published in the American Economic Review, that answered in the affirmative. The study was a hit, widely cited and quoted. 

Christina Hoff Sommers takes a look at new evaluations of the study in this morning’s Wall Street Journal.

As Christina Hoff Sommers points out, acceptance of the study should have been more tentative:

[Goldin and Rouse] collected four decades of data from eight leading American orchestras. But the data were inconclusive: The paper includes multiple warnings about small sample sizes, contradictory results and failures to pass standard tests of statistical significance.

But few readers seem to have noticed. What caught everyone’s attention was a big claim in the final paragraph: “We find that the screen increases—by 50 percent—the probability that a woman will be advanced from certain preliminary rounds and increases by severalfold the likelihood that a woman will be selected in the final round.”

. . .

The study’s appeal is clear: Two prominent economists, in a top journal, wielding state-of-the-art econometrics, captured and quantified bias against women and documented a solution. Or so it seemed.

The study was unchallenged for almost twenty years. However, a few scholars have begun to re-evaluate the findings, according to Hoff Sommers. When they ran the numbers again, surprises emerged:

The first thing they noticed is that the raw tabulations showed women doing worse behind the screens. But perhaps, Ms. Goldin and Ms. Rouse explained, blind auditions “lowered the average quality of female auditionees.” To control for ability, they analyzed a small subset of candidates who took part in both blind and nonblind auditions in three of the eight orchestras.

The result was a tangle of ambiguous, contradictory trends. The screens seemed to help women in preliminary audition rounds but men in semifinal rounds. None of the findings were strong enough to draw broad conclusions one way or the other.

Hoff Sommers suggests that the two economists thought they had detected “something with real-world relevance despite an absence of statistical rigor.” Instead of calling for more research, they declared blind auditions helpful to women musicians.” One subsequent study, however, which was notable for academic rigor, found just the opposite of the Goldin Rouse study:

In 2017 a team of behavioral economists in the Australian government published the results of a large, randomized controlled study entitled “Going Blind to See More Clearly.” It was directly inspired by the blind-audition study. Iris Bohnet, a Harvard Kennedy School dean and Goldin-Rouse enthusiast, served as an adviser.

For the study, more than 2,000 managers in the Australian Public Service were asked to select recruits from randomly assigned résumés—some disguising the applicant’s sex, others not. The research team fully expected to find far more female candidates shortlisted when sex was disguised.

But, as the stunned team leader told the local media: “We found the opposite, that de-identifying candidates reduced the likelihood of women being selected for the shortlist.” It turned out that many senior managers, aware that sexist assumptions had once kept women out of upper-level positions, already practiced a mild form of affirmative action. 

Orchestra hiring is a fairly arcane matter, but the use of flawed statistics to promote the idea of rampant sexism should be of interest to all of us.