For decades, orchestras have used “blind auditions” in an effort to remove bias from the selection process. The idea was that if the performer was seen, the selection committee may bring personal bias into whether that person was going to get a job. For example, subtle or overt bias against women or minorities might factor into whether a person auditioning was selected.
The success of blind auditions in increasing opportunities for women was allegedly confirmed in a 2000 study by Harvard economist Claudia Goldin and Princeton economist Cecilia Rouse. The economists set out to determine whether women’s success in joining orchestras was related to the adoption of blind auditions.
But as American Enterprise Scholar Christina Hoff Sommers wrote in The Wall Street Journal and explained in her “Factual Feminist” series, the data collected by these two economists “were inconclusive” and their paper “includes multiple warnings about small sample sizes, contradictory results and failures to pass standard tests of statistical significance.”
“Even though our sample size is large, we identify the coefficients of interest from a much smaller sample,” one such warning stated. “Some of our coefficients of interest, therefore, do no pass standard tests of statistical significance and there is, in addition, one persistent result that goes in the opposite direction.”
Sommers said many readers and media outlets appeared to ignore these warnings, instead focusing on statements made in the final paragraph of the study, which stated, “we find that the screen increases – by 50 percent – the probability that a woman will be advanced from certain preliminary rounds and increases by severalfold the likelihood that a woman will be selected in the final round.”
This conclusion was cited in approximately 1,500 scholarly articles and countless media reports. The Independent Women’s Forum pointed out that it was cited by Supreme Court Justice Ruth Bader Ginsburg in one of her dissents and praised by Malcolm Gladwell.
“The research went uncriticized for nearly two decades,” Sommers said in her “Factual Feminist” video about the study. “That changed recently when a few scholars and data scientists went back and read the whole study.”
These scholars found a number of “contradictory findings,” Sommers said.
“The screens seemed to help women in preliminary audition rounds but men in semifinal rounds. None of the findings were strong enough to draw broad conclusions one way or the other,” Sommers wrote in the Journal.
In addition, the raw tabulations “showed women doing worse behind the scenes,” according to Sommers.
“But perhaps, Ms. Goldin and Ms. Rouse explained, blind auditions ‘lowered the average quality of female auditionees,’” Sommers wrote. “To control for ability, they analyzed a small subset of candidates who took part in both blind and nonblind auditions in three of the eight orchestras.”
Sommers suggested that auditioning behind screens had little to do with women’s advancement in the orchestras, but rather a change in attitudes during the same period the screens were being used.
“Truth matters,” Sommers concluded. “Overhyped claims create confusion and undermine public trust – and they don’t solve problems. Sex discrimination in the workplace is a serious matter, but improvements require solid data, replicable research, and careful evaluations of causation.”