A Portrait of the 2014 Philosophical Gourmet Report by the Numbers

 

In a recent post I asked why Professor Leiter had decided to replace reputational rankings with impact studies in his law school rankings, while sticking with reputational surveys in philosophy. In response Professor Leiter made the following claim:

The other big difference between academic law and academic philosophy is that in the former there is far less consensus on scholarly paradigms than in the latter.

More than few people in philosophy reacted with amazement to the notion that there is far less consensus in law than there is in philosophy. In order to explain his claim, Leiter posted an addendum:

As I pointed out to [a colleague], and is perhaps worth sharing, Kieran Healy’s research found remarkable convergence in overall evaluations of programs across almost all areas of philosophical specialization–that’s the evidence about consensus I had in mind [evidence not refuted by noting that at the margins there are dissenters from the consensus, obviously].  {Emphasis added, square brackets in original.}

This is a false statement. Professor Healy did not find a “remarkable convergence…across almost all areas of philosophical specialization.” In a post on Leiter’s blog comparing “the level of consensus or disagreement between specialists” regarding the overall rankings, Healy discovered that there was “a relatively high degree of consensus around the top seven or eight departments,” out of ninety-nine departments, but varying degrees of disagreement for other ranked schools, with those at the bottom showing more consensus than those in the middle. (Gregory Wheeler has argued that Healy’s results show that, except for the top six schools and a handful at the bottom, “rankings vary quite a lot”*.)

Leiter’s adamance about the “remarkable convergence” is no small matter.  He insists that there is a consensus and that the evidence for the consensus is the PGR. The criticism of colleagues, some with considerable training in statistics and survey methodology, over many years has not budged him. In his most recent remark it is clear that he’s doubling down. He cannot or will not see the circularity of his position. The PGR is philosophy and philosophy is the PGR. QED.

Using the data of the 2014 PGR I show here that the consensus Leiter insists on is artificial.  But make no mistake: this is not just about the PGR. It is about whether our vision of philosophy is like Raphael’s “The School of Athens,” in which there is room for multiple and widely different ways of doing philosophy, or whether a particular style, method, and set of concerns should crowd out current and future philosophical diversity.

There are too many problems with the current PGR to address in one post. This is an overview of data that call into question assumptions about the PGR’s reliability and its ostensible consensus. While I focus on the specializations here, note that the problems with this evaluator pool bear on the overall rankings because the pools are so similar.  For the sake of clarity and interest, I proceed by presenting a series of issues, asking the reader to decide–without any obligation to rank them–which are worse.

Which is worse?

• That the current PGR lumped together the 2011 and 2014 rankings of Feminist Philosophy, unheard of in any other specialization, because there were too few evaluators this year, and still only had 12 evaluators?

• Or that in 32 specializations–leaving Feminist Philosophy aside because we don’t know how many or who ranked in 2014–there are only 32 women?  (That’s correct, 32 women philosophers for all 32 specializations. It may seem like more because the same evaluators often rank in multiple areas.)

• Or that in the Philosophy of Language, Philosophy of Mind, Metaphysics, Epistemology, and Philosophical Logic, there are only 11 different women evaluators?

• Or that in the 27 other specializations not listed immediately above, there are a total of 21 women?

• Or that 75% of all specializations have three or fewer women philosophers evaluating, including those who evaluate multiple times? (Or that 44% have 0 or 1, or that 63% have 0, 1 or 2.?)

• Or that none of this may be a problem for Professor Leiter because, based on his experience with the PGR pool of evaluators, “There is no evidence we have ever seen that the gender of the evaluator affects the evaluations?”

Which is worse?

• That 72% of the specializations have 20 or fewer evaluators, up from approximately 60% of the specializations for the 2011 PGR?

• Or that 38% have 10 or fewer?

• Or that 28% have 8 or fewer?

• Or that 8 out of 9 areas, 89%, in the History of Philosophy have 20 or fewer evaluators, up from 6 out of 9 for 2011?

• Or that, as troubling as these figures are regarding the small number of evaluators in so many areas, they only tell part of the story, because evaluators frequently rank multiple times? (So there are actually fewer different individuals doing the ranking in these specializations. See next.)

Which is worse? (Keeping in mind Leiter’s claim that there is a “remarkable convergence in overall evaluations of programs across almost all areas of philosophical specialization”.)

• That 41% of the Philosophy of Language evaluators were also Philosophy of Mind evaluators (11/27)?

• Or that 69% of the Philosophy of Law evaluators were also evaluators for Political Philosophy (11/16)?

• Or that 59% of Political Philosophy evaluators were also evaluators for Ethics (20/34)?

• Or that 79% of Cognitive Science evaluators were also evaluators for the Philosophy of Mind (22/28)?

• Or that 37% of Metaphysics evaluators were also Philosophy of Mind evaluators (11/30)?

• Or that 63% of Metaethics evaluators were also Ethics evaluators (10/16)?

• Or that 85% of the 20^th Century Continental evaluators were also Kant or 19^th Century evaluators (11/13)?

• Or that 46% of 20th Century Continental evaluators were also Kant evaluators (6/13)?

• Or that 67% of those who ranked in Medieval also ranked in Philosophy of Religion (4/6)?

• Or trumpeting the convergence between the specialty rankings and the overall rankings of the PGR without mentioning overlaps like these in the 2014 PGR?

Which is worse? (Bearing in mind that in four specializations, including Feminist Philosophy, the PGR recognizes that adjustments should be made for the small numbers of evaluators.**  Remarkably, none of the departments below fall into this category, even though some have few evaluators.)

• That only 6 evaluators in the Philosophy of Mathematics ranked 41 departments, while 39 evaluators in Ethics ranked 42 departments?

• Or that 7 evaluators in the History of Analytic Philosophy ranked 42 departments, while 42 evaluators in Philosophy of Mind ranked 37?

• Or that 10 evaluators in Applied Ethics ranked 59 departments, while 41 evaluators in Epistemology ranked 37?

• Or that 12 evaluators in Mathematical Logic ranked 43 departments, while 30 evaluators in Metaphysics ranked 37 departments?

Which is worse?

• That the Philosophy of Religion has only 2 evaluators currently at Catholic universities—the same one, St. Louis University—effectively excommunicating a large number of philosophers who work in the Philosophy of Religion?

• Or that in the Philosophy of Law only 7 of its 16 evaluators, 44%, are at US institutions, for a rankings report in which the vast majority of ranked programs are located in the US, and strangely, which is spearheaded by Brian Leiter, a law and philosophy professor at a US university?

• Or that American Pragmatism, a specialization whose experts have ties to both analytic and continental philosophy, is only represented by 3 evaluators, 2/3rds of whom are not at universities in the United States?***

• Or that the Philosophy of Race has only three evaluators?

• Or that Chinese Philosophy only has four?

• Or that there isn’t any other non-Western philosophy represented, for example, Japanese philosophy, or Indian philosophy?

• Or that there isn’t a separate category for Latin American Philosophy?

• Or that Feminist Philosophy didn’t have enough evaluators this year so that 2011 and 2014 were combined?  (Wait . . . this is where we came in.)

Enough.  In my next post I will establish that there are evaluators who are not leading experts or experts at all in the specializations they are ranking. (I will not embarrass individual evaluators, or list names, etc.)  I will also show that there are imbalances in the evaluation of specializations, for example, that eight of the experts in 19^th Continental Philosophy are Nietzsche scholars, while other major figures are hardly represented–Kierkegaard gets one expert.  There is also the issue of whether there are more highly qualified people who have been overlooked or dismissed because they do not fit the confines of The Consensus.

I began this post by raising concerns about Brian Leiter’s lack of appreciation for philosophy’s diversity and about the PGR’s aspiration to mold and hold the profession to Leiter’s vision of it. However, his assertion about consensus is only one piece of the puzzle over why Leiter continues to argue so vehemently for the PGR, often attacking those who disagree with him–personally and sometimes brutally–on his blog and elsewhere. I have wondered whether there was something in his philosophical outlook, beyond mere personal idiosyncrasy, that leads him to conduct himself like this, and which shapes both the PGR and his defense of it.  And then I happened across a quotation apparently very important to Leiter; a passage from Nietzsche that, he makes clear, he has quoted more than once. The quotation appears in his essay “How to Rank Law Schools.” ****  The quotation and its context occur near the very end of the article.  He introduces the quotation with these words:

 Academic rankings that provide actual information on matters of educational value have a useful role to play for students, quite obviously, but they also have a constructive role to play for faculty. Professor Korobkin suggests that in ranking schools we want to discourage “status competition.” I guess my own view is more Nietzschean, and so let me close with a quote I have used before. This is Nietzsche from his early essay on “Homer’s Contest”:

 [Then Leiter quotes Nietzsche:]

[J]ealousy, hatred, and envy, spurs [sic] men to activity: not to the activity of fights of annihilation but to the activity of fights which are contests. The Greek is envious, and he does not consider this quality a blemish but the gift of a beneficient [sic] godhead . . . . The greater and more sublime a Greek is, the brighter the flame of ambition that flares out of him, consuming everybody who runs on the same course.

. . . .

Every talent must unfold itself in fighting: that is the command of Hellenic popular pedagogy, whereas modern educators dread nothing more than the unleashing of so-called ambition . . . . And just as the youths were educated through contests, their educators were also engaged in contests with each other. [All ellipses in original.]

Whatever one may think of the sentiment–or its employment in this way, or Leiter’s evident attraction to it–its invocation in a discussion of the right way to rank academic programs should give us pause.  “The greater and more sublime a Greek is, the brighter the flame of ambition that flares out of him, consuming everybody who runs on the same course.”  For people who see philosophy as a contest and vanquished opponents as fit for nothing but the flames, well, Leiter and his PGR are the way to go.   But in “The School of Athens,” all we’d see is that this guy can really empty a room.

 

__________

I compiled all of the statistics in this post.   Of course I may have made errors.  But I do believe that any errors would be minor and not undermine the basic points or patterns under discussion.  There are too many similar results.  If a reader finds any errors, please notify me.

___________

* Commenting on Healy’s work, “Ratings and Specialties,” on Leiter’s blog, Wheeler says, “Those rankings were then aggregated to see how much variation there is across the specialties, which gives a sense of  how much (or how little) consensus there is across specialties. The box and whisker plots (top 25: .png, .pdf; total population: .png, .pdf) give a picture of this.  Except for the top 6 departments, and a handfull rounding out the bottom, the answer is that the rankings vary quite a lot:  people vote according to whom they recognize, and invariably those are the people working in their area(s) of specialization(s).” Gregory Wheeler, “Manufactured Assent: The Philosophical Gourmet Report’s Sampling Problem,” in Choice and Inference.

**“Due to the small number of evaluators, we are not printing the rounded mean scores, but just a list of programs, broken into two groups based on the scores received.”

*** It’s not as if there isn’t a wealth of experts to draw on in the US for American Pragmatism.  Perhaps Leiter should consider contacting the Society for the Advancement of American Philosophy to get some names, which has hundreds of members.  Oh, wait, this is one of the organizations that is outside of the consensus.

**** “Commentary: How to Rank Law Schools,” Indiana Law Journal, Vol. 81, 2006