A Note on “A User’s Guide To Philosophy Without Rankings”

owlofminerva   Readers of UP@NIGHT may be interested in a site I created to address the rankings controversy in philosophy, specifically the Philosophical Gourmet Report, and to offer resources to prospective graduate students that do not depend on rankings:  “A User’s Guide To Philosophy Without Rankings.”   There is a good deal of information and data available.  And as always, regarding my own posts, I ask readers to alert me to any errors, factual or inferential.  My goal at UP@NIGHT  and with the “User’s Guide” is to provide the most reliable information available.  I invite corrections and comments.

I know that people feel passionately about the role of rankings in philosophy.   I ask that comments on this site or at the “User’s Guide” be directed  to content and avoid personal attacks and ad hominems.

upnight.com

NYC looking North from the EMPIRE STATE BUILDING. Photo (proportions altered) by DAVID ILIFF, license HERE.

UP@NIGHT

UP@NIGHT

UP@NIGHT

 

dsc_0867-3

……………

Early to bed, and early to rise,

Makes a man healthy, wealthy and wise

– Benjamin Franklin.

I don’t see it.

– George Washington

Now both of these are high authorities – very high and respectable authorities – but I am with General Washington first, last, and all the time on this proposition.

Because I don’t see it, either. . . .

Put no trust in the benefits to accrue from early rising, as set forth by the infatuated Franklin – but stake the last cent of your substance on the judgment of old George Washington, the Father of his Country, who said “he couldn’t see it.”

And you hear me endorsing that sentiment.  

Mark Twain, “Early Rising, As Regards Excursions to the Cliff House,” MARK TWAIN IN THE GOLDEN ERA 1863-1866.

……………

Brian Leiter’s Continuing Influence on the Philosophical Gourmet Report: The Past as Future

615

THE PHILOSOPHICAL GOURMET REPORT, JUNE EDITION, 1995-1996   A Ranking of U.S. Graduate Programs in Analytic Philosophy, by Brian Leiter*

Before the PGR arrived on the web, it was circulated in photocopies.  The link above will take you to a PDF of the 1995-1996 edition, which was entirely the product of one person, Brian Leiter.  Much would seem to have changed since this edition: there is now an Advisory Board and evaluators.  But much is the same.  What kind–and how much–influence will Brian Leiter have on the PGR once he’s stepped aside as co-editor?  Odds are his influence will be significant.  I’m going raise some pertinent observations and questions that suggest why this is likely, in part by looking back to the PGR of 1995-1996.

In my reading of the situation, Professor Leiter managed to negotiate a pretty sweet deal for himself this fall with the Advisory Board:  he will remain on the Board–presumably, first among equals as founder and the man with the knowledge of how the PGR operates.  Professor Brogaard will be left with the arduous and time-consuming task of actually gathering and presenting the data for the PGR, while Leiter will continue posting material that is functionally and historically tied to the PGR on his blog, as he now does, maintaining his influence over the PGR and the profession.  N.b., Leiter holds the copyright to the PGR.  Members of the Board, or any c0-editors, who try to institute real changes to the PGR might someday face real opposition from Leiter as the PGR’s owner.  What that might look like–well, I will leave this to the reader’s (and his or her lawyer’s) imagination.**

  • The 1995-1996 PGR presents the rankings in a few initial pages and then devotes considerable space to Professor Leiter’s lists of noteworthy faculty moves, retirement age proximity, tenurings, job offers, new PhDs, etc. Notice that in its earlier incarnation–before there was an Advisory Board, or any evaluators, other than Brian himself–the PGR was not independent from Leiter’s VIP tracking activities, which now occur on his blog.   Leiter’s efforts on the blog suggest that he believes that keeping abreast of these items is important for PGR evaluators.  As a matter of fact, there is currently a link on the home page of the  2014-2015 PGR to Leiter’s blog,  “Updates on faculty movements in philosophy.”   There isn’t merely a historical or hypothetical connection between the blog and the PGR: it’s current and explicit.  Presumably the link to the blog will stay on the 2014-2015 PGR site for at least the next two or three years, that is, as long as the 2014-2015 is the latest version of the PGR.  One of the things we don’t know, because these arrangements are closely held by Leiter and his Advisory Board members, as well as his new co-editor, is how, if at all, they decided to handle this relationship between the PGR and the news items on Leiter’s blog.  The continuing presence of the blog link on the 2014-2015 PGR site suggests that this issue either wasn’t addressed in the agreement, or, if it was, that Leiter succeeded in negotiating a continuation of the relationship of the PGR to his blog.  So in addition to his presence on the Advisory Board, he will be the PGR’s blogger, at the very least regarding the movement of faculty if not much more.
  • Professor Leiter had not yet defended his dissertation in philosophy when he produced the 1995-1996 PGR, which was in part based on “conversation” (see p. 4, 1995-1996 PGR). Nevertheless, we find claims about the discipline in the 1995-1996 PGR from which Professor Leiter hasn’t budged in twenty years, for example,  “[a]s it turns out, the best scholarly work on Continental philosophy is generally done at the predominantly analytic departments” (p. 23).  Note that Leiter had been at Princeton as an undergraduate and did his graduate work at Michigan, two of the top analytic programs according to the 1995-1996 PGR, which Leiter called a ranking of analytic graduate programs.  (Socrates to Anytos in the Meno: “Astonishing! Then how could you know anything about this matter, whether there is anything good or bad in it, if you are quite without experience of it?” [Rouse, transl.])  Can we expect any innovations in the PGR’s assumptions about what counts as good philosophy? In the current 2014-2015 PGR, now with a new co-editor, we find the following remarks:

It is also because analytic philosophy remains very much a specialty that it is possible to rank departments: the standards of success and accomplishment are relatively clear, maintained as they are by a large, dedicated scholarly community.

Indeed, it is fair to say that what gets called “analytic” philosophy is the philosophical movement most continuous with the “grand” tradition in philosophy, the tradition of Aristotle and Descartes and Hume and Kant.  Only analytic philosophers aspire to the level of argumentative sophistication and philosophical depth that marks the great philosophers—even as analytic philosophers typically fail to achieve the grand visions, the “ways of seeing” of the great historical figures.

So, not only do non-analytic philosophers fail to aspire to the “argumentative sophistication and philosophical depth that marks the great philosophers,” it would seem that it is the set of standards peculiar to analytic philosophy that makes rankings possible.  This does not bode well for the PGR’s treatment of other approaches or traditions, including all of non-Western philosophy.  (This, notwithstanding the hand-waving in posts like this one.)

  • Leiter’s descriptions of the differences among the top thirty schools in the 1995-1996 PGR do little to distinguish them.  (I say “thirty” because of the way that Leiter divided the 1995-1996 rankings.  See the “Report” linked above.)  Further, the players among the top thirty schools haven’t changed much in almost twenty years.  Specifically, 25 of the 30 schools in the top 30 in 1995 are still in the top 30 in 2014, and two of the other five are tied at 31.  Perhaps this is because Leiter picked the majority of the top 30 in 1995-1996 correctly.  Or perhaps it is because the PGR’s deep methodological flaws and Leiter’s faulty, or at least debatable, assumptions about philosophy have sustained a convergence of results from one edition to the next.  Or perhaps the PGR has gotten it mostly right by chance.  We don’t know, and that’s the point.  But I haven’t heard any commitment from the Advisory Board–or the new co-editor–to engage a team of independent survey experts to address these questions once and for all.  Without this, I don’t see how there will be any change in grounds for confidence in Leiter’s methodology.
  • The document is explicit about what it thinks the core areas in philosophy are: “‘Core’=Metaphysics, Epistemology, Philosophy of Language & Mind (p. 2).”  (Sorry Socrates, Ethics isn’t a Core area in philosophy.)  These areas are still given pride of place in the PGR’s rankings (notwithstanding the later addition of Ethics and other areas further down), and a large proportion of the evaluators rank in these areas.  (Professor Brogaard, the new co-editor of the PGR, lists the following areas as specializations on her CV: Mind, Language, Epistemology, Philosophical Psychology, Cognitive Neuroscience.)

IMPORTANT: this focus on observations about analytic philosophy and the PGR is in no way intended to pigeonhole or belittle the analytic tradition itself.  What is objectionable here is the use of a certain conception of the analytic tradition to parochialize philosophy as a whole.  Every tradition in philosophy has what we might call its ideologues.  I see no philosophically acceptable reason to encourage ideologies to dominate our profession.  Regarding the last point about the core areas, I am not at all suggesting that these aren’t important areas in philosophy.  It’s the balance I worry about.  In 2014-2015 PGR there were 140 evaluations of departments in these areas.  There were three in American Pragmatism, three in the Philosophy of Race, four in Chinese Philosophy–no other non-Western and no Latin American–and thirteen in all of 20th Century Continental.   And the Report didn’t have enough evaluators to do a separate ranking of Feminist Philosophy this year. (In the 2011 PGR, before any of the recent issues surfaced, there were only eleven evaluators in this area.)

Check out the 1995-1996 PGR.   Enjoy the ride down memory lane.  Or if you are too young to remember, enjoy a virtual blast from the past.  Also, you might do a thought experiment: try to imagine how one can move from the current PGR to something non-trivially different, given Professor Leiter’s past and future influence.

_____________

*The “Report” can be found in full at the link, and is shared under the terms of the copyright declaration from the bottom of p. 28:  “”The Philosophical Gourmet Report” Copyright 1995 by Brian Leiter.  The Report may be copied and distributed without written permission only if distribution is free of charge and the content of this Report, including its name and authorship, is in no way altered, omitted or deleted.”  The light markings on p. 23 are incidental to and in no way alter the content of the “Report.”

**From Leiter Reports c. 12/29/14: “UPDATE:  ON THE CULTURE GAP BETWEEN PHILOSOPHY PROFESSORS AND LAWYERS:      ( . . . ) I have realized how my many years as a philosopher in a law school have left me out of touch with the sensibilities of those making careers in philosophy departments.  In my world, lawyers are not scary, talk of defamation is not scary, talk of lawsuits is not scary:  all this is just part of the civilized infrascture [sic] of modern societies in which possible wrongdoing can be subjected to formal procedures for adjudicating the merits.” [Recursive link in original.]

The Philosophical Gourmet Report, by the Experts

Shipwreck-watching    “[T]here isn’t any fact in the world that can prove or disprove the quality of particular philosophical work.  All there is in philosophy is the opinion of experts.  Research universities–in their hiring and tenure decisions–are based on the premise that the opinion of experts is what matters.   We have nothing else to go on.”–Brian Leiter*

As I have established in series of recent posts, the problems with the Philosophical Gourmet Report are legion.  These flaws have nothing whatever to do with the controversy surrounding its editor.  They are native to the Report and are symptoms of its poor methodology, in particular, its use of snowball sampling, its faulty assumptions about consensus in the profession, its use of pools of evaluators with too-similar professional backgrounds, its dearth of women evaluators, its small to very small pools of evaluators for the specializations, its marginalization of many specializations, its favoring of sub-specializations within certain specializations, etc.  Problems with PGR have been masked to a degree by statements that Professor Leiter has made to defend it, which often involve exaggerated or misleading claims about its virtues, for example, his recent claim that there is a “remarkable convergence in overall evaluations of programs across almost all areas of philosophical specialization.”  There isn’t, as we saw in the recent post, “The 2014 Philosophical Gourmet Report by the Numbers.”

In the “By the Numbers” post, I promised to address the fact that there are evaluators in specializations who are not experts or specialists in these areas.  Leiter has insisted time and again that his evaluators are experts and experts in their specializations.  But before we get there we need some context for this issue.   The pools of evaluators for the PGR are relatively small, and in many cases very small.  Here is a breakdown for the 2014 PGR, excluding Feminist Philosophy:**

  • 72% of the specializations have 20 or fewer evaluators.
  • 38% have 10 or fewer.
  • 28% have 8 or fewer.

Given how few people are involved, it would seem that at minimum every voice should be that of an expert.  And given how hard Professor Leiter has sold the accomplishments of his evaluators, I think it is fair to expect specialists to be currently active in the areas they are evaluating, because this is, after all, a ranking, and one would need to be as up-to-date as possible about the field in question.

On the 2014 Philosophical Gourmet Report’s website, the first line under “Breakdown of Programs by Specialties,” Leiter states flatly that the specializations (or specialties) use experts, in a statement that has been on the site for years.

The rankings of programs in the specialty areas are based on surveys by experts in those specialties (emphasis in the original).

The language here is important.  He is saying that the evaluators are experts in the specializations (or specialties).  Bear with me here: this may seem obvious, but it is crucial to avoid any confusion about this.  It is clear that this has been and still is Leiter’s public position.  We can see it, for example, in a discussion of this year’s rankings in physics:

98% of undergraduates and their advisors wouldn’t know to recommend these programs as top choices in philosophy of physics–how could they, but for the PGR?  They’d recommend Oxford, of course, which happens to be excellent in this area, but also Harvard, Yale, MIT, Stanford, which are not.  But the PGR makes available to students everywhere the opinion of leading experts.  It makes available the judgment of philosophers like Princeton’s Hans Halvorson (who teaches at another top ten philosophy of physics program, as it happens), and Huw Price (the Bertrand Russell Professor of Philosophy at Cambridge [another top 10 program in philosophy of physics] and a Fellow of the British Academy), and Jill North (a leading young philosopher of physics at Cornell University), and Lawrence Sklar (Michigan Professor, former John Locke Lecturer at Oxford and Fellow of the American Academy of Arts & Sciences), and David Wallace (leading young philosopher of physics at Oxford), among others.  (Remember:  evaluators can’t evaluate their own departments.) (Emphasis added.)

It’s slightly crazy that some people think making these assessments available to everyone somehow “harms” students and the profession.  But as we’ve seen, some “philosophers” are slightly crazy.

First, there is no question that Leiter is referring here to experts in the philosophy of physics.  In addition, he is making a case for using the PGR by saying, in effect, that this is what we do in the PGR, we use experts, in fact, leading experts.  But perhaps only some of the evaluators in the philosophy of physics are experts, leaving him some wiggle room.  However, the last phrase, “among others,” implies that everyone else evaluating in this specialization is an expert.  (N.b.: only ten people evaluated in philosophy of physics in 2014 and he lists five of them before resorting to the phrase “among others.”)  Second, the ranking in philosophy of physics, with its accompanying plug for the merits of the PGR, was the first specialization that Leiter posted on his blog this year, the first in a series of early releases in advance of the publication of the 2014 PGR on December 8th.  At the time I was puzzled about why he released the philosophy of physics ranking first.  But I now have a hypothesis: it was his strongest case, or one of his strongest cases, for advertising the discipline-specific expertise of every evaluator in the specializations, and, therefore, the value of the PGR, which would have been important for Leiter at the time, given the attrition in the ranks of evaluators for the 2014 PGR and the other events of the fall.

Be that as it may, this use of real experts in the philosophy of physics is not something we see consistently in the other specializations of the PGR.   There are evaluators who are clearly not experts in particular specializations, and by expert I mean nothing arcane or excessively demanding, merely, someone who specializes (claims an AOS, not only an interest or an AOC) in an area and publishes regularly, repeatedly, and, at minimum, recently in that area.  (The “recently” is important here since these folks are ranking departments in the present.)  Whether we are dealing with real experts is no doubt important to know.  If this were not the case, it would make the rankings in different specializations less reliable.  In addition, inconsistencies in the level of expertise between and among different specializations would be a serious methodological problem for the PGR, and God knows it can’t afford any more.

However, I was left with something of a dilemma regarding how to proceed.  I would not willingly single out any individual in public for participating in the PGR.  I did not want to name names or embarrass anyone.  I thought that I might work around this problem by describing the evaluators’ specializations in their own words, and saying something about their publication records.  But of course in the age of Google, people would be able to figure out who everyone is very quickly.  I discussed this dilemma with colleagues and there didn’t appear to be any clear solution.  I decided to hold off and try to find another way.

As it turns out, Professor Leiter’s avidity for public defense of the PGR has relieved me of having to establish that there are evaluators who aren’t experts in certain specializations, because Professor Leiter himself declared three years ago that the PGR’s evaluators are often not experts in the specializations.  Yes, really!  He announced this in an on-line exchange during a discussion of French philosophy and the 2011 PGR.  John Protevi had written a post on NewAPPS discussing evaluators in 20th Century Continental philosophy.  Leiter, as is his wont, joined the discussion in the comments section.  I begin with Protevi’s remarks in the original post, discussing whether any of the evaluators specialize in 20th Century French philosophy.  (I have deleted comments in whole or part that did not pertain to this discussion.  The title of the original post was  “A brief look at 2011 PGR 20th-Century Continental Philosophy evaluators”.) Here is Protevi:

These are however only mentions of secondary interests; a quick, non-systematic, but I believe accurate survey of their posted CVs show that none have consistent publishing records on these figures – at most an article here and there. Such consistent publishing is I think a good indicator of those cognizant of the relevant contemporary secondary literature – that is, that produced by people actively working in contemporary French philosophy (emphasis added).

Brian said…

Your “quick” survey of the CVs isn’t very accurate, and I invite you to do a more careful one. And you might try this same exercise for any of the other specialties; most evaluators are asked to evaluate more than one area, and inevitably that means they evaluate areas in which they don’t necessarily work primarily. The one point I do agree with you on is that none of these people are interested in Irigaray, Kristeva, Badiou et al. (some are interested in Deleuze). But I think anyone interested mainly in those figures should probably not be going to a philosophy department anyway for a PhD (emphasis added).

Okay, pause, full stop.  Yes, Leiter just said, “most evaluators are asked to evaluate more than one area, and inevitably that means they evaluate areas in which they don’t necessarily work primarily.”   One of the observations I made in “By the Numbers” was that people often evaluated multiple times.  You can see it in the proportion of evaluators who rank in multiple areas.  So this is not news.**  But it is news when Leiter claims that he in fact asks evaluators to do this and the result is that “most evaluators are asked to evaluate areas in which they don’t necessarily work primarily.”  At the time, Protevi was surprised by this admission:

John Protevi said in reply to Brian…

A systematic review of what’s web-available turned up 7 articles by 5 evaluators on French figures. So “an article here and there” seems an apt way to describe the research of 22 of the 24 evaluators.

On your other point, I was surprised to hear that in other parts of the PGR people evaluate areas outside their primary research (emphasis added).

Brian said in reply to John Protevi

John, you can’t really believe that unless one does “primary research” on 20th-century Continental philosophy that one can’t, therefore have a reasonably informed opinion about where to work on 20th-century Continental philosophy. In any case, that’s a side issue–the evaluators here have rather obviously unimpeachable credentials as scholars, senior and junior, in the Continental traditions in philosophy. If someone really doesn’t care what Pierre Keller, Michael Rosen, Beatrice Han-Pile think, then Godspeed to them! (emphasis added).

Okay, another stop.  So–now we have moved from “in which they don’t necessarily work primarily” to a “reasonably informed opinion.”  This is a far cry from the claim on the PGR’s web site, above, or the way Leiter characterizes the philosophy of physics evaluators, again, leaving the impression in that case that this is what we do in the PGR, we use experts.  (Oh, and by the way, not just ordinary, everyday experts, but those with named chairs.  At least this is what Leiter says when we he is trying to make the case for the PGR and calling those who can’t see it “slightly crazy.”)   Notice that Brian also tries to change the focus of the discussion by saying that “this is a side issue.”  He wants to get back to bragging about the general excellence of his evaluators in the Continental traditions in philosophy.  But that wasn’t the question.  The issue is whether certain evaluators are in fact experts in the area.  That someone publishes on Kierkegaard and German Idealism doesn’t make him or her an expert in 20th Century Continental philosophy, no matter how excellent their work is in these other areas.  Protevi picks right up on this:

John Protevi said in reply to Brian…

Brian, nothing in my post reflects in any way on the obvious excellence of the evaluators. I was analyzing your choices as to the composition of the board*** (emphasis added).

[Here another commenter weighs in:]

Commentator said in reply to Brian…

Obviously one can have an “informed opinion” about where to work on an area without being expert in that area. I have an informed opinion about where to work on any area of philosophy whatsoever. But I was under the impression that the goal of the PGR is to give expert opinion. (You have certainly described it that way many times.) If not, then why bother with multiple area groups at all? Everyone on the general list is more than capable of giving an “informed opinion” on any area. In fact, John has simply pointed out that expertise in French philosophy is significantly under-represented on the continental list. (I am confident that the philosophers making up the list would agree.) (Emphasis added.)

Brian Leiter said…

[Commentator] et al: one can have an expert opinion without being a full-time specialist in an area. That was the only point (emphasis added).

Leiter is now claiming one can have an “expert opinion,” which might sound better than a “reasonably informed opinion,” but, really, this sleight of hand won’t work.  Using “expert” as an adjective is not the same thing as using it as a noun, at least not in this situation, especially given what Leiter had already conceded about his evaluators, namely, that “most evaluators are asked to evaluate more than one area, and inevitably that means they evaluate areas in which they don’t necessarily work primarily.”

Leiter tells us that “the opinion of experts is what matters.  We have nothing else to go on.”  If so, all of Leiter’s evaluators should be experts in their areas, but, as we’ve seen, for Leiter they need not be.  Professor Leiter and the PGR have not met their own standards, and the Philosophical Gourmet Report needs no “smear campaign” to discredit it.  It does very well on its own, thank you.

1280px-Catastrophe_du_pont_sur_le_Tay_-_1879_-_Illustration

__________

*This quotation is from Leiter’s post entitled “The five most common objections to the PGR,” under #2.  In a later post, in which he attacks critics of the PGR (“Why the visceral and largely irrational response to being evaluated?“), Leiter links back to the “five most common objections” post.  The “irrational response” argument of the later post is a red herring.  People do not object to being evaluated.  They object to being evaluated or ranked by a system that quantifies in methodologically unsound ways.  The fact is that Leiter has given up defending the PGR by means of anything anybody would call cogent arguments.  He now cites his own past statements as a one-size-fits-all defense of the PGR, or resorts to ad hominems.  In an update to the “irrational response” post, here is one way that he dismisses his critics, after referring to their “ranting and raving about the PGR in Cyberspace” in the original post:

UPDATE:  A note of caution to students:  do not be misled by philosophers who teach at or took their PhDs from poorly ranked departments (or unranked departments) who profess with great certainty and earnestness that there are serious methodological problems with the PGR:  there are not, and no one serious who has studied it thinks there are.

**If one doesn’t include Feminist Philosophy–because Leiter and Brogaard combined the evaluators for 2011 and 2014, and we don’t know who evaluated solely in 2014–there were a total of 548 evaluations done in 32 specializations.  Leiter has told us that there were nearly 200 evaluators for the overall rankings, and 230 when the overall and the specializations are combined.  This might mean that there were only 30 +1 persons ranking for the specializations.  But let’s be generous.  Let’s assume that 200 different individuals participated in the specialty rankings.  This would mean an average of 2.74 evaluations per person.  If you go through the PGR, you will find numerous instances of people ranking 3 and 4 times.  For example, 44 members of the Advisory Board participated in the specialty rankings.  Of these 11 ranked 4 times and 17 ranked 3 times.

***Protevi uses “board” to refer to the list of evaluators in 20th Century Continental philosophy.

William James on the Philosophical Gourmet Report

200px-Wm_james  Here is a passage from James’s essay “The Will to Believe,” which UP@NIGHT recently learned was written in anticipation of the Philosophical Gourmet Report.

Mr. Balfour gives the name of  “authority” to all those influences, born of the intellectual climate, that make hypotheses possible or impossible for us, alive or dead. Here in this room, we all of us believe in molecules and the conservation of energy, in democracy and necessary progress, in Protestant Christianity and the duty of fighting for “the doctrine of the immortal Monroe,” * all for no reasons worthy of the name. We see into these matters with no more inner clearness, and probably with much less, than any disbeliever in them might possess. His unconventionality would probably have some grounds to show for its conclusions; but for us, not insight, but the prestige of the opinions, is what makes the spark shoot from them and light up our sleeping magazines of faith. Our reason is quite satisfied, in nine hundred and ninety-nine cases out of every thousand of us, if it can find a few arguments that will do to recite in case our credulity is criticized by some one else. Our faith is faith in some one else’s faith….**

*[and the Philosophical Gourmet Report,]

**William James, “The Will to Believe,” in Essays in Pragmatism (New York: Hafner Press, 1948) 93-94.

A Portrait of the 2014 Philosophical Gourmet Report by the Numbers

 

school-of-athens-detail-from-right-hand-side-showing-diogenes-on-the-steps-and-euclid-1511.jpg!Blog    In a recent post I asked why Professor Leiter had decided to replace reputational rankings with impact studies in his law school rankings, while sticking with reputational surveys in philosophy. In response Professor Leiter made the following claim:

The other big difference between academic law and academic philosophy is that in the former there is far less consensus on scholarly paradigms than in the latter.

More than few people in philosophy reacted with amazement to the notion that there is far less consensus in law than there is in philosophy. In order to explain his claim, Leiter posted an addendum:

As I pointed out to [a colleague], and is perhaps worth sharing, Kieran Healy’s research found remarkable convergence in overall evaluations of programs across almost all areas of philosophical specialization–that’s the evidence about consensus I had in mind [evidence not refuted by noting that at the margins there are dissenters from the consensus, obviously].  {Emphasis added, square brackets in original.}

This is a false statement. Professor Healy did not find a “remarkable convergence…across almost all areas of philosophical specialization.” In a post on Leiter’s blog comparing “the level of consensus or disagreement between specialists” regarding the overall rankings, Healy discovered that there was “a relatively high degree of consensus around the top seven or eight departments,” out of ninety-nine departments, but varying degrees of disagreement for other ranked schools, with those at the bottom showing more consensus than those in the middle. (Gregory Wheeler has argued that Healy’s results show that, except for the top six schools and a handful at the bottom, “rankings vary quite a lot”*.)

Leiter’s adamance about the “remarkable convergence” is no small matter.  He insists that there is a consensus and that the evidence for the consensus is the PGR. The criticism of colleagues, some with considerable training in statistics and survey methodology, over many years has not budged him. In his most recent remark it is clear that he’s doubling down. He cannot or will not see the circularity of his position. The PGR is philosophy and philosophy is the PGR. QED.

Using the data of the 2014 PGR I show here that the consensus Leiter insists on is artificial.  But make no mistake: this is not just about the PGR. It is about whether our vision of philosophy is like Raphael’s “The School of Athens,” in which there is room for multiple and widely different ways of doing philosophy, or whether a particular style, method, and set of concerns should crowd out current and future philosophical diversity.

There are too many problems with the current PGR to address in one post. This is an overview of data that call into question assumptions about the PGR’s reliability and its ostensible consensus. While I focus on the specializations here, note that the problems with this evaluator pool bear on the overall rankings because the pools are so similar.  For the sake of clarity and interest, I proceed by presenting a series of issues, asking the reader to decide–without any obligation to rank them–which are worse.

Which is worse?

  • That the current PGR lumped together the 2011 and 2014 rankings of Feminist Philosophy, unheard of in any other specialization, because there were too few evaluators this year, and still only had 12 evaluators?
  • Or that in 32 specializations–leaving Feminist Philosophy aside because we don’t know how many or who ranked in 2014–there are only 32 women?  (That’s correct, 32 women philosophers for all 32 specializations. It may seem like more because the same evaluators often rank in multiple areas.)
  • Or that in the Philosophy of Language, Philosophy of Mind, Metaphysics, Epistemology, and Philosophical Logic, there are only 11 different women evaluators?
  • Or that in the 27 other specializations not listed immediately above, there are a total of 21 women?
  • Or that 75% of all specializations have three or fewer women philosophers evaluating, including those who evaluate multiple times? (Or that 44% have 0 or 1, or that 63% have 0, 1 or 2.?)

Which is worse?

  • That 72% of the specializations have 20 or fewer evaluators, up from approximately 60% of the specializations for the 2011 PGR?
  • Or that 38% have 10 or fewer?
  • Or that 28% have 8 or fewer?
  • Or that 8 out of 9 areas, 89%, in the History of Philosophy have 20 or fewer evaluators, up from 6 out of 9 for 2011?
  • Or that, as troubling as these figures are regarding the small number of evaluators in so many areas, they only tell part of the story, because evaluators frequently rank multiple times? (So there are actually fewer different individuals doing the ranking in these specializations. See next.)

Which is worse? (Keeping in mind Leiter’s claim that there is a “remarkable convergence in overall evaluations of programs across almost all areas of philosophical specialization”.)

  • That 41% of the Philosophy of Language evaluators were also Philosophy of Mind evaluators (11/27)?
  • Or that 69% of the Philosophy of Law evaluators were also evaluators for Political Philosophy (11/16)?
  • Or that 59% of Political Philosophy evaluators were also evaluators for Ethics (20/34)?
  • Or that 79% of Cognitive Science evaluators were also evaluators for the Philosophy of Mind (22/28)?
  • Or that 37% of Metaphysics evaluators were also Philosophy of Mind evaluators (11/30)?
  • Or that 63% of Metaethics evaluators were also Ethics evaluators (10/16)?
  • Or that 85% of the 20th Century Continental evaluators were also Kant or 19th Century evaluators (11/13)?
  • Or that 46% of 20th Century Continental evaluators were also Kant evaluators (6/13)?
  • Or that 67% of those who ranked in Medieval also ranked in Philosophy of Religion (4/6)?
  • Or trumpeting the convergence between the specialty rankings and the overall rankings of the PGR without mentioning overlaps like these in the 2014 PGR?

Which is worse? (Bearing in mind that in four specializations, including Feminist Philosophy, the PGR recognizes that adjustments should be made for the small numbers of evaluators.**  Remarkably, none of the departments below fall into this category, even though some have few evaluators.)

  • That only 6 evaluators in the Philosophy of Mathematics ranked 41 departments, while 39 evaluators in Ethics ranked 42 departments?
  • Or that 7 evaluators in the History of Analytic Philosophy ranked 42 departments, while 42 evaluators in Philosophy of Mind ranked 37?
  • Or that 10 evaluators in Applied Ethics ranked 59 departments, while 41 evaluators in Epistemology ranked 37?
  • Or that 12 evaluators in Mathematical Logic ranked 43 departments, while 30 evaluators in Metaphysics ranked 37 departments?

Which is worse?

  • That the Philosophy of Religion has only 2 evaluators currently at Catholic universities—the same one, St. Louis University—effectively excommunicating a large number of philosophers who work in the Philosophy of Religion?
  • Or that in the Philosophy of Law only 7 of its 16 evaluators, 44%, are at US institutions, for a rankings report in which the vast majority of ranked programs are located in the US, and strangely, which is spearheaded by Brian Leiter, a law and philosophy professor at a US university?
  • Or that American Pragmatism, a specialization whose experts have ties to both analytic and continental philosophy, is only represented by 3 evaluators, 2/3rds of whom are not at universities in the United States?***
  • Or that the Philosophy of Race has only three evaluators?
  • Or that Chinese Philosophy only has four?
  • Or that there isn’t any other non-Western philosophy represented, for example, Japanese philosophy, or Indian philosophy?
  • Or that there isn’t a separate category for Latin American Philosophy?
  • Or that Feminist Philosophy didn’t have enough evaluators this year so that 2011 and 2014 were combined?  (Wait . . . this is where we came in.)

Enough.  In my next post I will establish that there are evaluators who are not leading experts or experts at all in the specializations they are ranking. (I will not embarrass individual evaluators, or list names, etc.)  I will also show that there are imbalances in the evaluation of specializations, for example, that eight of the experts in 19th Continental Philosophy are Nietzsche scholars, while other major figures are hardly represented–Kierkegaard gets one expert.  There is also the issue of whether there are more highly qualified people who have been overlooked or dismissed because they do not fit the confines of The Consensus.

I began this post by raising concerns about Brian Leiter’s lack of appreciation for philosophy’s diversity and about the PGR’s aspiration to mold and hold the profession to Leiter’s vision of it. However, his assertion about consensus is only one piece of the puzzle over why Leiter continues to argue so vehemently for the PGR, often attacking those who disagree with him–personally and sometimes brutally–on his blog and elsewhere. I have wondered whether there was something in his philosophical outlook, beyond mere personal idiosyncrasy, that leads him to conduct himself like this, and which shapes both the PGR and his defense of it.  And then I happened across a quotation apparently very important to Leiter; a passage from Nietzsche that, he makes clear, he has quoted more than once. The quotation appears in his essay “How to Rank Law Schools.” ****  The quotation and its context occur near the very end of the article.  He introduces the quotation with these words:

 Academic rankings that provide actual information on matters of educational value have a useful role to play for students, quite obviously, but they also have a constructive role to play for faculty. Professor Korobkin suggests that in ranking schools we want to discourage “status competition.” I guess my own view is more Nietzschean, and so let me close with a quote I have used before. This is Nietzsche from his early essay on “Homer’s Contest”:

 [Then Leiter quotes Nietzsche:]

[J]ealousy, hatred, and envy, spurs [sic] men to activity: not to the activity of fights of annihilation but to the activity of fights which are contests. The Greek is envious, and he does not consider this quality a blemish but the gift of a beneficient [sic] godhead . . . . The greater and more sublime a Greek is, the brighter the flame of ambition that flares out of him, consuming everybody who runs on the same course.

. . . .

Every talent must unfold itself in fighting: that is the command of Hellenic popular pedagogy, whereas modern educators dread nothing more than the unleashing of so-called ambition . . . . And just as the youths were educated through contests, their educators were also engaged in contests with each other. [All ellipses in original.]

Whatever one may think of the sentiment–or its employment in this way, or Leiter’s evident attraction to it–its invocation in a discussion of the right way to rank academic programs should give us pause.  “The greater and more sublime a Greek is, the brighter the flame of ambition that flares out of him, consuming everybody who runs on the same course.”  For people who see philosophy as a contest and vanquished opponents as fit for nothing but the flames, well, Leiter and his PGR are the way to go.   But in “The School of Athens,” all we’d see is that this guy can really empty a room.

 

__________

I compiled all of the statistics in this post.   Of course I may have made errors.  But I do believe that any errors would be minor and not undermine the basic points or patterns under discussion.  There are too many similar results.  If a reader finds any errors, please notify me.

___________

* Commenting on Healy’s work, “Ratings and Specialties,” on Leiter’s blog, Wheeler says, “Those rankings were then aggregated to see how much variation there is across the specialties, which gives a sense of  how much (or how little) consensus there is across specialties. The box and whisker plots (top 25: .png.pdf; total population: .png.pdf) give a picture of this.  Except for the top 6 departments, and a handfull rounding out the bottom, the answer is that the rankings vary quite a lot:  people vote according to whom they recognize, and invariably those are the people working in their area(s) of specialization(s).” Gregory Wheeler, “Manufactured Assent: The Philosophical Gourmet Report’s Sampling Problem,” in Choice and Inference.

**“Due to the small number of evaluators, we are not printing the rounded mean scores, but just a list of programs, broken into two groups based on the scores received.”

*** It’s not as if there isn’t a wealth of experts to draw on in the US for American Pragmatism.  Perhaps Leiter should consider contacting the Society for the Advancement of American Philosophy to get some names, which has hundreds of members.  Oh, wait, this is one of the organizations that is outside of the consensus.

**** “Commentary: How to Rank Law Schools,” Indiana Law Journal, Vol. 81, 2006

 

 

Why Did Leiter Give Up Reputational Surveys in Law, but Not in Philosophy? The Mystery Deepens

220px-Diogenes_looking_for_a_man_-_attributed_to_JHW_Tischbein   Two days ago I compared Brian Leiter’s philosophy and law school rankings in a post (“Before You Consult the 2014 Philosophical Gourmet Report, Consider Leiter’s Words: “Reputation tends to be yesterday’s news”).  I referred to comments Leiter made in 2010 about the relative value of reputational surveys and scholarly impact studies.  Yesterday Leiter declared on his blog that I had taken his quotation out of context and misrepresented his views.  This is what Leiter says in his post:

The “vitriolic criticsms” [sic] are understandable and also unrepresentative–obviously the critics have tremendous incentives to be a salient presence on social media, given the huge influence the PGR actually has in the real world.  As I’ve noted before, it’s important not to be misled by the volume and persistence of the critics–pay attention to who they are, where they teach, where they earned their PhD, this will usually tell you more about what’s really going on.  Not all of them have self-serving motives*, to be sure–some just have no judgment (vide Velleman). . . .

*Some are also pathologically dishonest, and are getting increasingly desperate now that the PGR is out.  The most amusing is the former SPEP Advocacy Committee member who purports to quote me saying, “Reputation tends to be yesterday’s news–what happened 25 years ago,” without noting that I was discussing the awful U.S. News surveys of law schools, which are random surveys (not suveys [sic] of experts) and which provide the respondents with no information at all–of course, those kinds of surveys are yesterday’s news.  The other big difference between academic law and academic philosophy is that in the former there is far less consensus on scholarly paradigms than in the latter.

I made two basic claims in my original post:  first, that Leiter gave up reputational surveys for impact studies in his law rankings, and, second, that he thinks that impact studies are superior to reputational surveys.  This second claim was based on the decision he made to give up reputational surveys in favor of impact studies as well as on his comments about the superiority of impact studies over reputational surveys.

The problem that Leiter faces in trying to confine his comments to criticism of the U.S. News reputational survey is that he doesn’t qualify his claim in this manner in the article I cited.  Here once again are the remarks that I quoted in my post.

Most of the law schools in the top twenty are not surprising.  But Leiter and Sisk agree that the study is a good indicator for future reputation.

“[Scholarly] Impact tells you things that reputation doesn’t,” Leiter said. “Reputation tends to be yesterday’s news–what happened 25 years ago.  I think [this study] is useful for students who care about the academic experience” [brackets in original].

This quotation states that “[Scholarly] Impact tells you things that reputation doesn’t.”  There is no qualification here: he doesn’t except his own expert-driven reputational surveys.  In addition, he drops this portion of the quotation in his criticism of my post and only quotes the second sentence.  “Reputation tends to be yesterday’s news–what happened 25 years ago.”   This allows him to suggest that he is only talking about the U.S. News survey, or at least to make that reading more plausible.  Restore the rest of the quotation and my point is all the stronger.

But there’s another difficulty with Leiter’s story:

The last reputational survey Leiter did for his law school rankings was in 2003.  As I point out in my original post, the reputational rankings were later replaced by impact studies based on citations. (Leiter has also relied on lists of inductees to the American Academy of Arts and Sciences to rank faculty quality in the same period.) The shift to impact studies started in 2005, and is beyond dispute.  What remains a mystery is why the shift, especially since Leiter was quite happy with the reputational survey approach for his law school rankings, and championed it in the same terms as he now describes the methods of the PGR–look at all of the fabulous people participating, etc.  Here is Leiter in his “Introduction” to the 2003-2004 reputational survey results for his law school rankings:

Since high-quality survey data may ultimately be more informative than “objective” measures, it is my intent, for now, to rely on this data (emphasis added).

Further down the page he says this:

The quality of evaluators in this survey is unparalleled: it includes the President and President-elect of the Association of American Law Schools; a dozen members of the American Academy of Arts & Sciences, the nation’s most prestigious learned society; dozens of the most frequently cited legal scholars in numerous fields; and leading figures, junior and senior, in corporate law, criminal law, health law, constitutional law, jurisprudence, international law, comparative law, legal history, feminist legal theory, and many other fields.

Sounds like he has a pretty good thing going.  But then the reputational surveys stop, and Leiter begins using impact studies based on citations.  So the other difficulty that his story faces is that he abandoned reputational surveys for impact studies in his law school rankings.  Why would he do this if he didn’t believe that impact studies were superior to reputational surveys for law school rankings?**   The quotation I cited is what one would expect to hear after Leiter decided to switch.

Finally, a word about the last line in his criticism of my post.   Leiter claims that there is “far less consensus on scholarly paradigms” in law than there is in academic philosophy.***  He appears to hope that people will believe that this is the reason that he stopped doing reputational surveys in law while continuing to use them for philosophy.  The problem is, it’s not believable.  Philosophy is certainly not enjoying a greater consensus about “scholarly paradigms” than law schools(!), and asserting that it does is completely unconvincing as a reason for assessing the fields differently.  At best it’s a piece of wishful thinking on Leiter’s part, with the PGR itself something of a fantasy.

_________

**As I noted in my post, the U.S. News’s rankings are only mentioned at the start of the article in order to contrast them with what Sisk and his colleagues were doing, with a very brief mention that part of what U.S. News does is reputational.  The article then moves on, with Leiter’s words coming near the very end.  I should add that in the middle of the piece there is another quotation from Leiter regarding how “Scholarly impact is a measure of the intellectual quality of the faculty…”  Nothing more is said about U.S. News’s rankings.

*** UPDATE:  12:00 PM, December 10.  I inadvertently switched law and academic philosophy in this sentence when I first published the post early this morning.  It’s been corrected.  Thanks to John Protevi for catching it.