Dying With Indignity: An Update on the PGR

It is not news that the PGR is in serious trouble.  Last fall (2014) saw a significant number of evaluators abandon the PGR, so many in fact that the PGR could not find nearly enough replacements, leaving many specializations with markedly fewer evaluators than in 2011 (here and here).  (For example, one of the mainstays of the analytic tradition, Philosophy of Language, lost 60% of its evaluators from 2011 to 2014.  After some replacements were found, the loss was still 48%.)  In addition, the transition to new leadership has proven problematic, to say the least.  We learned last spring, and again this fall, that Berit Brogaard, who was scheduled to take charge of the PGR, has concluded that she will need the assistance of a co-editor.  When Brogaard and Leiter ran into trouble trying to find a new co-editor—incredibly, after all that happened last year—she offered the position the Leiter, who has said that he no longer wants the position.  They are still searching for a co-editor.  It seems that no one wants the job or has the necessary resources for it.  But we were also informed that, “Since Brit does not want to produce a new PGR until fall 2017 (which seems quite reasonable), there is no hurry fortunately.”  (I would myself be a little more worried if I couldn’t find a co-editor for the renowned PGR after many months of searching.)  In addition, the advisory committee that was supposed to be appointed to address possible changes to the PGR was never constituted, or if it has been, the philosophical community has not been informed. (1)  Last but not least, we were recently told that the Advisory Board has been disbanded, and a new Board will be formed, which may have members from the old Board or at least have old Board members involved in some way in the PGR.  No explanation for how the new Board members will be chosen has been given.  And to date, as far as we know, no one has been appointed.

Adding to the PGR’s leadership challenges is the fact that there is less and less need for a product like the PGR.  Prospective graduate students have far more access to pertinent information today than just a few years ago on the internet, including Philwiki and real data about placement through a project funded by the APA; the latter’s data is golden compared to the “data” that the PGR supplies.  Given the different sorts of challenges that the PGR faces, you can see why the folks at the PGR may feel under siege.  It’s becoming a divisive dinosaur.

This is the backdrop for the brouhaha over a paper published by Brian Bruya in Metaphilosophy, “Appearance and Reality in The Philosophical Gourmet Report: Why the Discrepancy Matters to the Profession of Philosophy.”  Bruya raises questions that many of us in the profession continue to have about the biases, implicit or explicit, that are built into the PGR.  There is ample evidence to support that this is a legitimate concern, and there are many technical and non-technical discussions about the problem (see posts and articles linked here).  But it doesn’t require a rocket scientist to demonstrate that there is a problem.  One can just look at the number of evaluators in the various specializations in the PGR to see that it is heavily weighted to certain “core” areas, and then recall that practically same pool of people does the overall rankings.  Now combine these two facts with the fact that evaluators rank many many departments–according to Kieran Healy, Leiter’s sociologist, almost 40% of the evaluators ranked 90 or more departments in 2006 and the median number in the U.S. was 81.  And let’s add one more fact, the guidance given to the evaluators on the PGR’s site, which basically says to them, use your own judgment on criteria, even if its different, or weighted differently, than those of other evaluators. (2)

Different respondents had different “centers of gravity” in their scoring: some gave no 5s, others gave no score lower than a 2. It was also clear that respondents had different philosophies of evaluation: some clearly tried to consider the breadth of strength in a department, while others ranked a program highly or lowly based simply on its strength in his or her fields. . . .

“Faculty quality” should be taken to encompass the quality of philosophical work and talent represented by the faculty and the range of areas they cover, with the two weighted as you think appropriate. Since the rankings are used by prospective students, about to embark on a multi-year course of study, you may also take in to account, as you see fit, considerations like the status (full-time, part-time) of the faculty; the age of the faculty (as a somewhat tenuous guide to prospective availability, not quality); and the quality of training the faculty provide, to the extent you have information about this (emphasis added). . . .

Given the open-ended nature of the guidance, it defies common sense to believe that people’s familiarity with the faculties in their own specializations would fail to have a significant impact on their overall evaluations, especially when they are ranking so many departments.  And of course there are many other problems with the PGR, including the relatively small number of graduate programs from which many of the PGR’s evaluators are drawn, the small number of evaluators in most of the PGR specializations, the use of the same evaluators multiple times (often in areas in which they aren’t experts), and the underrepresentation of women. (3)

Returning to the PGR’s current difficulties, given its falling rate of evaluator participation and its leadership challenges, perhaps we should not be surprised to find a near-hysterical headline in the Leiter Reports regarding Bruya’s recent critique of the PGR: “Brian Bruya’s attack on the PGR is even a bigger fraud than it first appeared to be.”  The PGR just can’t handle any more bad news, especially on the diversity issue. The questions that Bruya raises are longstanding, but he and his paper have been subject to scathing attacks by Leiter and David Wallace (Oxford, USC).  They insist that Bruya’s methodology is so flawed that the paper should be retracted.  Now that’s harsh!  But, you see, it’s such bad science that we must kill it to protect the integrity of science and philosophy.  And of course the boys at the PGR have no other concern than the damage that is being done to science and philosophy by the publication of an article in a journal with a relatively small circulation. (4)  After all, there is no self-interest at the PGR, only among the PGR’s opponents.

Yes, we need to protect science.  But the level of intensity and invective here, especially about an article in a journal that Leiter and his people would normally pay little attention to, doesn’t ring true.  And there is a simple reason why.  It’s obviously not just about the science.  The PGR can’t take too many more hits, and because Bruya’s piece was highlighted on the Daily Nous, its proponents had to go on the offensive.  But the hypocrisy involved is mind-boggling.  There is no way that the PGR could stand up to anything nearly as intense as the attack that these guys have launched against this article.  The PGR is not scientific and it is methodologically impoverished.  No social scientist worth his or her salt would set out to discover the quality of philosophy departments using the ultra thin criteria that the PGR uses, which basically tells evaluators to use their own judgment.  This allows not only a mixing of apples and oranges but provides a veritable fruit basket of options.  We simply don’t know if the evaluators are using the same criteria or weighing the criteria in a consistent fashion.

What’s actually happening here in terms of Wallace and Leiter is more akin to what climate change denialists do.  They find some allegedly flawed study or glitch in the work of investigators on climate change and they trot this out as proof that climate science can’t be trusted. “See, they are mistaken about this, so they must be mistaken about climate change.”  In this case, the supporters of the PGR hope that a take-down of Bruya in the name of science will help restore faith in the PGR.  It won’t and it shouldn’t.  (And this would be true whether they are completely right, partially right, or totally off the mark about Bruya’s article.)  The only thing that can help the PGR is to change the way it which evaluates, but that’s like asking wolves to remain wolves without their fangs.  Don’t hold your breath.

If we believe in the value of science, what is to be done?  The PGR should be evaluated by a committee of survey specialists and statisticians from outside of philosophy.  I made this suggestion just the other day to Wallace in a thread on the Daily Nous. (5)  He avoided responding.  He only wanted to talk about his criticisms of Bruya.

Let’s agree that people in glass houses shouldn’t throw stones.  As long as the PGR’s leadership is unwilling to subject the PGR’s methodology and procedures to the scrutiny of an impartial committee of experts, preferably people from outside of philosophy, they need to get off their high horses about protecting science and acting as if Bruya’s article is the worst thing to happen to science in the last century.

__________________

1.  Here is Leiter’s own account of the deal for the changes to the PGR, from an October 10, 2014 post on his blog.

Folllowing (sic) up on last week, the Advisory Board and I have agreed on the following statement regarding the plan for the PGR:

The 2014-15 PGR will proceed as planned, with Berit Brogaard joining Brian Leiter as co-editor and taking over responsibility for the surveys and the compilation of results, with assistance as needed from Brian and the Advisory Board.  At the conclusion of the 2014-15 PGR, Brian will step down as an editor of the PGR and join the Advisory Board.  Berit will take over as editor until such time as a co-editor can be appointed to assist with future iterations of the report.  After 2014, Berit will have ultimate decision-making authority over the PGR.  Upon completion of the 2014-15 PGR, Berit will appoint a small advisory transition committee that she will consult on possible improvement, both substantive and operational, in the PGR going forward

2. For more on the criteria issue, see, “The Dog Ate My (Philosophical Gourmet) Report.”  If you want to get some idea of the lack of care and concern that Leiter shows for presenting the evaluation process, here is another statement, which follows the quotations I just cited. You can find it on the current PGR site.  Notice the date, which is almost ten years out of date for the 2014-2015 PGR.

The faculty lists are based on current information as to their shape for fall 2005, and are confined to full- or part-time faculty (status indicated), excluding lecturers, but including those with joint appointments in Philosophy and another unit. Emeritus faculty are also excluded.

In addition to the date error, which has been there for years, the description of the report is from 2011, so if one doesn’t pay attention to the date, he or she would think that there were more philosophers involved, over 300, than were actually involved in 2014.  Does leaving this misinformation up on their website months and months after the 2014-2015 PGR was published constitute a sufficient dereliction of duty that we should ask Blackwell withdraw its support and find new editors?  Wait!  This will never happen.  Leiter still holds the copyright to the PGR.

3. As noted by Greg Wheeler, for the 2011 PGR:

To put a finer point on this, there are 299 [4] evaluators and 126 universities represented in the 2011 PGR rater pool—113 individual Home institutions (blue) and 58 PhD institutions (green).  But of the 736 individual rankings submitted for the 33 areas of specialization covered by the PGR, half (48.5%) were submitted by alumni from just 8 universities.”

For the underrepresentation of women, see here.  For figures on the number of evaluators in the specializations and the multiple use of evaluators, see here; on the expert question, see here.  Leaving aside Feminist Philosophy—because the PGR used evaluators and evaluations from 2011 for 2014 for Feminist Philosophy, we don’t know how many women actually evaluated in this area in 2014—there were a total of 32 different women evaluators in the PGR’s 32 specializations.   (Not a typo: 32 is correct for both.)

4. Here is Wallace’s explanation for why he has gotten so involved in criticizing Bruya’s article:  “I have taken a large amount of time and energy to engage with this issue at an inconvenient time for me not because I particularly want to get involved in a debate about the pros and cons of the PGR but because it is important to me that philosophy, as a discipline, is able to engage with science properly and in a way that can reasonably ask to be taken seriously. If we’re prepared to let this kind of thing appear uncorrected in our research literature, and to suppose that straightforward and widespread error on elementary matters of methodology is just one more form of philosophical controversy, we will make fools of our discipline. We will appear to others as if we are children playing with tools that we do not understand.”  (From the thread to “Criticism of the Philosophical Gourmet Report.”)

Really, “not because I particularly want to get involved in a debate about the pros and cons of the PGR.”

So, if the same alleged errors had been made in a critique of a ranking system used by say, The Society for Phenomenology and Existential Philosophy, and it was published in a journal like Metaphilosophy, Wallace would have spent as much time and effort criticizing the article?  If you believe that, I have a bridge in Brooklyn to sell you.  Oh, and if the concern is to engage science properly, why hasn’t this happened with the PGR?  Exactly the same words that Wallace uses about his worries regarding how philosophy looks to the outside world could be applied to how philosophers look for continuing to take the PGR’s “science” seriously.  “We will appear to others as if we are children playing with tools that we do not understand.”  By the way, this is not hypothetical.  You can find discussions on the web by people in other disciplines responding in disbelief to the influence of the PGR and its methods.  Or just talk with colleagues in the social sciences, those without axes to grind, who have taken a good look at the PGR.

With all his concern about the use of science by philosophers, I’m curious about what a Wallacean review of the science behind Professor Brogaard’s,  The Break Up Cleanse, 28 Day Miracle Mind-Body Heart Break Recovery System, would look like.  The site for The Cleanse makes the following claim, “What is the best way to recover from your breakup?  This scientifically proven method of breakup recovery is the only method out there that comforts your mind, body and brain.”

5. Here is the offer that I made to Wallace in the Daily Nous’s thread on Bruya’s response to criticisms of his paper.

“I will bet $1,000. that an independent panel of survey experts and statisticians, drawn from outside of philosophy, would find that the PGR is not only unscientific, but not a reliable indicator of the quality of philosophy departments. To my knowledge, Leiter has never been willing to submit the PGR to such scrutiny.  Are you, David?  It’s time.”

Wallace’s response:

“@Mitchell Aboulafia: the topic of this thread is specifically Brian Bruya’s paper. Criticising that paper doesn’t somehow obligate me to defend the PGR against any other objection that anyone else makes.”

Wallace was incorrect about the topic of the thread.  The comments were linked to a thread on Bruya’s reply to criticisms of his article, not an earlier one directly on his article, and other folks on the thread had raised issues that related to the larger questions that Bruya was seeking to address in his defense of his article.  I responded to Wallace that I wasn’t asking him to respond to every criticism, just the ones that I was raising, which were relevant, given the topic under consideration.