The Dog Ate My (Philosophical Gourmet) Report

TO RANK OR NOT TO RANK….in the year 2014?

Consider some necessary but not sufficient conditions for the evaluation of individuals or groups in order to rank them: (1) there should be explicit standards—criteria—on which to base the evaluations, and (2) there should a public code of conduct for the evaluators—judges—to follow, as well as a detailed description of their function as judges. Currently neither of these exist for the PGR, but they do exist for organizations well-known in the U.S.

Here is the way that the PGR addresses #1.

Please give your opinion of the attractiveness of the faculty for a prospective student, taking in to account (and weighted as you deem appropriate) the quality of philosophical work and talent on the faculty, the range of areas the faculty covers, and the availability of the faculty over the next few years (emphasis added).

In addition, the PGR provides the following guidance to the evaluators regarding criteria and their respective weights.

“Faculty quality” should be taken to encompass the quality of philosophical work and talent represented by the faculty and the range of areas they cover, with the two weighted as you think appropriate. Since the rankings are used by prospective students, about to embark on a multi-year course of study, you may also take in to account, as you see fit, considerations like the status (full-time, part-time) of the faculty; the age of the faculty (as a somewhat tenuous guide to prospective availability, not quality); and the quality of training the faculty provide, to the extent you have information about this (emphasis added).

Hmmm.  Not much here by way of any definitive and well-defined criteria: “attractiveness,” “weighted as you deem appropriate,” “quality,” “talent,” “weighted as you think appropriate.” “may take into account as you see fit,” and “to the extent you have information about this.” *  Judges can decide which criteria to use and in what combination. Note that the PGR’s criteria for evaluation are run together with the standards for the judicial function (#2).

Compare this with the way that the Westminster Kennel Club discusses the breed standards by which individual dogs should be evaluated. For breeds you might think of specializations; for individual dogs and their attributes, perhaps departments and their faculties. The important thing is the way in which an enterprise develops and provides criteria for judgment of its objects, not the objects themselves, dogs or departments.

Each breed’s parent club creates a STANDARD, a written description of the ideal specimen of that breed. Generally relating form to function, i.e., the original function that the dog was bred to perform, most standards describe general appearance, movement, temperament, and specific physical traits such as height and weight, coat, colors, eye color and shape, ear shape and placement, feet, tail, and more. Some standards can be very specific, some can be rather general and leave much room for individual interpretation by judges. This results in the sport’s subjective basis: one judge, applying his or her interpretation of the standard, giving his or her opinion of the best dog on that particular day. Standards are written, maintained and owned by the parent clubs of each breed (CAPS in original).

Notice both the attention to detail and the willingness to acknowledge that there will still be room for interpretation by judges, or evaluators. We have here both reasonably objective criteria and a bow to the interpretive dimension in one paragraph. Impressive! In addition, and perhaps most importantly, the Club refers to the relations of form and function (“the original function that the dog was bred to perform”), that is, there is some agreed upon standard/function for each breed. And WKC expects each breed’s parent club to provide a written standard for its breed.

Now consider how the PGR handles #2, that is, how judges should understand their roles and rules of conduct.

Different respondents had different “centers of gravity” in their scoring: some gave no 5s, others gave no score lower than a 2. It was also clear that respondents had different philosophies of evaluation: some clearly tried to consider the breadth of strength in a department, while others ranked a program highly or lowly based simply on its strength in his or her fields. The range of evaluations for single departments should be a cautionary note to all undergraduates about relying too much on the advice of just one or two faculty advisors. Idiosyncrasy abounds, even at top departments! (emphasis added).

Yes, indeed, idiosyncrasy abounds. Now, for contrast, let’s look to the hounds: the American Kennel Club has a thirty page document detailing the expectations for proper judging and the requirements for judges. Here is the way the AKC introduces the document:

Judging at AKC® shows should be enjoyable for the judge and beneficial to the sport of purebred dogs. In this publication, you will find Rules, Policies and suggested Guidelines. The Policies and Rules will be clearly designated as such. The suggestions have been developed over the years based on the experience of many seasoned judges and the AKC staff. You will find them most helpful in learning the judging process. Policies are adopted by the Board of Directors, and Rules are approved by the Delegate body. Compliance with these is mandatory.

“Policies are adopted by the Board of Directors, and Rules are approved by the Delegate body. Compliance with these is mandatory.” Seems eminently sensible.  The PGR, on the other hand, says virtually nothing about the role of judges, except the catch as catch can points mentioned under #1 above and the acknowledgement, “It was also clear that respondents had different philosophies of evaluation.” (Different philosophies of evaluation to rank different philosophy departments. No comment.)   And the AKC provides a staff to assist those learning the judging process.

I know, you can say that we philosophers don’t need any such guidelines because, well, we all know how to serve as judges. However, the problem is that without agreed-upon expectations for our judges we can’t be sure that they are actually judging in the same way, with the same care, and with the same set of assumptions. As far as I know, neither the PGR’s editors nor its Advisory Board provide detailed policies or any policy at all in this regard. The criteria and the guidelines for judgment should be explicit and known not only to the judges but to the judged.

This is embarrassing. The dominant rankings system in philosophy, the “Philosophical Gourmet Report,” doesn’t provide its evaluators with anything like the detailed or comprehensive guidance that judges in dog shows receive.

I’ve heard complaints that the PGR is equivalent to a dog and pony show. Wrong! The dog fanciers have us beat, paws down.  Or they’re eating our lunch, I mean, our Report.

So, the question remains: To Rank or Not to Rank in the year 2014? **

    * How many philosophers does it take to change a quality light bulb?   Not going to happen, my friend.  The philosophers, any number, will first have to agree that the object in front of them is a quality light bulb.

** Coming Soon:  A  list of ten reasons that evaluators can use to excuse themselves from this year’s PGR rankings.  [UPDATE: Now Available here.]