What Should Count for Tenure and Promotion?

Why couldn’t God get tenure? Because He wrote only one book, and it wasn’t refereed. —Academic jokelore

This essay will be published in the forthcoming book Academic Ethics Today: Problems, Policies, and Prospects for University Life, ed. Steven M. Cahn (Rowman & Littlefield, 2022). This collection of thirty-one new essays will focus on ethical questions raised by institutional policies of colleges and universities. As a service to readers, over the next several months the APA Blog will publish four of these essays in their entirety, including articles on the role of adjuncts (Alexandra Bradner), assessing publications for tenure (David Shatz), student discipline (David Hoekema), and the responsibilities of administrators (Karen Hanson). All materials​ are copyrighted by the publisher and reprinted with permission. The entire list of topics and authors can be found online.​ 

Anyone who labors at academic scholarship knows vividly—perhaps even painfully—how much that enterprise depends on the process known as peer review. Peer-reviewed articles and books are the gold standard in decisions about hiring, tenure, and promotion. This is as it ought to be: the process provides quality control in scholarly communication. It doesn’t just filter out inferior work; it provides incentive for scholars to do their best work. Moreover, the criticisms and suggestions of referees improve submissions dramatically.

Even so, assessments of candidates for tenure and promotion (T&P) can go awry by adopting a narrow paradigm for what should count as a meritorious contribution, deviation from which can destroy careers. This system is based on an exclusionary way of thinking that I deem unjustified and unfair, as well as unwise. In this essay I explain why.

Jean’s Plight

Jean, a freshly minted Harvard PhD in philosophy, has just landed a position as an assistant professor at Dumont University. Although not in the same league as Harvard or Princeton, Dumont is a respected school that expects scholarly productivity of its faculty. Jean’s position is of that increasingly rare breed—a tenure-track job. She is eager to begin compiling a record of publications that down the road will earn her tenure and promotion.

Her mentors ranked Jean highly as a graduate student. She published an article in grad school, and her name had been circulating in the publishing grapevine. No sooner did she earn her doctorate than she received an email from her Harvard professor, whom a university press had contacted to edit an anthology in philosophy of language—her specialty. With his own plate already full of commitments, he decided to turn this one down while recommending Jean as a promising scholar who is on top of the literature, reliable, and judicious. Highly flattered by this opportunity, Jean undertakes work on the collection.

A few days later, a top journal invites Jean to review a new book by a giant in her field, a philosopher whose work she grappled with in her dissertation. The thought of publishing in such a fine venue on such a prominent author excites her. She gets to work immediately.

Not long afterward, a well-known philosopher who has heard wonderful things about Jean’s work asks her to contribute to New Essays in the Philosophy of Language. Jean jumps at the chance.

Her apparent good fortune continues. Jean’s paper is accepted for a highly prestigious conference. Then a well-known publisher recruits her as the sole author of a text introducing students to the philosophy of language. A nice royalty offer sweetens the publishing pot.

In total, in the first few weeks of her job at Dumont, Jean has landed editorship of an anthology, a contribution to an anthology sure to be read and respected, a presentation spot on a conference program, and authorship of a textbook.

After this rousing start, more invitations follow for articles and book reviews, as do offers to do a second text and other attractive options. Jean is riding high. The run continues: she publishes part of her dissertation and a well-placed article coauthored with a colleague. Then she posts a lot of her work in progress online.

Highly knowledgeable on social issues, Jean also assumes the role of public intellectual. Widely read highbrow magazines carry her political writings; newspapers of distinction publish numerous op-ed pieces by her. Space in these venues is at a premium, but Jean is good and gains the coveted spots.

A few years later, Jean’s tenure committee convenes. And the bubble bursts as her evaluators question most of the projects that had so excited her.

• Anthologies don’t count.
• Reviews don’t count.
• Conference presentations don’t count.
• Textbooks don’t count.
• Invited contributions don’t count.
• Coauthored pieces do not count.
• Online postings do not count.
• Work as a “public intellectual” does not count.

Ultimately, Jean is turned down for tenure.

Variants of Jean’s seemingly outrageous narrative occur in real life. Junior faculty who accept offers like those that came her way can in certain schools find themselves in precarious positions. Indeed, they would be well advised to turn down such work. Other conditions may even be imposed: some schools count only books published by a major university press toward tenure or promotion. Favorable external book reviews from highly regarded scholars are not enough; the publisher must be prestigious.

The consequences of junior faculty living under Dumont’s policies would be decidedly negative for their profession. Scholars and students would feel the impact. In some areas, there wouldn’t be enough textbooks, or enough high-quality textbooks, without the Jeans of the world writing them. There also wouldn’t be enough articles to fill certain anthologies, conferences would be understocked, and so forth. Alternatively, there would be such books and articles, but people as talented as Jean would not be writing them. Applying Dumont University’s standards would harm the dissemination of knowledge to scholars and students—in other words, education.

Certain jobs need to get done in a well-functioning profession; certain types of scholarly work needs to be produced; certain settings need to exist (such as symposia). Arguing that people who do these jobs or who join these settings are jeopardizing their chances of professional success seems anomalous. Prima facie there is something fundamentally wrong with a system that prevents much work from helping professional advancement even if it is (1) of quality and (2) beneficial to professors, students, and society. You might retort that, by my logic, a candidate’s regularly taking out the garbage or making coffee (tasks beneficial to the profession) should count toward tenure. But obviously these tasks are not professional activities that call upon philosophical skills or whatever skills an academic’s field requires. Someone needs to write the textbooks; someone needs to do oral presentations. And these, unlike garbage-dumping and coffee-making, require expertise in one’s discipline.

Should Invited Work Count?

Jean’s situation, while obviously a caricature, nevertheless raises serious questions about how T&P committees assess the work of faculty. I want to address one question in particular posed by Jean’s dossier: How should these committees weigh invited articles that appear in collections?

Some years ago, I invited an assistant professor at a major research university to contribute to an anthology of new essays I was editing. I did not know this individual personally, but his published work convinced me that he would be the best person to write on a particular area that needed coverage. Before extending the invitation, I consulted my colleagues and friends in the field to confirm my assessment. Thus my invitation was based on an evaluation of his previous work and in no way betokened a friendship with the invitee.

Two days after his initial acceptance, much to my surprise and chagrin, this young assistant professor turned my invitation down. He explained that senior colleagues had cautioned him that in tenure review, articles in books of essays would count far less than those in refereed journals, if at all. So the time needed to write the anthologized article would be better spent on an article (on the same or a different subject) that would undergo competitive peer review. Accepting invitations, he was told in effect, should be the luxury of the secure and famous.

Disappointed over losing what I expected to be a splendid contribution, I lamented the senior colleagues’ general attitude (or the attitude they were ascribing to others). For one thing, publishers often send proposed essays to reviewers, albeit they comment less expansively than when judging journal submissions. Moreover, when candidates come up for tenure or promotion, their work is sent out for review by outside specialists anyway, regardless of where the writings were published, in addition to being reviewed by colleagues. The T&P committee could have a dozen or so evaluations in hand for an anthologized article—far more than a journal editor has—and some are likely to be from experts of greater renown and experience than referees the invitee would have had for a refereed submission. If these evaluators are superior in number and quality to the potential referees, why should it matter that the piece was not refereed? Also, if a refereed journal rejected the articles, one wouldn’t know who the referees were or even perhaps their reasons for rejection; how, therefore, could one judge the quality of their particular reviews? By contrast, the personnel letters would be signed and could serve as a check on a referee’s recommendations. By soliciting further evaluations of refereed published material, universities seem to be accepting a higher court than that of the original referees, one with still higher standards for assessing the quality of an invited published work. Further complicating the picture, some articles in peer-reviewed journals are in truth invited articles. If an article thought to be peer reviewed received a fine reaction and was influential, are we supposed to retract our favorable view when we learn that an editor invited it?

Any assumption that an invited piece signals nothing more than a friendship is unfounded and unfair without hard evidence. Indeed, in the particular instance I cited, I had formed my opinion of this individual by conducting my own “peer-review” process when I assessed his articles and solicited opinions of others. Besides, academics often form “friendships” like this: X has heard of an article Y wrote. X finds it online, likes the article, and writes a nice note to Y. Y finds material by X and likes it. Each asks the other to send work in progress. Out of this mutual appreciation and respect for each other’s work, a friendship grows. The two scholars may meet personally only rarely, perhaps at conventions, or they may never meet. But they become friends, via correspondence, because they value each other’s scholarly abilities. The assumption that an invitation from a “friend” is unrelated to the invitee’s competence is gratuitous, indeed false in many cases.

This is not to say that a T&P committee member can be expected to differentiate cases in which people become friends out of mutual respect for each other’s work from cases in which an editor invites a schoolmate, cousin, or lover. Furthermore, an invitation does not usually entail a positive evaluation of a particular piece already written but rather of the author’s previous work. Acceptance to a journal, by contrast, testifies to a particular piece’s quality. But once again, scholars will evaluate that work when the time comes for a decision about tenure or promotion, and it may even be assessed in print before then. Remember, too, that editors want to protect their own reputations and to that end will want papers that meet a high standard. For that reason, they are unlikely to invite a paper only as a favor.

Speakers are invited to forums without prior knowledge and review of the particular speech they will deliver. Yet (1) one would not normally infer that the invitation came from friendship alone; and (2) if someone rates the speech high and hears others praise it as well, it would be absurd to moderate that assessment just because the speech was not subject to prior review. As with good invited speeches, so too with good invited articles. And let us not forget that a policy of disparaging unrefereed pieces would diminish the works of Plato, Aristotle, Descartes, Hume, Kant, and, of course, God.

I am not denying that the fact a piece was peer reviewed creates a presumption of quality via a reasonable inductive argument. But even if peer review were truly sufficient to guarantee high quality (which it is not—peer-reviewed published work is often later discovered to be seriously flawed), it is not a necessary condition, as non-peer-reviewed works can also be excellent. And, of course, some peer-reviewed articles are accepted with minimal revisions, suggesting they might be of high quality even before the review process. Invited works are often presumed to be in this category. Granted, without a peer-review system, committees would have a tough time sorting the wheat from the chaff. But it would be doable.

It might be replied that outside evaluators for a tenure and promotion case are likely not as demanding as referees and editors for a competitive journal or press. Personnel evaluators who decide to recommend X are not normally forced to choose between X and someone else, to compare X to other candidates. Journals and presses are competitive, however. So a positive recommendation from a referee carries more heft than a positive recommendation in a personnel evaluation.

This rebuttal is specious. The referee’s original reports were not made in the context of a competition. As a rule, referees are not comparing one article to others submitted to the journal. With a vast literature out there, referees may have missed some relevant recent work and thus cannot even do a good job of comparing an article to other published ones. An editor perhaps makes a competitive decision, but an editor will not always have read submitted papers carefully and will instead adjudicate the competition based on referee reports. Since an individual referee’s report is usually not based on a straightforward competitive evaluation, the editor may not have a direct way of ranking two “competing” submissions.

Personnel committees, by contrast, are often asked to offer comparisons—not only about where candidates rank relative to others in the field but also where particular works rank and what those works have contributed. Evaluators often say things like “this is the best piece I know of on the subject…” or “it’s one of the two or three best.” Why not count such praise of an invited article if one counts it for a refereed article? Can it in good conscience be ignored?

A personnel evaluator has one advantage that the original referees lack: hindsight. By the time the evaluation is written, the reviewer will be aware of something the referees could not know—namely, how well the article or book has been received and has withstood public scrutiny. Needless to say, the personnel referee’s evaluation should not slavishly follow others’ reactions, but an awareness of those reactions will help the evaluator form a more meaningful assessment. Nobel laureate Rosalyn Yalow originally had her prizewinning work rejected, and J. R. Mayer’s 1842 paper reporting the first law of thermodynamics was rejected by a leading journal and accepted by a relatively obscure one. The value of ideas is better judged after publication than before.

There are additional considerations in favor of counting invited articles heavily. First, many scholars are more familiar with key anthologies in their fields than with journals (in part because anthologies are used in courses). If more scholars would read a paper in certain anthologies than would read it in a journal, surely this ought to be significant in assessing the publication’s value to the field. Second, an exclusionary approach has a most untoward consequence. Departments must advise anyone planning to apply for tenure and promotion (like the young scholar I mentioned earlier) not to contribute any invited paper, nor even to deliver an invited lecture, when doing so is accompanied by the expectation that the paper will be published in conference proceedings. The cost of not heeding this advice could be a career. Whatever benefits there are to gaining exposure and name recognition are not likely to outweigh this drastic outcome of joblessness.

Forcing young faculty to decline invitations seems unfair. A young scholar may have a better chance of earning a reputation by publishing in a well-placed anthology than by publishing in a journal. Any good publication leads to expanded contacts and further opportunities. Several volumes of mostly invited essays have had great impact on the philosophical profession. I know of no one who disparages the quality of those books just because the essays were invited. Similar arguments apply to another criterion often used—the prestige of the journal or press that published the article. Were great works of philosophy judged by “who published it?” Merit is merit, and post-publication reviews, published or unpublished, can detect and reflect it.

Objections

At this point someone might object that I have been too generous to outside evaluators for hiring, tenure, and promotion. Letters of recommendation can be exaggerated and written irresponsibly. Often the letter writers include some whom the candidate requested. Evaluations may be to some degree dishonest or inflated. Further, an evaluator called upon to offer an opinion of a candidate’s oeuvre perforce does not examine each thesis and point with the same care as a referee. The evaluator will form overall opinions of individual works and cast them in the form of a few sentences. A truly conscientious evaluator will probe as deeply as a referee, and that happens, but it is not the rule.

If, however, we fear irresponsible external evaluations, would this not call into question the wisdom of relying on those letters altogether? If outside evaluations are the core of personnel decisions, we have to assume a basically honest and dependable system. As for the claim that an evaluator will not probe an article as a referee would, it doesn’t matter if this is the case. Awarding people tenure or promotion is not tantamount to proclaiming their positions correct and their arguments unassailable. If it were, then published criticism of a tenured professor’s article should be grounds for claiming tenure was a mistake, which needless to say it is not. Doing high quality work and being correct are not equivalent. Great philosophers often have views that most people reject, but this does not diminish the quality and significance of their work. Personnel evaluators can spot quality and importance.

A remaining argument for more heavily weighing certain organs of publications is status. Every department in a research university wants members of stature in the field, which top journals and presses create. Also, the impact of a work emanating from certain journals or presses is greater. Although not to be dismissed lightly, this argument gives an imprimatur to bias and snobbery that ought to be remedied. The perception of what constitutes status should itself change once publication in those journals or by those presses is seen not to necessarily spell higher quality. Letters to potential evaluators could emphasize that they are to comment on quality and not be influenced by provenance. Quality is quality and can be judged so by personnel evaluators; it cannot be discarded in the interests of prestige. Additionally, an invitation from certain editors might carry its own prestige.

Two concessions must be made. First, in hiring decisions, looking for candidates with peer-reviewed works in prestige journals or presses is a good mode of operating. It is much more efficient and cost-effective to pare down a large initial pool, with hundreds of candidates, by identifying applicants who have published in peer-reviewed journals, unless the invited articles listed are known to the search committee. And members of the search committee are not necessarily expert enough in the applicant’s field to assess those works. Second, the distinction between prestigious and non-prestigious journals is relevant when evaluators decide whether to be impressed by the quantity of a candidate’s publications. Ten papers in relatively undemanding journals may be less impressive than five in exacting ones. Hence, I am ready to countenance an adjustment of quantity requirements based on the journals’ rejection rates. Still, quality is quality no matter where it appears.

Should Unpublished Works Count?

My analysis confronts a number of challenges of the reductio ad absurdum variety. Wouldn’t my arguments on behalf of invited pieces apply to unsolicited papers in unrefereed electronic journals? Wouldn’t they support counting unpublished, privately circulated works as well as invited pieces? Why not just rely on outside evaluators in these cases too? In fact, why insist on candidates publishing their work at all?

Is the conclusion of these reductio ad absurdums really absurd? Publication is thought to be essential for disseminating knowledge. But what if someone frequently presents work in oral form to a large audience over Zoom? What if Fred e-mails or snail mails his unpublished work—good work—to every member of a professional association and pays them to actually read it? Won’t more people then read it than if it were in a print journal or digital venue? Some important papers (for example, David Kaplan’s “D-That” back in the 1970s) were originally circulated widely but not published immediately, achieving a degree of status and fame in the process. With so many scholars now posting work online in advance of print, this can happen more often. In short, content, not medium, is crucial.

A key advantage to the practice of publication is that it provides an access point for the scholarly community; by means of indexes, library records, citations, and so on, scholars both in and out of one’s field can locate the work for generations to come. Unpublished work cannot be as readily accessed this way. In other words, publication enables authors to share their work with the larger community—and with future generations.

At the same time, as others have pointed out, ideas can be perpetuated in forms other than traditional publication. Why would we not want all that we think to be passed on to future generations? Referees and editors shape their disciplines according to concerns and trends of their own time. But the interests, methods, and standards of the next generation may differ from ours, and work overlooked in our period may become vibrant and show new merit in another. We need to think about how scholars will think in later times, as well as how rapidly tides may turn in our own. Taking the long view leaves us wondering whether, by being as selective as we are and by privileging some modes of dissemination over others, we are neglecting future generations. Admittedly, there is a countervailing consideration: inundation. We serve future generations by preventing this consequence. I do not know how to strike the right balance here, but both lines of thought should be considered.

Let me emphasize that a candidate for tenure or promotion would be wise to submit material to prepublication peer review. Doing so prevents committees from judging that individual adversely and maximizes quality, since the author’s work would profit from peer review and from the extra incentive to do one’s best for referees. But after the fact, that is, if the person has published work in, say, response to an invitation, the work should be judged by postpublication review.

Summation

I have argued, first, that the practice of discounting certain sorts of publications and presentations in tenure and promotion decisions sits uneasily with the fact that those sorts of work contribute significantly to the profession. The scholarly community may have unfairly belittled quality work of capable scholars and discouraged some from placing their work in forums more likely to reach a large audience than refereed journals. My main point, however, is that in certain ways—notably by its ability to speak from hindsight—postpublication peer review is a more reliable guide to quality and of course impact than the refereeing process. Committees no doubt want their departments to have “status,” but if they would cease to view status as dependent on where something appeared and view it instead as dependent on postpublication reception, the objection to ignoring where the work appeared would fall away.

Peer review creates a highly competitive environment and therefore a higher quality set of submissions; it also improves work, making it likely that the published version of a manuscript will be better than what came over the transom. However, this does not settle the question of how to weigh items that were not subjected to competitive refereeing when we have other instruments by which to evaluate them.

To return to the misfortunes of Jean, let us hope that her next university will give Jean’s invited publications their due and accord greater weight to other publications and presentations that are the lifeblood of the profession and indispensable to its members.