By Richard Tol, University of Sussex, Vrije Universiteit Amsterdam, Tinbergen Institute and CESifo

Academics do many things. We teach, research, run departments, engage in public debate, and train young researchers. The outputs of some of these activities are routinely measured and thus play an important role in rewards and promotion. Other activities are rarely assessed and thus left to the intrinsic motivation of the academic. Research supervision is one. My recent work – summarized in this blog post – proposes a measure of the quality of advice to PhD candidates, the h1 index.

My proposed indicator of research supervision excellence shows that, by and large, good researchers make good advisors; but there are many people who excel at research but are less good at supervision, and many more that make great supervisors without being well published themselves.

This index could be an useful additional measure of university quality and transparency, by assisting students in choosing a university and PhD supervisor. Moreover if with measurement comes saliency this could be an additional  motivating factor for academics.

The index

A well-known indicator for research output  is  the h index, where h is the highest number of papers a researcher has published that have each been cited h times or more. In other words, an academic has an h index of 5, if she has published 5 papers which have each been cited at least 5 times. The h index is a simple way to combine the quality and quantity of research output. Quality without quantity is punished, as is quantity without quality. Invented only in 2005 by Jorge Hirsch, it has taken the world of research assessment by storm.

The h-index naturally lends itself to evaluating collectives. Frances Ruane and I (2009, Scientometrics, 80, 613-624) introduce the h1-index to quantify the quality of research supervision. A professor is a good PhD adviser if she trains many good researchers. We define a professor’s h1-index based on the h-indices of her supervisees; a professor with an h1 index of 4 has had 4 protégés each with an h index of 4 or more.

Data are a challenge. All universities keep records on PhD completions, but few do so in an accessible way let alone a structured database. Fortunately, academics are snobs and take pride in their lineage. (Jan Tinbergen was my advisor’s advisor’s advisor, so there.) Academic genealogies can be found. We used Alan Deardorff’s family tree of trade economists. There are 65 professors with 4 or more supervisees, for a total of 356 PhDs. Publication and citation data were taken from the Web of Science. The 12 best professors are listed in Table 1.

Like the h-index, the h1-index is a measure of lifetime achievement. It thus favours those with an earlier career start. Like the h-index is roughly proportional to the number of years since first publication, the h1-index is roughly proportional to the number of years since first graduation – see Figure 1. We can thus compare academics of different vintage by their h- and h1-rate, defined as the index over the number of active years.

Table 1. Selected characteristics of the 12 most successful advisers in trade economics

Figure 1. The h1-index as a function of the year of first graduation; dots are observations; thick central line is the expected value; thin lines are the 95% confidence interval.

Three professors stand out in that they graduate more high quality supervisees per year than most: Tony Venables, Gene Grossman and Ron Findlay.

By and large, good researchers make good advisors. There is a weak but positive correlation (0.41) between the h-rate and the h1-rate – see Figure 2. However, there are people who are excellent researchers but less good at research supervision. There are also excellent PhD advisors whose own research is nothing special. People are good at neither are excluded from the sample.

Figure 2. The h1-rate as a function of the h-rate; dots are observations; line is the least squares fit.

The flamboyant Harry Johnson stands out. He made important contributions to our understanding of international trade. He also trained a fair number of trade economists who themselves made important contributions. And he taught them how to be good PhD advisors themselves: Johnson’s students tend to have high h-indices as well as high h1-indices. Johnson thus has a high h-, h1- and h2-index; he was an excellent researcher, professor and grand professor.

There are two key problems with the analysis. Restricting attention to trade economists disadvantages those who also work in other fields. We did not control for student quality at the start of the PhD. That is, we did not measure the value added of research advice but rather intrinsic quality plus all education. We can get some idea, however, by looking at the institutions in Table 1. Some teach at top schools and have first pick from PhD candidates. Others do not, but nevertheless succeed in graduating excellent PhDs.

The results here merely illustrate our proposed indicator of excellence in research supervision. More systematic data collection is needed – see http://ideas.repec.org/top/top.person.students.html for another attempt – not only for a more comprehensive assessment but also for a better understanding of what makes a good PhD advisor.

Policy conclusions

Supervising PhD candidates is an important task. Yet, evaluations rarely go beyond the number of PhD completions with a few anecdotes about placements thrown in. The quality of research supervision can be measured, however, and should – at the moment, quantity is rewarded, and the incentive is thus to lower standards. The indicator proposed here should be interpreted with care. It takes time for a PhD candidate to manifest himself as a good researcher, and thus for a professor to manifest herself as a good supervisor. The h1-index is fine for celebrating past achievements. It should be used for current decisions only to the extent that present quality correlates with past glory.