Five Essays on the Quantification and Measurement of Intellectual Achievements

Abstract

This dissertation is a collection of five research papers written during my doctoral studies at the University of Konstanz between October 2007 and November 2011. The first four studies cover bibliometric topics whereas the last study deals with grading university students. All five studies contain empirical work, yet the main focus of the third and the fifth paper is theoretical. The first article analyses different aspects of the methodology of the Handelsblatt Ranking of the year 2007. The second article evaluates the research productivity of universities in business economics in Austria, Germany, and the German-speaking part of Switzerland. The third article presents a new method to evaluate research which provides a common basis for valuating research from different scientific disciplines. The fourth article investigates determinants of editing learned journals. Finally, the fifth paper presents a method to increase the precision of grades as a predictor of student ability. The paper also contains a case study. In the following I briefly summarize the main results.Chapter 1 is a reprint of a joint article with Prof. Heinrich Ursprung (University of Konstanz). The article „Das Handelsblatt Ökonomen-Ranking 2007: Eine kritische Beurteilung“ appeared in the Perspektiven der Wirtschaftspolitik, Vol. 9, No. 3, pp. 254-266, in 2008. We examine the methodology of the Handelsblatt Ranking which is currently the most recognized ranking of academic research in economics in Austria, Germany, and the German-speaking part of Switzerland and which is published on a yearly basis. Because it is frequently used as a research indicator the Handelsblatt ranking needs to be incentive compatible. We argue that (i) selecting a small set of journals instead of considering all journals, (ii) giving a bonus for co-authorship, and (iii) ignoring the length of an article distorts the research process and does not reflect of research achievements correctly. We analyze the latter for each of these aspects separately and show that the rankings of some researchers differ significantly depending on which method we use. We further show that these differences persist also at the aggregate level of departments. Finally, we argue that the Handelsblatt should focus more on research output per professor than the total output of a department as the latter is highly driven by the number of department members and is not necessarily related to high average productivity.Chapter 2 is joint work with Prof. Oliver Fabel (University of Vienna) and Miriam Henseler, née Hein (DFG). Our paper “Research Productivity in Business Economics: An Investigation of Austrian, German and Swiss Universities” was published in the German Economic Review, Vol. 9, No. 4, pp. 506-531, in the year 2008. We draw on a new dataset that collects the research output of business economists employed by Austrian, German and Swiss universities in spring 2008. The data set comprises publication records and personal data of roughly 1,800 scientists. We compute research rankings of departments and identify the leading departments in selected subdisciplines. Our results indicate that productivity differences between departments are relatively small and concentration of research output across departments is low. Using Tobit and Hurdle regressions, we investigate how institutional design and individual characteristics affect research productivity. We find that research productivity increases with department size as measured by the number of department members and with the number of a department's professors who actively publish. Moreover, productivity is higher in departments that run an economics study program. In line with the life cycle hypothesis we observe that the productivity of active researchers decreases with higher career age. Female business economists appear to be less productive than their male peers. It should be noted, that the paper has received considerable attention in the scientific community and gave rise to two comments. In two replies we show that our results are robust with regard to the points raised in these two comments.Chapter 3 develops a new method to evaluating research and is also available as a working paper of the University of Konstanz. In this paper I adopt the notion that the ultimate aim of research is not to gain intellectual insights but to contribute to the well-being of mankind. Therefore I propose a generational accounting approach to valuating research. Based on the flow of scientific results, a value-added (VA) index is developed that can, in principle, be used to assign a monetary value to any research result and, by aggregation, on entire academic disciplines or sub-disciplines. The basic idea of the VA-index is to distribute the value of all applications that embody research to the works of research which the applications directly rely on, and further to the works of research of previous generations which the authors of the immediate reference sources have directly or indirectly made use of. Thereby a piece of research is valued at its final contribution to utility. The major contribution of the VA-index is to provide a measure of the value of research that is comparable across academic disciplines. To illustrate how the generational accounting approach works, I present a VA-based journal rating and a rating of the most influential recent journal articles in the field of economics.Chapter 4 is an article written with Matthias Krapf (University of Vienna). The article “How Do Editors Select Papers, and How Good are They at Doing It?” appeared in The B.E. Journal of Economic Analysis & Policy, Vol. 11, Iss. 1 (Topics), Article 64. We use data on the B.E. Journals that rank articles into four quality tiers to examine the accuracy of the research evaluation process in economics. We find that submissions by authors with strong publication records and authors affiliated with highly-ranked institutions are significantly more likely to be published in higher tiers. Citation success as measured by RePEc statistics also depends heavily on the overall research records of the authors. Moreover, when controlling for the research topic as defined by JEL codes, we find that women receive significantly more citations than men. Finally and most importantly, we measure how successful the B.E. Journals' editors and their reviewers have been at assigning articles to quality tiers. While, on average, they are able to distinguish more influential from less influential manuscripts, we also observe many assignments that are not compatible with the belief that research quality is reflected by the number of citations.Chapter 5 is the current version of a joint paper with Prof. Heinrich Ursprung (University of Konstanz) in which we develop a new method to standardize grades when not all students take the same exams. Our method uses regression analysis to control for differences in the difficulty of individual exams to arrive at more informative grades. Our approach relies on the idea that the difference between a student's ability and the received grade reflects the difficulty of the exam. We estimate the students' abilities via a system of equations, i.e. we derive the abilities endogenously. A key feature as compared to other methods to standardize grades is that we do not only compute a standardized grade point average (GPA) but also provide standardized grades at the level of individual exams. Using genuine examination data, we illustrate that our approach fares better than ECTS grading. Moreover, we show that grading standards differ significantly between different exams and, arguably, different courses. If grades are not standardized in a sensible way, students are likely to choose soft courses. Standardizing grades eliminates such distortions and will lead to a more efficient selection of students into courses.