Thoughts on Research Excellence – Part 1: The conflict between research ideals and academic working conditions

I hear it everywhere, but what is “research excellence”? 

The term “excellence” keeps floating around in the research ecosystem. In Germany, when people talk about excellence in the context of research, they usually refer to the Excellence Strategy — a funding scheme that gives German universities lots of money and prestige. In the UK, the “Research Excellence Framework” evaluates institutions and the Horizon Europe’s ERC selects projects based on “scientific excellence”. Since the term research “excellence” is so frequently used, one could expect it to be a well defined term with clear criteria of what makes research excellent. Unfortunately, it is not. In a nice review article, Jong et al. show that the term “excellence” is in fact very poorly defined, its origin is the meritocracy movement of the Western world and researchers often assume they can simply recognize excellence “when they see it”.

So, how do people recognize excellent research?

The lack of clarity regarding the definition of excellence becomes apparent when I ask researchers for characteristics of excellent research in my workshops. Sometimes, I get answers such as “innovative” and “interdisciplinary”, but generally, what people list is indirect indicators of excellence — proxies. A proxy is a stand-in, something we use to measure a concept that is hard to define or assess directly. In this case, proxies are things like:

• Whether research is published in high-impact journals
• Whether it brings in competitive grant funding
• Whether it’s cited often
• Whether it raises the profile of a university or institution

These proxies serve a practical purpose and at first glance, the proxies seem reasonable. High-impact journals are very selective about the research they publish. Funders also use elaborate peer-review procedures to identify proposals with high chances for success. Research that is ground-breaking or useful for a field is more frequently cited, and research that contributes to solving society’s problem can be showcased by institutions and raise their profile.

Yet, something appears to be going wrong. There seems to be a discrepancy between the pursuit of “excellence” that is in particular happening in German institutions and the repeated reporting of misconduct, fraud and power abuse. So what is happening?

Ulrich Dirnagl postulates that certain personality traits — the so-called “dark traits” of narcissism, Machiavellianism, and psychopathy — are quite successful at scoring in these indirect proxies, while leaving a lot of damage along the way. These traits are associated with overconfidence, a need for admiration, extreme self-centeredness, manipulative behavior, and a cold, strategic mindset. While it makes sense that such strong personality traits can lead to power abuse and misconduct, I believe there are also more subtle mechanisms at play that affect all of us and that lead to broader problems with research integrity. These have to do with our proxies for excellence. Goodhart’s Law explains this well.

So what does this law have to do with bad science?

Goodhart’s Law is a model from economy that can be widely applied to any situation where success is measured. It has been rephrased to “When a measure becomes a target, it ceases to be a good measure.”

For individual researchers, the proxies for excellent research are usually based on their publication list: number of papers, impact factor of journal, h-idx etc.  Grant income matters too, but this is also largely dependent on having a competitive publication list. So, as an individual researcher, we get pressure to publish, pressure to get grants, pressure to compete. And what we are seeing systemically is a growing number of paper retractions (these can be due to honest errors, but also a result of carelessly conducted work or intentional misconduct, such as manipulating data and results) and fake citations being sold online.

While research fraud and buying fake citations online are clearly huge problems, most of us do not engage in such practices. However, Goodhart’s law still has an effect and the proxies for research excellence incentivises to take shortcuts that we are often not even aware of or that are easier to rationalise.

So people cut corners. This is often called “questionable practices”. Some examples are:

• Cherry-picking: Only reporting the results that support your hypothesis and hiding the rest.
• P-hacking: Testing multiple ways to analyse data until one gives significance.
• HARKing: Hypothesising after results are known, so exploring your data until you find something and then pretending you had a hypothesis about that finding all along.
• Salami slicing: Splitting one study that could be easily published in one paper into multiple papers.
• Guest authorship: Adding people to the authorship list who didn’t do work that qualifies for authorship.
• Manipulating peer review: For example by nominating reviewers who are allies.

These behaviours are not always the result of conscious and intentional cheating. Instead, some of these are deeply engrained in our scientific culture and PIs have learned them from their own PIs and pass them on to the new generation. This happens because most often they come with seemingly convincing arguments: “A stronger narrative of the paper makes it easier to follow, these ambiguous results would just make it difficult for the reader and would make the paper way too long.” or “We spent a lot of money on acquiring these data, we have to publish some results now, let’s try some more analyses.” or “If we do not add our boss’ name on the paper, she will not be happy and we may not be able to use the equipment anymore.” How can researchers, whose future career depend on outputs, argue against this?

Most researchers are intrinsically motivated, surely, these behaviours are exceptions!?

None of this is rare, even though reliably estimating prevalence is not easy. A meta-analysis by Fanelli (2009) estimated 2% of researchers admitted to misconduct, and about 34% to questionable practices. Gopalakrishna et al. (2022) found that in the Netherlands, 4% of researchers admitted to misconduct, and over 50% said they regularly use questionable methods. Martinson et al. (2005) found similar numbers in the U.S. 

I strongly believe that these numbers are not a result of most researchers having dark traits. And most likely, no scientist has started their careers by planning to fake data or deceiving others. Most of us are motivated, curious, and ambitious. But we are also under a lot of pressure and often have unmanageably high work loads. There’s the short-term contract. The next grant deadline. The journal that wants a clean story. And what is behind all this is a survival instinct. Because if you do not succeed with the proxies, then you may be out of the system. Or lose your employees.

One striking example of how the combination of ambition with work place pressure can escalate is seen in the following quote by a researcher who was found guilty of falsifications and fabrications (source: NIH document).

“First, I believed that because the research questions I had framed were legitimate and worthy of study, it was okay to misrepresent “minor” pieces of data to increase the odds that the grant would be awarded to UVM and the work I proposed could be done. Second, the structure at UVM created pressures which I should have, but was not able to, stand up to. Being an academic in a medical school setting, I saw my job and my laboratory as expendable if I were not able to produce. Many aspects of my laboratory, including salaries of the technicians and lab workers, depended on my ability to obtain grants for the university. I convinced myself that the responsibility I felt for these individuals, the stress associated with that responsibility, and my passion and personal ambition justified “cutting corners”. Third, I cannot deny that I was also motivated by my own desire to advance as a respected scientist because I wanted to be recognized as an important contributor in a field I was committed to.”

Carol Tavris and Elliot Aronson have an interesting theory for how such a situation can evolve: we start with good intentions, and then under pressure take some small decisions into a non-ethical direction. To minimise the resulting cognitive dissonance, the uncomfortable feeling of having violated our own values, we are rationalizing our behaviour. The next time, we take another step and rationalise again. At some point, there is serious misconduct and from the outside, it seems unexplainable how this could have ever happened.

But isn’t bad or unethical science easily caught by the system and corrected?

People often say “the system self-corrects.” But we know the correction mechanisms are slow and unreliable. By the time Wakefield’s fraudulent 1998 study on vaccines was debunked by a journalist, his harmful ideas had spread wide and far. A whole research field on the link between 5-HTTLPR gene and depression was elaborately established in over 20 years through more than 400 statistically underpowered studies before a methodologically very thorough study seriously cast doubt on all previous findings. Recently, it came to light that a lot of the dementia research that had provided the direction for the entire research field was partly based on manipulated images. Whilst in these examples, the self-correction has at least been initiated, the effects on the entire field and on society had already been profound.

On one hand this also has to do with the strong reliance on the peer-review system (also see my post on rethinking quality assessment). On the other hand, the badly needed replication studies are still undervalued and rarely rewarded. Since research always builds on prior work, and journals tend to favour positive results, this combination creates a structural vulnerability in the system.

What role do institutions play in all this?

Institutions themselves neither have the motivation nor resources to question or double-check their researchers work. Instead, recruitment and promotion criteria, especially for professors, often build on the assumption that excellent research and only excellent research gets accepted by prestigious journals. And that excellent researchers are recognized by funding agencies and therefore get money.

Of course, if we see research institutions as profit-oriented organisations, this hiring strategy makes a lot of sense. Institutions want people who can bring in money, so that these people then do bring in money. Even though a lot of higher education institutions are technically public and not profit-oriented organisations, the sad truth is that they do need to care about money to keep the organisation going. The current political situation that leads to states investing more money in defence, will not make this easier.

So is there a way to do better?

Since institutions partly outsource their definition of excellence to funders and journals, these actors play a key role in shaping how excellence is operationalised. These definitions often include methodological soundness of a paper or proposal, but also the perceived novelty and attractiveness of the topic, positive results and the track record (typically the publication list) of the authors. In the current academic system, we are dependent on these external assessments of our “excellence”.

While approaches to evaluating research output are evolving (also see my blog post on research evaluation reforms), progress is often quite slow. And given that most researchers start off being very motivated and idealistic, institutions should support them to do responsible and meaningful research, whilst also producing outputs required by the system. This includes enabling more efficient and sustainable ways of working through self-management, effectively organised teams and the targeted use of technology.

In Part 2 (to come soon), I will outline my concrete ideas about how researchers and institutions can align meaningful work with the structural realities of the system.