Loading…

Academic ecosystem is damaged, here’s how we should restore it

I was halfway through my master’s in organisational psychology at University College London when I realised the ecosystem of academia is damaged and currently we are not doing much to restore it.

It all started when I learnt about p-hacking and replication crisis. P-hacking basically means selectively analysing data and only reporting results that support the hypotheses. In a survey of more than 2,000 psychologists, Leslie John from Harvard Business School discovered that more than 50% of psychologists had waited to decide whether to collect more data until they had checked the significance of their results, thereby allowing them to wait until their hypotheses are confirmed.

The survey also found that around 10% of research psychologists have engaged in data falsification, and more than half engaged in fraudulent behaviours such as reporting that a result was statistically significant when in fact it was not, or deciding between two different data analysis techniques after looking at the results of each and choosing the more favourable.

John, L.K., Loewenstein, G. & Prelec, D. (2012). Measuring the Prevalence of Questionable Research Practices With Incentives for Truth Telling, Psychological Science, 23(5), 524–532.

Although p-hacking in itself is not a fraud, it leads to publication of misleading findings. P-hacking would not be such a major issue, if studies in psychology and other social sciences were regularly replicated. If research group A finds an interesting pattern in data and research group B, following exactly the same procedure, does not, science would in a way correct itself. Unfortunately, not many replications are carried out.

Open Science Collaboration. (2015). Estimating the reproducibility of psychological science. Science, 349(6251)

A brilliant study by Brian Nosek and his colleagues published in Science in 2015 examined 100 cognitive and social psychology studies published in major journals. Only 39 of these studies were successfully replicated. Although the so-called ‘replication crisis’ is a hot topic in psychology, it’s not just psychology that suffers. In 2011, Bayer, the pharma company, examined 67 drug discovery publications and reported that three-fourths of them were not right.

To make matters worse, somewhere on the spectrum of scientific misbehaviours there is also scientific fraud. Take for example Professor Diederik Stapel of Tilburg University in the Netherlands, who faked data on at least 55 papers on topics such as the human tendency to stereotype or discriminate.

Looking for a needle in a haystack

Historically, most social sciences were about data collection. Information needed to learn about behaviours of individuals, groups of individuals, countries or regions took years to collect. This is no longer the case. It is estimated that by the year 2020, about 1.7 megabytes of data will be generated every second for each individual in the world. If today we are occasionally unable to tap into some of the existing data sources, often it is not due to lack of data, but due to privacy concerns, missing data sharing agreements or insufficient computational power.

Interestingly enough, in this era of data abundance, most social scientists continue to collect experimental data one experiment participant at a time. Unfortunately though, most of these studies do not generalise very well. Even when data is collected, we cannot be certain if it tells us something about the behaviours of white educated university students, who tend to participate in most experiments, or our diverse world.

Not long ago, I’ve read a book on dating by Christian Rudder, “Dataclysm”. Most of insights in the book come from the largest US dating site “OkCupid” and reflected real behaviours of people registered on the site. One could argue that findings still only generalise to people who actually use dating sites, but 10 million Americans who do are still much more substantial in terms of insight, than a study featuring a small sample of white, middle-class college students who are easily accessible to researchers.

Take for example a typical experimental study completed by Geoff Cohen from Stanford. In 2003 and 2006 he and his team worked with 158 black seventh-graders. Half of them were randomly chosen to write about something that was important to them, the other half wrote about something unimportant. The exercise lasted just 15 minutes, but those who wrote about their values had added 0.3 points to their GPA by the end of the term, closing the academic gap between their white peers and themselves by 40%. According to authors, this simple exercise breaks a vicious psychological cycle, experienced by many black students. Evidence shows that black students worry about the negative stereotype that they underperform at school, and that this worry causes so much stress that they actually end up underperforming. By asking the children to write about their values, authors believed to have mentally ‘vaccinated’ them against this stereotype threat.

Yet 158 students is not a large sample. So when Paul Hanselman from the University of California, Irvine tried to replicate results from the study twice with larger samples, he first found a much weaker effect of just 0.065 GPA increase and then discovered that the writing exercise had no effect at all.

Broken incentives

Many issues in academia are systemic because incentives for the people working at universities are broken. Some academic structures and processes, such as professional societies and journal publishing date back to the 17th century, but their suitability for the modern society is questionable. During my years at University College London, where I read psychology, I have heard many times that publications are the ‘‘currency’’ of science.

In some countries though, the analogy between publications and cash are taken to the extreme. To understand cash-for-publications system in China, Wei Quan from Wuhan University and colleagues surveyed the financial incentives offered by the top 100 Chinese universities and mined that data for interesting trends. They found that in 2016, the average reward for publication of a single paper in top science journals was almost $44,000 and the highest payment was $165,000. Although these figures are quite impressive in absolute terms, it is important to note that the average salary of a university professor in China was just $8,600 at the time of study.

Quan, W., Chen, B., & Shu, F. (2017). Publish or impoverish: An investigation of the monetary reward system of science in China (1999–2016). Aslib Journal of Information Management

Incentives to publish are increasing but so is journals’ hunger for novel, sexy findings. As a result, clickbait-worthy findings occasionally lead editors to overlook methodological flaws. Furthermore, the current system does not reward replication. Studies that replicate (or in fact fail to replicate) are by definition not novel and are therefore typically rejected (the so-called ‘file drawer’ issue).

https://www.nature.com/news/replication-studies-bad-copy-1.10634

According to the infamous Diederik Stapel, who faked data on at least 55 papers, the public fails to realise that academic science is becoming a business. In his interview for the New York Times, Stapel argued that “science is of course about discovery, about digging to discover the truth. But it is also communication, persuasion, marketing. I am a salesman. I am on the road. People are on the road with their talk. With the same talk. It’s like a circus.”

To make matters worse, academia is becoming more competitive. The production of PhDs has far surpassed demand for lecturers. According to Andrew Hacker and Claudia Dreifus, from 2005 to 2009 America alone produced more than 100,000 holders of doctoral degrees, but only 16,000 new professorships appeared in the same period. In such a competitive landscape, networking and lobbying play a crucial role in scientific career. Yet, academics who excel at gaming the system do not necessarily produce studies of the best standard.

On bridging the gap between academia and practice

I must have heard this phrase at least a thousand times by the time I graduated from university. Many academics do great research but don’t speak the language of practitioners. In other words, they are unable to explain how the research they conduct can solve real-life problems. Practitioners, on the other hand, often don’t have access to much of the tools and data that academics possess. As a result, they are often trying to reinvent the wheel, wasting resources. There are of course certain magazines and blogs that regularly introduce and explain some of the most recent studies to non-academic audiences, but this tends to be an exception rather than the rule.

In my senior undergraduate year, I attended a public lecture by a prominent academic and an advocate for a closer collaboration between research community and practitioners. I was shocked to learn that it took an average of nine years for scientific findings to be implemented in practice. Whilst not much science feeds into practice, most of decisions in the world of practice are not backed by any evidence. For instance, according to Peter Orszag of Obama administration and John Bridgeland of Bush administration, less than $1 out of every $100 of government spending is backed by even the most basic evidence.

This is not to suggest that the sole purpose of academic inquiry is to serve policy-makers or other practitioners. Yet, when practitioners don’t listen, hear or understand what academics are saying and academics continue to produce research that can hardly generalise due to small sample sizes, questionable research practices or outright fraud, a lose-lose situation is inevitable.

Towards a solution

Today we need science and evidence-based policies more than ever. If scientists gradually choose to subscribe to the postmodern ideology of clickbait and salesmanship instead of committing themselves to truth-seeking and scientific and moral integrity, we are in deep trouble.

In this era of post-truth politics, it is essential to ensure rigour and quality of each and every scientific publication, invest into science education and better communication between science community and practitioners and redesign flawed incentive matrices that promote quantity and novelty over quality. Universities should also acknowledge their institutional responsibility. The ‘publish or perish’ mentality that is currently deeply entrenched in most higher education institutions threatens the very foundations of scholarly pursuit and the quest for knowledge. In a way, science needs to return to its glorious dawn, when understanding how the world worked and conveying it to others was its primary, and potentially sole, purpose.

We also need to create a culture that promotes replication and other good practices widely discussed for some years now. Graduate students should be rewarded for completing replications in this way developing their skills and ensuring published research replicates well at the same time. Papers should require registration of studies, data and any other relevant materials that are important in determining the quality of academic submissions. Better training in methods and statistical literacy should be provided as part of all levels of university education. Interdisciplinary inquiry should be promoted. Practitioners and academics should be encouraged to forge meaningful partnerships that allow academics to rely less on government funding and source data from practice instead of collecting it from small samples that can hardly generalise.

Willingness to learn from mistakes, scrupulous testing and deliberate attempt to question methodological choices should be the foundation of science. Unfortunately, today many academics are driven by quick successes and are often tempted to skip these crucial steps. Such mentality only strengthens public suspicion and certain cynicism towards science and undermines belief in shared reality. In such a world, it is only natural for people to lock themselves up in echo-chambers and label scientific findings that they find unpalatable as biased, unreliable or simply fake. The irony is that sometimes they might be right.