Some people are speaking of a crisis in Cochrane. This so-called ‘crisis’ arises partly from the treatment of conflict of interest issues. Dealing with conflict of interest is an important issue for the evidence-based policy and practice. Cochrane is currently reviewing its conflict of interest (COI) policy, and Campbell will be following suit. So what is it all about and why do I care?
I care because I believe in evidence. I use James Wong’s evidence-based cookbook, adding mustard to my broccoli and black pepper to my turmeric. I wear compression stockings on long-haul flights and didn’t bother with the post-operative exercises after my knee operation because I read the Cochrane reviews. And my day job is to advocate for an evidence architecture for child protection, for nutrition, for early child development… for everything, building on what Cochrane and agencies such as WHO and NICE have achieved for health.
So I take the concerns which are being raised seriously. And my conclusion is, hold on a minute. What crisis? The current so-called ‘crisis’ presents an opportunity to up the game on research transparency, to further expose the attempts by the drug companies to distort the evidence. We won’t achieve that by destroying Cochrane, which has been one of the main champions of transparency in the last 25 years.
I would make four main points. First, Cochrane isn’t the bad guy here. It is the drug companies and others with vested interests who deliberately game the system to hide and misrepresent evidence not to their liking. Second, we should focus on issues, not personalities. And, third, there are indeed substantive issues which have been raised and should be addressed. But, fourth, the natural sciences remain far ahead of the social sciences with respect to all of this, so have some perspective.
Let me start on the last point. A big difference between natural and social sciences is the attitude to replication. In natural sciences the norm is for different research teams to attempt to independently replicate a result, uncovering error, freak results or outright fraud. In social sciences replication doesn’t count as ‘original research’ and so is not publishable in leading journals. That remains so despite the replication crisis in recent years when it has been found that a large proportion of results are not replicable and very fragile, because of errors, data mining or outright invention (e.g. LaCour and Green’s analysis of attitudes to gay marriage). It is only in recent years that some journals request the data and code behind the analysis, so the statistics have to be taken on trust. And publication of pre-analysis plans (protocols) remains the rare exception not the norm.
Lack of transparency and conflicts of interest
This lack of transparency matters since there are conflicts of interest in many areas of research. It is not just that researchers are attached to their results and don’t like to be contradicted. Branded programmes are big business in the US. These are education and social welfare interventions which are marketed to schools and social work departments across the US and increasingly overseas. These programmes are ‘evidence-based’, meaning there is an RCT showing the programme ‘works’. But that RCT is often conducted by the designers of the programme, without any pre-analysis plan and without making the data publicly available. This is a much worse state of affairs than in medicine.
Campbell supports the AllTrials campaign which is campaigning to require that all trials – not just those in medicine – are registered. Those complaining about certain Cochrane reviews should look at some of these studies – which are included in the relevant Campbell reviews. And several Campbell reviews confirm that studies by the programme designers find a bigger impact than independent studies.
This brings us back to Cochrane. Cochrane is a founding member of AllTrials. It has invested in building capacity to undertake and use reviews around the world. The problem is that drug companies, and the researchers in their pay, don’t play fair. They don’t register their trials, they suppress studies showing the ‘wrong result’, the selectively report outcomes in the studies they do publish, and they support ghost written articles – that is studies written by an industry consultant but published under the name of academics who are paid to allow their name on the paper to give the impression of independence.
These malpractices make it difficult for systematic reviewers to identify, assess and summarize all available relevant evidence on a particular topic. Cochrane reviews do try to make the best of a bad situation which is not of their making. Cochrane already has one of the toughest and most explicit COI policies of any scientific publisher. It does not allow industry support for reviews. Employees of companies with real or potential financial interests cannot be authors on reviews. Individuals who have received financial support from commercial sponsors with real or potential financial interests in the previous three years can only be included authors at the discretion of Cochrane’s funding arbiter if the majority of the review authors and lead author have no relevant conflicts of interests.
Both Campbell and Cochrane reviews analyse if there is publication bias by testing for ‘missing studies’ with no effects (and a work in progress on a methods paper for Campbell finds that non-Cochrane/Campbell reviews don’t do this). The risk of bias assessment in Cochrane reviews assesses outcome reporting bias.
What more can be done
So the question is should they do more? The more they might do concerns both the treatment of potentially conflicted studies in Cochrane reviews, and dealing with conflict of interest amongst review authors.
On the first of these issues: Should they not include studies with no published protocol? Should they exclude ghost-written articles? Should they exclude studies for which the data are not in the public domain? Should they exclude studies in which the author had a conflict of interest? The answer to all these questions is no.
No, they shouldn’t, as an important principle in the review process is the distinction between the study inclusion criteria and critical appraisal. Studies are screened against the inclusion criteria, and then critical appraisal conducted on included studies. So studies with no protocol or which are ghost-written should get included if they satisfy the PICOs. But it is worth considering whether the risk of bias assessment should be refined to take account of these issues. There has been discussion as to whether funding sources should be part of the risk of bias assessment – with views both for and against.
Other issues for reconsideration are studies for which the data are not in the public domain, or studies with no protocol. Such studies may have a high risk of bias, or at least an unclear risk of bias. Ideally, the assessment should be based on the manifestation of the possible bias, e.g. selective outcome reporting. But that is not possible in the absence of a study protocol. Excluding such studies from the analysis would send a clear signal regarding research transparency and may have important systemic effects.
Then there are possible conflict of interest issues. Cochrane is currently reviewing its policy on these. These conflicts may apply to included studies in reviews or to the reviews themselves. For included studies an assessment of conflict of interest is part of the critical appraisal and could be used more proactively in excluding the studies which are compromised by such conflicts. In the case of reviews, at the extreme there are researchers writing reviews commissioned by drug companies to deliberately distort the evidence.
Clearly Cochrane shouldn’t be publishing such reviews and its contract of interest policy rightly identifies and excludes them. But there are a broader range of conflicts – having received drug company sponsorship in the past for another study, or attending an event they sponsored, or being the author of an included study. These factors may all cause bias. Campbell’s policy is presently that all conflicts have to be declared. But that it is all. It is far weaker than Cochrane’s current policy – we are reviewing our policy at the same time as they are reviewing theirs.
Cochrane has the opportunity to revise the scope of critical appraisal to minimize the risk of distortion by the shenanigans of the drug companies. And Campbell is trying to catch up.