How to improve assessments of publication integrity

In 2020, a systematic assessment of clinical-trial reports rang alarm bells. Out of 153 reports submitted to a single journal, Anaesthesia, 44% were found to contain false data1. This high proportion — which tallies with other estimates of the percentage of randomized controlled trials in medical journals that are unreliable — reveals a wealth of dubious publications in the public domain. Such papers can harm clinical practice and skew the direction of future research, clinical guidelines and systematic reviews.
Beyond clinical trials, as many as 5% of researchers have admitted that they have fabricated, falsified or modified data at least once2. A survey of scientists across disciplines in the Netherlands suggests that many others have made unintentional errors or engaged in questionable research practices — such as inadequate note taking during the research process or transparency about of study flaws and limitations in publications3.
When readers spot problems with a paper, they notify the publishing journal. It’s important that journals promptly label, correct or retract research articles that are found to be unreliable. When handling integrity concerns, journal editors and publishers routinely follow guidance provided by the Committee on Publication Ethics (COPE) — including 15 sets of guidelines, 49 flowcharts and 788 case studies.

Replication games: how to make reproducibility research more systematic

Although it is not a regulatory body, this global, publisher-funded organization holds much sway with journals and publishers. It provides a range of guidance on publication integrity, covering issues such as how to participate in peer review, how to respond to data fabrication and when to retract a paper. More than 14,000 journals are members of the organization (including Nature and its publisher, Springer Nature), agreeing to adhere to its ‘principles of transparency and best practice in scholarly publishing’. Publishers routinely advertise their membership as evidence of their commitment to publication integrity.
However, we argue that COPE guidance is not working adequately. For instance, five years after three of us (A.G., A.A. and M.J.B.), and our colleague Greg Gamble, raised overlapping concerns about 36 publications from a single research group4, journals that are COPE members had retracted fewer papers than had non-member journals (see ‘Mixed responses to integrity concerns’). Similarly, in July 2021, one of us (J.A.B.), together with colleagues, notified 78 journals of uncorrected errors in 686 laboratory-research publications on human genetics5. However, by December 2022, COPE member journals had published editorial notices for only about 6% (11 of 192) of these publications; statistically, no more than non-members (nearly 5% or 22 of 494).
We think that it’s too easy for journals and publishers to claim that they have followed COPE guidance, when — in our view — they have only paid lip service to publication integrity while sidestepping the harder work of assessing, flagging and resolving concerns.
Here we argue that updating and improving COPE guidance and flowcharts (see Supplementary information, Fig. S1, for example) could make the process of checking publication integrity more efficient and effective. We also propose five ways to do it.
Current COPE guidelines provide no time frames in which journals should publish an expression of concern or complete an integrity assessment. The guidance also recommends that the journal contact the authors for feedback — but gives no indication of how long editors should wait for a reply, or what constitutes an adequate response. Yes, complex investigations will occasionally take time. But in our view, under the current system, publishers can avoid due diligence by claiming that they have followed COPE guidance, yet do nothing or drag their heels if authors fail to engage, or engage inadequately.
In some cases, the recommended time frames for contacting institutions can cause delays. Take plagiarism, for instance — if neither author nor institution responds to concerns, a COPE flowchart recommends that journals “keep contacting the institution every 3–6 months”, potentially creating an endless loop. But institutions often struggle to provide timely, objective and expert assessments6, and focus heavily on perceived misconduct by their employees. They might be concerned mainly with protecting their reputations.

Retractions are part of science, but misconduct isn’t — lessons from a superconductivity lab

In our opinion, this lack of clarity can again allow publishers to pass the buck. For example, when the editorial board of the Journal of Bone and Mineral Metabolism recommended retracting 11 publications in 2020, the publisher, Springer Nature, refused to do so (go.nature.com/4fg9xes), on the basis that the university of the author in question had not investigated the matter, as was required under COPE guidelines. (Nature is editorially independent from its publisher.)
Thus, we suggest that COPE should recommend that journals issue editorial notifications as soon as their investigations begin. These announcements should be readily visible on the journal’s website and referenced in bibliographic databases. If no problems with publication integrity are found, the notice can be updated to show this.
And COPE should set time frames for decision-making. For example, a 2023 report on research integrity by the UK Parliament’s Science, Innovation and Technology Select Committee proposed that decisions about publication integrity should ideally be made within two months (see go.nature.com/3y4r2bq). The proposal was made on the basis of feedback from funding bodies, universities and researchers. In practice, we consider six months a reasonable time frame, with editorial actions undertaken immediately after the process is completed.
Two further recommendations could help journals to improve engagement from unresponsive authors and institutions. One is to require authors, when submitting a paper, to agree that they will engage with any post-publication assessments of integrity concerns in defined time frames, making them more likely to respond. Another is to make pragmatic decisions that favour protecting readers — defaulting to retraction, for instance — if no adequate response is received in that time frame.
Critics might argue that this would result in the retraction of some research that was sound, or that prolonged investigations are sometimes needed. But we think that improved assessments of publication integrity will minimize the risk of both.
Although COPE recommends that expressions of concern and retraction notices state the underlying reason(s), the guidelines don’t state the level of detail that is appropriate. In our view, this lack of information can prevent readers from evaluating concerns for themselves. COPE should recommend that all editorial notices clearly outline the nature and scope of the integrity concerns, the authors’ responses, the timelines of the assessment and of the decision-making process. In a similar way, COPE should recommend transparent disclosure of the concerns, responses from authors and institutions and details of assessments undertaken if concerns are resolved without needing correction or retraction.
Some might argue that publishing potential research-integrity concerns immediately might tarnish the reputations of scientists unfairly, should an investigation not find a problem. But we think it’s more important to prioritize the interests of readers and researchers whose work might build on the literature. If undertaken in a respectful way, using neutral language, early notification of integrity concerns need not harm reputations.
COPE guidance focuses on processes that journals and publishers should follow when communicating with affected parties — for instance, which parties should be contacted at each step of a journal’s investigation — rather than how to assess the veracity of concerns. It thus fails to assist journals in the key task of determining whether publication integrity is compromised and allows publishers to say they are compliant without undertaking adequate assessments.
We think that COPE guidance should be redesigned to help journals to perform systematic, methodology-driven assessments of publication integrity, at all stages of the publication process. This ‘how to’ guidance should be drawn up with the help of academics specializing in such methodologies.

Pay researchers to spot errors in published papers

In our opinion, recommendations should include that journals routinely collect and verify key study information at the manuscript-submission stage. Details of ethical oversight and institutional governance — location, research infrastructure, funding and more — could be assessed by staff members before peer review, and statistical advisers could be employed to check raw data. These checks would reduce the number of papers needing such assessment after publication.
COPE should recommend the use of tools for assessing publication integrity during peer review and after publication. There is a burgeoning literature on methods that assess publication integrity7–9, including checklists applicable to a broad range of publications10,11, which could be incorporated into guidance.
COPE should also recommend that journals and publishers set up independent assessment panels to investigate integrity concerns. These panels should be funded by, but be independent from, publishers and include members with expertise in the evaluation of publication integrity12. Examples of such panels exist, both to look into post-publication concerns (such as the one convened by the Journal of Bone and Mineral Research13) and to assess submissions, such as that set up by one of us (A.A.K.) to assess clinical-trial reports at Anaesthesia14.
Furthermore, if serious concerns about a paper are verified, we think COPE should recommend assessing the integrity of all potentially affected publications by the research group involved. From extensive experience, if one publication by a research group is found to have compromised integrity, there is an increased risk of others being similarly affected. In the automobile industry, if a few cars of a particular model are found to have problems, the manufacturers assess the rest — publishers should adopt a similar mindset.
In our view, the most important question for scientists and the public is whether research is reliable, not why it is unreliable. If journals, publishers and institutions focus first on investigating researcher malfeasance, it can cause long delays or prevent a paper’s reliability from being assessed.

Structure peer review to make it more robust

Yet COPE guidance enables conflation between publication integrity and scientists’ behaviour15. For instance, COPE’s March 2024 guidelines on interactions between journals and institutions mention “misconduct” 102 times, and “publication integrity” just 4 times (see go.nature.com/3wb3muh).
We think COPE should ensure that its guidance avoids accusative language such as ‘misconduct’ and ‘allegations’. Concentrating on the reliability of the study will lessen the need for journals and publishers to contact institutions before deciding on editorial action.
Critics might argue that this change of focus would make it less likely that misconduct would be exposed. But that need not be the case — after identifying integrity issues and correcting the literature, the reasons for compromised integrity can be examined.
As well as updating its guidance, we propose that changes in how COPE operates could benefit readers. We and others have, in the past, requested assistance from COPE in resolving integrity concerns that we felt had been inadequately addressed by journals and publishers, only to find responses slow — often taking several months and more than one enquiry — opaque and unhelpful. Some correspondence received no reply, and some responses acknowledged problems that were not followed up on without further prompting.
Efficient and transparent responses from COPE would reassure readers that requests for assistance are being taken seriously, provide readers with an opportunity to point out inadequate journal responses and improve the efficiency of journal assessments.
And although COPE can apply sanctions to its members by revoking membership, to our knowledge it has never done so. The use of sanctions, and making that action public, would probably improve performance of journals and publishers.
Finally, because COPE is financially dependent on membership subscriptions from the publishing industry, it’s important that the organization strives to avoid conflicts of interest. At present, nearly 80% of COPE’s 33 trustees and council members are currently or were recently employed in the publishing industry or being reimbursed for services to it. Reconfiguring the board to include specialists in publication integrity and more members who are independent of the publishing industry would reduce concerns about undue commercial influence and better protect readers’ interests.
Journals and publishers might argue that these recommendations would involve a lot of work. But just because it might seem daunting, doesn’t mean it’s not worth doing. The publishing industry is extremely profitable, and some of those profits should be invested in quality control.
It is past time that the maintenance and restoration of publication integrity were prioritized. Improving COPE guidance would help achieve that goal. Human health and the standing of science and academia are at stake.