REVIEW ARTICLE
THE CHANGING FACE OF CLINICAL TRIALS
Jeffrey M. Drazen, M.D., David P. Harrington, Ph.D., John J.V. McMurray, M.D., James H. Ware, Ph.D., Janet Woodcock, M.D., Editors
- Article
- References
- Citing Articles (1)
- Comments (1)
- Metrics
- Pragmatism in clinical trials arose from concerns that many trials did not adequately inform practice because they were optimized to determine efficacy.1 Because such trials were performed with relatively small samples at sites with experienced investigators and highly selected participants, they could be overestimating benefits and underestimating harm. This led to the belief that more pragmatic trials, designed to show the real-world effectiveness of the intervention in broad patient groups, were required. Medical researchers, both academic and commercial, must deliver health care innovations (drugs, devices, or other interventions) that are safe, beneficial, and cost-effective, and they must identify the subgroups for whom the innovation will provide the greatest benefit relative to risk. A broad view of an intervention, including approaches to improve its effectiveness, is critical. An ideal trial includes a population that is relevant for the intervention, a control group treated with an acceptable standard of care, and outcomes that are meaningful, and it must be conducted and analyzed at a high standard of quality. Pragmatic trials frequently include complex interventions, sometimes consisting of several interacting components2 and often involving the skills and experience of one or more health care professionals to deliver the intervention — for example, surgeons, physiotherapists, or cognitive behavioral therapists.In this article, we do not provide a definitive exposition of the methods used for pragmatic trials. Rather, we explore the contexts in which a pragmatic design is most and least attractive and identify the strengths and limitations of — and challenges in implementing — pragmatic trials.
WHAT IS A PRAGMATIC TRIAL?
Schwartz and Lellouch1 proposed a distinction between explanatory trials, which confirm a physiological or clinical hypothesis, and pragmatic trials, which inform a clinical or policy decision by providing evidence for adoption of the intervention into real-world clinical practice. The original PRECIS (Pragmatic–Explanatory Continuum Indicator Summary) tool3 attempted to clarify the concept of pragmatism and provided a guide, scoring system, and graphical representation of the pragmatic features of a trial. Features included the recruitment of investigators and participants, the intervention and its delivery, follow-up, and the determination and analysis of outcomes. Many trials could be deemed to be pragmatic with regard to at least one of these dimensions, but few are truly pragmatic on all dimensions. Pragmatism has been discussed widely,4-20 and a special issue of Clinical Trials had 12 articles focused on ethical and regulatory issues in pragmatic trials.21 The requirements for pragmatism were loosened substantially in PRECIS-2,22 and a pragmatic extension to the CONSORT statement has been proposed.23 Key dimensions for assessing the degree of trial pragmatism, following PRECIS-2, are provided in Table 1TABLE 1Nine Dimensions for Assessing the Level of Pragmatism in a Trial, as Proposed in the Pragmatic–Explanatory Continuum Indicator Summary 2 (PRECIS-2) Tool.. The trials that are used as pragmatic exemplars throughout this article are summarized in Table 2TABLE 2
Exemplar Trials and Their Main Pragmatic Features..
CHALLENGES TO PRAGMATISM AND POTENTIAL SOLUTIONS
Recruitment of Study Participants
Pragmatic trials require that participants be similar to patients who would receive the intervention if it became usual care, which may be unknown for new interventions. Participation in trials has fallen over time; for example, among persons without established disease, a lower than 10% rate of response to a screening invitation is common. The fact that volunteers participating in certain types of trials are often healthier than persons in the general population (the “healthy-volunteer effect”) and competing recruitment from other trials, particularly in academic centers, undermine attempts to achieve generalizability. Financial incentives associated with recruitment to industry trials can substantially affect recruitment to less-well-funded academic trials. Minimization of inclusion and exclusion criteria and reduction in the number and complexity of study visits, study procedures, and questionnaire burden are important but are likely to be only partial measures to increase participation in trials.In this regard, the development of large, simple trials (e.g., the Heart Protection Study24 and the Corticosteroid Randomization after Significant Head Injury [CRASH] trial25) has been important. The Thrombus Aspiration in ST-Elevation Myocardial Infarction in Scandinavia (TASTE) trial,26 a trial of thrombus aspiration versus usual best care before percutaneous coronary intervention, involved 7244 participants and had a primary end point of 30-day all-cause mortality. The trial used a national registry (the Swedish Coronary Angiography and Angioplasty Registry) and achieved high participation because of the simple design and absence of a need for additional follow-up. The trial did not show a differential response to the treatments.Informed consent is a barrier to unselected participant recruitment. To guarantee that everyone who is eligible is included, this requirement would need to be waived. In some contexts, it is possible — subject to ethics approval — to conduct trials without consent or with modified consent. In the Post–Myocardial Infarction Free Rx Event and Economic Evaluation (MI FREEE) cluster-based trial,27 2980 sponsors of health care plans were randomly assigned to provide either usual prescription coverage or full prescription coverage, with a primary end point of a first major vascular event or revascularization among patients. In this trial, which required consent from plan sponsors but not from patients, the elimination of copayments for drugs that were prescribed after myocardial infarction was not associated with a significantly lower rate of the primary outcome. A trial involving 6394 participants was conducted to assess the effect of emergency short-term use of antiseptic-coated versus antibiotic-impregnated versus plain latex catheters with regard to the primary outcome of the incidence of symptomatic urinary tract infection for which an antibiotic was prescribed within 6 weeks.28 After the initial admission, prospective consent was obtained according to usual practice from participants who were undergoing elective procedures, and retrospective consent was obtained in cases of emergency admissions, thus maximizing the generalizability of the findings. Routine use of antibiotic-impregnated or antiseptic-coated catheters was not supported by the results of this trial.In the CRASH trial, more than 10,000 patients with head injury and impaired consciousness underwent randomization to determine whether glucocorticoids, as compared with placebo, affected the rates of death and neurologic disability. The trial was stopped early because of evidence that glucocorticoid treatment was associated with higher mortality.25 In CRASH, the nature of consent depended on local ethics decisions, with consent waivers or consent from a legal representative being allowed in some cases. If a trial neither interferes with normal clinical care nor adds nonstandard activities or data collection, the objection to waiving consent is reduced. In low-risk contexts, random assignment of patients to alternative established treatments may be possible without obtaining consent,19 as might a trial with a cluster design in which physicians are randomly assigned to prescribe only one of the alternative treatments.Cluster randomization, as in MI FREEE,27 which involves groups of patients (in the same health care facility) who are randomly assigned to the same intervention, is popular in pragmatic trials. Cluster–cluster trials assess outcomes aggregated at the cluster level, whereas cluster–individual trials assess individual-level outcomes. Cluster–cluster trials offer greater possibilities of waiving the need for consent at the cluster-member level.21,29 Cluster–individual trials offer the option of waiving consent for the intervention, with consent obtained only for participant follow-up. This approach was implemented in A Stop Smoking in Schools Trial (ASSIST), a cluster-randomized trial of a high school smoking-prevention intervention with a primary outcome of smoking in the past week.30 The results of the trial suggested that the ASSIST intervention could lead to a reduction in adolescent smoking prevalence of public-health importance.The ongoing High-Sensitivity Troponin in the Evaluation of Patients with Acute Coronary Syndrome (High-STEACS) trial31 is investigating the clinical implications of a high-sensitivity troponin I biomarker for the diagnosis of myocardial infarction, with a primary outcome of cardiovascular death or recurrent myocardial infarction within 12 months after admission. Like MI FREEE, this is a trial of policy change. It uses a stepped-wedge cluster design32 in which all sites transition from the control to the active intervention but with randomized assignments to the timing of transition, with some sites assigning patients before others. Such trials assessing a policy that is going to be implemented in any event arguably offer the greatest potential for pragmatic trials, since they require no individual consent while allowing for some degree of control of ecologic changes in care that may be happening simultaneously.In summary, pragmatic trials face some unique challenges along with many of the same challenges that are associated with traditional explanatory trials. Strategies to enhance recruitment have been proposed.33 When appropriate, various forms of cluster randomization offer advantages and may help avoid the need for informed consent. Disease registries provide cohorts of patients who have already given consent for registry inclusion, which facilitates recruitment and follow-up. A related approach is the cohort multiple-randomized design,34 in which a cohort of participants is recruited and consent is obtained for follow-up and possible recruitment into trials of new treatments versus standard care. In any particular trial of this type, additional consent is obtained only from participants who are randomly assigned to the new intervention, which reduces the concerns of participants who have been randomly assigned to usual care.Recruitment of Investigators
Trials need investigators to take responsibility for recruitment, treatment, and follow-up of participants. Many health care professionals outside of academic centers do not participate in clinical trials, in part because of the time pressures associated with their clinical duties or because they do not consider research to be a key component of their job. Hence, the investigators involved in a trial will often not encompass the heterogeneity of practice that is present in usual care. In contrast, investigators across Sweden who were contributing to a national quality registry were included in the TASTE trial.26Good trials include a variety of investigators with a representative mix of experience appropriate to the intervention under study. The trial of short-term use of antiseptic-coated versus antibiotic-impregnated versus plain latex catheters28 made substantial efforts to include a heterogeneous group of hospitals, specialists, and surgical procedures. Despite these examples, this is a dimension on which many trials fail the pragmatism test. A pragmatic approach is easier when an intervention is implemented at a group level rather than at an individual level — this is one reason that pragmatic trials commonly incorporate cluster randomization. In ASSIST, only 113 of 233 possible schools (48%) expressed an interest in participating in the trial.30 The percentage of potential clusters agreeing to take part will vary according to the trial context. A trial that is run by an overarching authority may achieve much higher participation. For example, a hospital could insist on the full involvement of all wards in a trial of approaches to infection control.However, the wrong type of heterogeneity can be harmful. For example, if participants in many countries with poorly developed health care systems are enrolled in a trial assessing the effect that primary care nurses monitoring patients with heart failure have on reducing emergency admissions for heart failure, the trial would not inform the implementation of a role for such nurses in a developed health care system. Likewise, if an intervention involves substantial technical expertise, then that intervention should be delivered by practitioners with an adequate throughput of patients to enable them to maintain their levels of expertise. This is particularly true in surgical trials, in which complex surgery is increasingly delivered in high-volume centers. This creates a conflict in the design of pragmatic trials. Should we conduct a trial in the health care environment that currently exists or in a context representing the health care environment that is likely to exist in the future in the relevant specialist area?If heterogeneity in responses to the intervention is likely, a trial must be large enough to permit an understanding of that heterogeneity; this may require a substantial increase in sample size to detect a treatment-by-subgroup interaction. Often, there will be power to detect treatment-by-subgroup interactions only in large meta-analyses conducted at the individual-patient level.Establishing a critical mass for efficient trial conduct is crucial. Providing incentives to investigators is important in the face of increasing demand to deliver clinical services more efficiently, since research takes additional time beyond standard clinical care. The development of clinical networks and establishment of disease-specific research communities is one way forward. Another would be to give credit to health professionals for research as a key component of professional work plans. In the United Kingdom, these approaches, along with the creation of a national network of clinical-trial units that have been registered as fit-for-purpose, has improved the recruitment and retention of clinical investigators and methodologists working together to deliver trials by avoiding the common approach of setting up a network to deliver a single trial that is then not reused for future trials.35The Intervention and Its Delivery within the Trial
A trial with blinded interventions is not fully pragmatic. In pragmatic trials, the randomly assigned group is commonly not masked. Efforts that are made to minimize biases in open trials include focusing outcomes on major events, such as death and emergency hospital admissions. This approach has been used in the Prospective Randomized Open Blinded End-point (PROBE) trials,36such as the Anglo-Scandinavian Cardiac Outcomes Trial–Blood Pressure Lowering Arm (ASCOT-BPLA) trial37 and the Systolic Blood Pressure Intervention Trial (SPRINT)38 of the effect on cardiovascular events of different strategies for lowering blood pressure. However, the reporting of nonserious adverse events, reasons for treatment discontinuation, and many patient-reported outcomes are subject to greater degrees of bias in open trials, which affects the quality of the trial. The Initial Antidepressant Choice in Primary Care trial,39 a policy trial of fluoxetine versus tricyclic drugs as first-line therapy for depression, assessed the consequences of the initial choice of an antidepressant agent under usual care conditions; adverse events were a main outcome, and the open nature of the trial could have compromised the integrity of this outcome. In the trial, clinical and quality-of-life outcomes and overall treatment costs provided no clear guidance regarding the initial selection of fluoxetine or tricyclic drugs. The CRASH trial involved a placebo control and blinding; nonetheless, it had many pragmatic elements. In many situations, the need to avoid reporting bias will override purist pragmatic considerations, making blinding the optimal approach. In complex intervention trials, in which blinding the intervention is often impossible, it is usually possible to blind the assessment of outcomes.36 In any trial, the advantages and disadvantages of blinding must be considered; blinding is particularly important when the reporting of key end points or safety events could be biased in an open trial.In pragmatic designs, the intervention should be delivered as in normal practice, by staff with typical experience and with the use of routinely available equipment. The MI FREEE trial27 tested a treatment policy by assessing drugs within a class, but decisions with regard to the specific drug and dose within that class were left to the investigators. (A pragmatic trial often investigates a general approach to treatment rather than dictating the specific details of that approach.) The degree of support for participants in treatment persistence (i.e., in ensuring that participants continue to undergo the treatment) can influence outcome. Traditional trials rely on study visits that involve discussion of adherence and recording of laboratory tests for safety, as well as other investigations beyond normal practice. A trial that is dominated by poor adherence to the protocol or poor delivery of the intervention is of limited use. Ideally, a balance would be achieved, and both the intervention and its mode of delivery would be taken into consideration. Investigators should be given basic advice on how to achieve good outcomes for participants, as well as reasonable levels of training in new interventions within the constraints of the environment in which the trial is conducted.The Nature of Trial Follow-up
The unobtrusive collection of trial outcomes is attractive; it reduces the burden on the participants and investigators without introducing artificial aspects to follow-up. Such a strategy is most feasible in health care systems with reliable and accessible electronic health records that capture the events of interest. This might be achievable where there is a unified electronic health care record, but it is at present challenging in many countries. The High-STEACS trial,31 which has no trial-specific data-collection visits at all, illustrates the potential of this approach. Likewise, MI FREEE27 followed participants through a health care database, with outcomes determined algorithmically. Linkage of trial records to routinely collected health records in the West of Scotland Coronary Prevention Study (WOSCOPS)40-43 illustrates the benefits of using health records to identify serious adverse events and that their use might replace traditional within-trial end-point determination, as well as in evaluating long-term poststudy safety, efficacy, and cost-effectiveness. An attractive alternative to trials in which electronic health records are used can be found in trials of alternative interventions involving patients who are already enrolled in disease-specific or intervention-specific registries that incorporate detailed patient phenotypes and long-term follow-up data. This framework provides an efficient and low-cost opportunity for conducting pragmatic trials (e.g., the TASTE trial26).The Nature, Determination, and Analysis of Trial Outcomes
Pragmatic end points should be important to patients — for example, major life events (e.g., death or emergency hospital admission). Pragmatic trials are also often large, identify limited treatment effects, and assess the safety of underinvestigated interventions in unselected populations. They are also often simple and minimize trial procedures and data-collection requirements. The CRASH trial29 achieved a high degree of simplicity with a two-page case-report form. The catheter trial28had a primary outcome of symptomatic catheter-associated urinary tract infection up to 6 weeks after hospital discharge, rather than laboratory-confirmed infections in the hospital, which emphasized the importance of health resource use over mechanistic outcomes.Symptoms, disability, and quality of life are commonly key outcomes in pragmatic trials. Unlike major life events, signs and symptoms and quality-of-life measures are seldom recorded consistently in routine practice and require patient visits or completion of questionnaires. Pragmatic trials often use mailed questionnaires or Web-based forms to avoid the need for study visits. Such methods reduce costs but can lead to substantial amounts of missing data, which creates challenges for analysis and interpretation. Offering participants alternative methods of providing responses, including mobile phones and other handheld devices, might increase response rates. Research into shorter, effective patient-reported outcome questionnaires continues.44 The ASSIST trial30 achieved a higher than 90% rate of return of self-reported data, an unusually high level. In mental health and other areas in which many outcomes are based on questionnaires, direct follow-up is difficult to avoid. For example, the Initial Antidepressant Choice in Primary Care trial37 (a trial with an otherwise pragmatic design) had study visits at 1, 3, 6, 9, 12, 18, and 24 months after randomization. The main results of the trial were based on the first three study visits, and 91% of these visits were completed. Quality-of-life outcomes play an important role in cost-effectiveness analyses, which are a common feature of pragmatic trials, as illustrated in MI FREEE27 and the Initial Antidepressant Choice in Primary Care trial of fluoxetine versus tricyclic drugs.37 Clearly, quality-of-life outcomes cannot be collected in a no-consent trial, such as MI FREEE, or in trials with follow-up within a registry or electronic health system, such as High-STEACS31 and TASTE,26unless they are routinely recorded.Pragmatic trials can provide long-term safety data for unselected populations. However, there are challenges in interpreting safety data, which are often self-reported or subject to delays in availability, incompleteness, and coding variability associated with national registries. Explanatory trials can also present interpretational challenges with respect to adverse events, because data on events are sometimes not collected after discontinuation of the randomly assigned treatment, which introduces bias into statistical analyses.It has been argued that pragmatic outcomes should not need adjudication. We believe this is a quality issue rather than a pragmatic issue. If the quality and consistency of outcome ascertainment can be improved by adjudication without affecting normal patient care, then surely that is desirable.DISCUSSION
Drug development involves the cautious introduction of a new substance into human participants, with gradual evaluation in patients who have the relevant disease, in order to evaluate safety, early evidence of efficacy, and appropriate doses for future evaluation. The development of nondrug interventions should, but often do not, involve proof-of-concept or pilot studies to tailor the intervention and evaluate its acceptability. Many such interventions also require selection of a dose, such as duration and intensity of physiotherapy or physical training. These trials by their nature could be, but need not be, pragmatic, because they involve careful refinement of the intervention and assessment of its potential value in clinical practice.It is only after phase 3 drug trials that we have any real understanding of whether the treatment is beneficial, who might benefit most, the potential adverse effects, and the most cost-effective implementation. The ideal time to perform a pragmatic trial would be during the implementation stage of a complex intervention or the postmarketing phase of drug evaluation, to help provide an understanding of what the effect of introducing the new technology might be on overall public health. This raises the question of who should pay for these trials. With regard to drugs and devices, industry representatives may think that they have already fulfilled their role in getting a drug to the registration stage. Perhaps the best solution would be joint industry–governmental funding.Some trials, by virtue of their context and the intervention studied, are more pragmatic than others. Trials that test a low-cost intervention, pose few risks to participants, or are applied at a cluster level will almost automatically be more pragmatic in nature or easier to organize in a pragmatic fashion than will trials with high-cost, complex interventions. Health care systems with comprehensive electronic records or condition-specific registries offer excellent environments for pragmatic, low-cost trials.The conflict between mechanistic trials and pragmatic trials is often expressed as the “greater internal validity of mechanistic studies” versus the “improved external validity of pragmatic trials.” Price et al.45 describe two pragmatic trials designed to evaluate the real-world effectiveness of a leukotriene-receptor antagonist (LTRA) as compared with either an inhaled glucocorticoid for first-line asthma-controller therapy or a long-acting β2-agonist (LABA) as add-on therapy in patients who were already receiving inhaled glucocorticoid therapy. The results at 2 months suggested that the LTRA was equivalent to an inhaled glucocorticoid as first-line controller therapy and was equivalent to a LABA as add-on therapy for diverse patients in primary care. Equivalence was not established at 2 years. Nonadherence to the prescribed regimen was a major limitation. To mimic real-world practice, the investigators constructed two treatment strategies that rapidly developed considerable similarity. This undercut the power of the trial to detect differences in the effectiveness of the drugs under investigation. The investigators noted that “the very features of pragmatic trials that support the generalizability, or applicability, of their results to real-world practice may also reduce assay sensitivity and therefore limit the interpretation of results.” These features include heterogeneous populations of patients in which some of the patients may not have the condition of interest, along with a lack of blinding, absence of a placebo group, and suboptimal adherence to therapy.A natural environment for clinical research might involve the integration of research and clinical practice through the development of “learning health care systems,” as advocated by the Institute of Medicine,46 with relevant clinical and patient-reported outcome data collected by default. However, some have questioned whether this is feasible, given the clinical delivery pressures within today’s health care systems.47,48Pragmatism should not be synonymous with a laissez-faire approach to trial conduct. The aim is to inform clinical practice, and that can be achieved only with high-quality trials. We believe that the concepts of internal and external validity and even the dichotomy between explanatory and pragmatic trials are overly simplistic. A better approach is to assess how a trial design adequately addresses the main objectives of the trial, including its ability to inform clinical practice.CONCLUSIONS
Some trials need not be forced to be pragmatic, and others will naturally have pragmatic features because of the nature of the intervention and the health care context in which the trials are conducted. Very few trials can be fully pragmatic. Trials of truly novel interventions can be game changers without being particularly pragmatic. No single trial, pragmatic or otherwise, is likely to answer all potential questions about the value of any health care technology. A pragmatic approach to pragmatism would be to adopt the features of pragmatic trials whenever feasible and sensible and when such features do not compromise trial quality and the ability to answer the clinical question of interest.Disclosure forms provided by the authors are available with the full text of this article at NEJM.org.SOURCE INFORMATION
From the Robertson Centre for Biostatistics, University of Glasgow, Glasgow (I.F.), and the Centre for Healthcare Randomised Trials, Health Services Research Unit, University of Aberdeen, Aberdeen (J.N.) — both in the United Kingdom.Address reprint requests to Dr. Ford at the Robertson Centre for Biostatistics, University of Glasgow, Level 11, Boyd Orr Bldg., Glasgow G12 8QQ, United Kingdom, or at ian.ford@glasgow.ac.uk.