The Misuse of Meta-analysis in Nutrition Research

Viewpoint September 18, 2017 Neal D. Barnard, MD1; Walter C. Willett, MD, DrPH2; Eric L. Ding, ScD2 1George Washington University School of Medicine and Health Sciences, Washington, DC 2Department of Nutrition, Harvard T.H. Chan School of Public Health, Boston, Massachusetts JAMA. Published online September 18, 2017. doi:10.1001/jama.2017.12083 JAMA. Published online September 18, 2017. doi:10.1001/jama.2017.12083 Controversial conclusions from meta-analyses in nutrition are of tremendous interest to the public and can influence policies on diet and health. When the results of meta-analyses are the product of faulty methods, they can be misleading and can also be exploited by economic interests seeking to counteract unflattering scientific findings about commercial products. The term meta-analysis was coined by Glass in the mid-1970s for a set of techniques designed to characterize and combine the findings of prior studies in order to increase statistical power, provide quantitative summary estimates, and identify data gaps and biases. When applied to studies conducted with similar populations and methods, meta-analyses can be useful. However, many published meta-analyses have combined the findings of studies that differ in important ways, prompting Eysenck to complain that they have mixed apples and oranges—and sometimes “apples, lice, and killer whales”—yielding meaningless conclusions.1 Nutritional science presents special challenges for meta-analyses. In clinical trials, nutrition interventions vary from one study to the next in many methodological details, weakening the argument for combining their results. This is in contrast to studies of drugs in which it is generally easier to assess the comparability of interventions. In observational studies, populations range widely in their dietary habits, and while some diet characteristics (eg, coffee use) are fairly consistent for individuals from day to day and are reliably reported, the consumption of most foods (eg, vegetables) and nutrients (eg, sodium) is variable and difficult to quantify. Different studies handle these issues in different ways. Moreover, different studies may report dietary intakes in tertiles, quartiles, quintiles, or other groupings of their own choosing. Combining results may require contacting the original investigators for participant-level data, which may have been produced using dissimilar dietary assessment techniques. When Populations Differ A 2014 meta-analysis examined the relationship between saturated fat intake and coronary artery disease.2 One of the included prospective studies, the Oxford Vegetarian Study,3 included vegans, ovolacto vegetarians, fish eaters, and meat eaters, with reported saturated fat intake ranging from 6% to 7% of energy in vegan participants to approximately twice that amount in the other diet groups. Those in the highest tertile of saturated fat intake had nearly triple the risk of fatal ischemic heart disease compared with the lowest tertile. In contrast, another study included in the meta-analysis, the Malmö Diet and Cancer cohort study,4 had no groups at the lower end of saturated fat intake, which ranged from 13% to more than 22% for the lowest to the highest quintiles, and no significant association between saturated fat intake and risk of cardiovascular events was detected. The Malmö authors cautioned, “only 1.2 percent of the present study population actually followed national Swedish recommendations (less than 10 energy percent) on saturated fat intake. Strictly speaking, the SFA-CVD [saturated fatty acids-cardiovascular disease] hypothesis is thus not fully testable in this population.” Nevertheless, the Malmö study was given substantial weight in the meta-analysis, which concluded that available evidence did not support limiting saturated fat, a conclusion repeated in a New York Times commentary proclaiming “Butter is Back” and a Time magazine cover displaying an artistic butter swirl and the bold headline “Eat Butter,” and cited by the 2015 Dietary Guidelines Advisory Committee. The following year, a Gallup poll registered a sharp decline in the number of US adults limiting fat in their diets. Compared With What? The effects of any given dietary exposure depend on what that exposure is compared against. A 2017 meta-analysis evaluated associations between red meat intake and blood lipid concentrations.5 Of the 39 trials that contributed to the analysis on low-density lipoprotein (LDL) cholesterol, 34 compared red meat with other meats, revealing little apparent relationship with LDL cholesterol. The remaining 5 studies compared red meat to plant-based foods, most of which found nonsignificantly increased LDL cholesterol after red meat consumption. However, the investigators combined the results of all these studies, concluding that red meat “does not negatively influence cardiovascular disease risk factors.” A better approach would focus on a single comparator and ensure that an adequate number of studies had used the method of interest. In addition, eating less of one type of food often means eating more of something else. If some research participants consume less saturated fat, for example, what is taking its place—polyunsaturated fat, monounsaturated fat, complex carbohydrates, simple sugars, or something else? Understanding the effects of specific substitutions can lead to more robust and informative findings than focusing on the effects of one nutrient or food alone compared with everything else in a diet. Study Quality Studies vary greatly in quality. For example, some clinical trials are randomized; others are not. A 2014 meta-analysis of the effect of palm oil intake on blood lipids included studies that varied in quality, concluding that evidence was too inconsistent to draw firm conclusions.6 A later meta-analysis limited to higher-quality randomized trials found that palm oil significantly increased LDL cholesterol concentrations, compared with nontropical oils.7 The issue of the quality of individual trials is relevant for all meta-analyses, which reinforces the need to follow reporting guidelines, such as the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses), and include a rigorous assessment of risk of bias in individual studies, among other steps outlined below. Do Meta-analyses Help? Combining the results of individual studies increases the total number of participants, and more participants should mean more statistical power. However, when there are differences in participant demographics and study methods, combining studies increases variability in findings that can reduce statistical power, making real effects more difficult to identify. So, for example, if saturated fat is associated with a disease outcome in an individual well-conducted study, but not in a meta-analysis, the null result may reflect heterogeneity among studies that dilutes real findings. Sensitivity analyses, which systematically remove some studies from the analysis, sometimes help by focusing, for example, on higher-quality studies. The most important contribution of a meta-analysis is not necessarily the single statistical summary of effect size, but rather may be the ability to elucidate why different studies have produced different results. Subgroup analyses may help explain observed differences, and unexplained heterogeneity should be acknowledged. When individual studies vary substantially in their populations and methods, a meta-analysis may be less useful than a single (or small number of) well-conducted investigation(s). Science and Money The food industry is well aware of the power of science-driven headlines and has invested in meta-analyses. In the process, nutritional science may be adversely affected. In a 2007 review of 111 industry-funded studies, funding source was significantly related to study conclusions.8 Even in the absence of commercial funding, bias is an important consideration, so transparency in the conduct of meta-analyses is as important as it is in the individual studies. A Way Forward Because meta-analyses, particularly involving diet, influence health policy, carry considerable weight in the media and in public perception, and have the potential to do harm, the peer-review process must go beyond ensuring that standard meta-analytic procedures have been followed. This could include (1) requiring review by editors with expertise in meta-analysis and in the subject matter at hand, (2) requiring authors to confirm with the authors of the original reports that their data were appropriately represented, to the extent possible, (3) requiring authors to share their summary data and methodological details to allow others to reproduce the analysis, and (4) prioritizing meta-analyses derived by pooling original primary data over those using published summary data. Potential conflicts of interest should be carefully scrutinized for meta-analyses and the studies they include. This process could be facilitated by a standardized, permanent financial disclosure registry. These steps will not eliminate controversial findings from meta-analyses of nutritional research or of other topics but may give them a more solid foundation. Back to top Article Information Corresponding Author: Neal D. Barnard, MD, George Washington University School of Medicine and Health Sciences, 5100 Wisconsin Ave, Ste 400, Washington, DC 20016 (nbarnard@pcrm.org). Published Online: September 18, 2017. doi:10.1001/jama.2017.12083 Conflict of Interest Disclosures: All authors have completed and submitted the ICMJE Form for Disclosure of Potential Conflicts of Interest and none were reported. References 1. Eysenck H. Meta-analysis squared: does it make sense? Am Psychol. 1995;50:110-111. Article 2. Chowdhury R, Warnakula S, Kunutsor S, et al. Association of dietary, circulating, and supplement fatty acids with coronary risk: a systematic review and meta-analysis. Ann Intern Med. 2014;160(6):398-406.PubMedArticle 3. Appleby PN, Thorogood M, Mann JI, Key TJA. The Oxford Vegetarian Study: an overview. Am J Clin Nutr. 1999;70(3)(suppl):525S-531S.PubMed 4. Wallström P, Sonestedt E, Hlebowicz J, et al. Dietary fiber and saturated fat intake associations with cardiovascular disease differ by sex in the Malmö Diet and Cancer cohort: a prospective study. PLoS One. 2012;7(2):e31637.PubMedArticle 5. O’Connor LE, Kim JE, Campbell WW. Total red meat intake of ≥0.5 servings/d does not negatively influence cardiovascular disease risk factors: a systemically searched meta-analysis of randomized controlled trials. Am J Clin Nutr. 2017;105(1):57-69.PubMedArticle 6. Fattore E, Bosetti C, Brighenti F, Agostoni C, Fattore G. Palm oil and blood lipid-related markers of cardiovascular disease: a systematic review and meta-analysis of dietary intervention trials. Am J Clin Nutr. 2014;99(6):1331-1350. PubMedArticle 7. Sun Y, Neelakantan N, Wu Y, Lote-Oke R, Pan A, van Dam RM. Palm oil consumption increases LDL cholesterol compared with vegetable oils low in saturated fat in a meta-analysis of clinical trials. J Nutr. 2015;145(7):1549-1558.PubMedArticle 8. Lesser LI, Ebbeling CB, Goozner M, Wypij D, Ludwig DS. Relationship between funding source and conclusion among nutrition-related scientific articles. PLoS Med. 2007;4(1):e5. PubMedArticle