As generally understood by statisticians (SYSTAT 10, 2000, I228-31), experiments are studies where the factors or variables that affect the outcome of interest are under the direct control of the experimenter. Because the experimenter directly manipulates the factors, he/she can make inferences about causality. The experimental design chosen to answer the research question determines how much data should be collected, which factor levels to use, and how to analyze the results in order to answer the research question. Prior knowledge guides specification of the research question and the design of the experiment. A poorly designed experiment can leave important questions unanswered, can confound the effects of two or more variables so that they are statistically indistinguishable, and can produce inadequate estimates of the relationship between factor levels and outcome.
Examining the design characteristics of an experiment whose results were statistically analyzed and reported permits inferences about the nature of the question that was addressed and the extent to which the experimenters anticipated and statistically controlled for the possible confounding effects of two or more variables that might have influenced the results of the experiment. The focus here is on examining the design characteristics and reported results of the ARDSNetwork (2000) study and later critiqued by others (Eichacker, Gerstenberger, Banks, Cui & Natanson, 2002; Eichacker, Banks, & Natanson, 2003).
There are several design and measurement issues that, if clarified, may illuminate the nature of the research question the experimenters set out to answer as well as the adequacy of their experimental design to do so in terms of the causal factors and their levels they employed to represent the medical care of acute respiratory distress syndrome (ARDS) patients and its hypothesized influence on patient survival and recovery.
Isolating and measuring the effect of adjunctive ventilator use in the treatment of ARDS is particularly challenging insofar as it is essentially supportive in the medical treatment of the condition that precipitated ARDS. As indicated on the ARDS Support Center (6/1/03) website, "There are no currently established standard medication treatment for combating ARDS itself. Most medication treatments are oriented toward the underlying medical problems which the patient is battling, such as sepsis and trauma." Therefore, researchers must take special pains to anticipate and measure the numerous components of the medical treatment of the ARDS-precipitating condition that could offer rival explanation for any differences in ventilator use outcomes ostensibly attributable to an experiment. In addition to drugs to combat or prevent infections, ventilator use sometimes requires the administration of sedating, paralyzing and anti-anxiety medications and special positioning beds.
As reported by Seeman (2003), as many as 150,000 Americans per year suffer life-threatening ARDS from which 40 percent now die--a 10 percent improvement from the mid-1980s when it killed more than half of its victims. ARDS is a complication of body-wide infections in one-third of patients but can also result from traumatic injuries, pneumonia, gastric aspiration, smoke inhalation, and near-drowning. Blood and fluids leaking into the lungs impede normal breathing and oxygenation.
Dr. Meryl Nass (2003) provides the following description of ARDS and its clinical management:
Almost anything can cause ARDS. Usually it is a lung condition, such as a severe pneumonia but other severe systemic illness, or post-op[erative] complications can also do it. Generally people are incapable of oxygenating themselves adequately, and require ventilatory assistance until the primary problem and/or the ARDS condition clears up. These patients may be on steroids, antibiotics, antifungals, IV alimentation, antiulcer medications, etc. They probably average about 10 drugs each. They get all kinds of dread complications that need to be treated as well. There are usually a number of physicians managing each patient. They often have a lot of tubes in them, often have bleeding disorders comcomitantly. They need a gentle but very observant and obsessive person in charge who will immediately look for a reason for a small increase in pulse, respirations or temp[erature], or white blood count.
The fixed-effect factorial experimental design selected by the ARDSNetwork researchers was one that addressed two primary questions, "Which of two extreme values in the distribution of ventilator volumes used by clinicians to assist ARDS patients while their primary conditions are also being treated is the more efficacious in terms of survival and breathing without assistance?" "Which of two investigational drugs - ketoconazole and lisofylline - interacting with ventilator volumes at the two extremes is more efficacious?" Secondary questions about efficacy addressed other outcomes, namely, the number of days without organ or system failure and the incidence of barotrauma.
Ideal design. Given clinical equipoise or uncertainty about which adjunctive ventilator volumes in the range used by clinicians in combination with the medical treatment of the ARDS-precipitating conditions are more efficacious than others, the ideal experimental design would build on non-experimental observational studies about the range used by clinicians and associated outcomes.[1] If some level of consensus exists about the "best practice" range, then the ideal experimental design would incorporate that range to create a control group against which to compare the results of treatment in one or more experimental groups fashioned to test hypotheses about the equal or greater efficacy of the treatment in the experimental group or groups in comparison to treatment in the "best practice" control group. In cases where there is uncertainty about the entire range, the experimental design considerations are much different than those for estimating the "fixed" effects for some specific values in the range.
The ideal fixed-effect experimental design would be such that ARDS-presenting patients would be randomly and IMMEDIATELY assigned to one of the two or more groups that define the experimental and control treatments. Data on all variables that might affect treatment outcomes would be collected before, during, and at the end of the clinical trial. Obviously, mortality would preclude continued collection of data and would account for declining numbers of patients and available data over time in the experimental and control groups.
The randomization procedure of the ideal fixed-effect experimental design theoretically assures that any possible variables that might confound (i.e., that might serve as a rival causal explanation for) any observed differences in outcomes between the experimental and control groups would be equally distributed in these groups and therefore be held constant so as to permit the inference that only the manipulated experimental variable accounts for any observed differences. In practice, however, random assignment of subjects into experimental and control groups sometimes does yield by chance an unequal distribution of possible confounding variables; hence, there is need to draw them into the data collection system in order to test the hypothesis that they are indeed equally represented in the experimental and control groups and are not exerting differential impact on outcomes over time. Statisticians often hold constant via covariate adjustment the effects of possible confounding variables when testing the statistical significance of observed differences between the experimental and control groups.
In cases where there is uncertainty about the effects along the entire range of clinical practice, a different experimental design is called for, namely, the "random" effects or a "mixed" design combining fixed and random factors. The logic of random and mixed experimental designs calls for a different data collection strategy that entails random selection of points in the range of clinical practice about which there is clinical equipoise or uncertainty. Such a design permits statistical inference about the entire range and the nature of the relationship between it and the outcome variables - whether positive or negative linear or positive or negative curvilinear. In addition to differences in randomization procedure, the fixed-, random-, and mixed-effect experimental designs require use of different statistical models with different computational methods and underlying assumptions.
The assumptions underlying specific tests of statistical significance are particularly important because they provide the theoretical justification for the analysis and determine the reliability and validity of the estimates that observed differences among experimental and control groups or between factor levels could have occurred by chance. The ARDSNetwork researchers used analysis of variance as their principal test for the statistical significance of fixed-effect differences between the mean outcomes in the low and high tidal volume groups.
According to Hays (1963), the principal assumptions for the fixed-effect ANOVA are:
1. There must be normal distribution of errors for any treatment population. Provided the sample size is sufficiently large, the consequences of non-normality are not horrendous.
2. Error variance must be the same, i.e., homogenous, for all treatment populations. If sample sizes are the same for each treatment population, again the consequences of non-homogeneity are not horrendous. But when sample sizes are unequal, the consequences are very serious for the validity of inferences about the statistical significance of differences between mean outcomes.
3. There must be statistical independence among the error components, namely, that observations between and across groups are in no way related to any other observation. The violation of this assumption has very serious consequences for statistical inference. Violation of the assumption frequently occurs in cases where the experimental subjects are unstable and receive different treatment as their circumstances change.
The ARDSNetwork researchers did not discuss the extent to which their data met the assumptions of the fixed-effect analysis of variance model they used to estimate the statistical significance of the outcome differences they report between the low and high tidal volume groups. It is impossible, therefore, to evaluate the accuracy of the Type I error probabilities they report in support of their findings.
Typically, power analysis is used in deciding how large a sample size to draw in order to satisfy the normality assumption and, more importantly, to detect literature-based predictions of small, medium or large size effects or differences in outcomes between the experimental and control groups or among factor levels. The ARDSNetwork researchers did a remarkably good job in this regard. The sample size of 861 is the precise number required to obtain a 0.85 power of the test to detect a small size effect with the probability of Type I error (falsely rejecting the null hypothesis) at 5 in 100 and Type II error (falsely accepting the null hypothesis) at 15 in 100 (See Figure 1). The precision of estimate in this regard suggests that ARDNet researchers knew their target sample size and worked diligently to recruit the needed number of experimental subjects.
ARDSNetwork design. Now let us consider the deviations of the ARDSNetwork experimental design from the ideal and ask why these deviations may have been permitted. Effective peer review judges the quality of reported findings on the basis of the logic of the research design, sampling and randomization, selection, appropriate use, and explanation of the statistics used, and handling of validity and reliability issues, including care and workmanship in handling data and analysis as well as discussion of assumptions (Noble, 1974). Our sources of information in this regard are the ARDSNetwork report in the New England Journal of Medicine (May 4, 2000) and two published critiques (Eichacker, Gerstenberger, Banks, Cui & Natanson, 2000; Eichacker, Banks, & Natanson, 2003).
To answer the first question about which of two extreme ventilator volume values was the more efficacious, 861 ARDS patients were randomly assigned into one of two extreme low and high static ventilator volume groups - the low defined as providing a tidal volume of 6 mL per kilogram of predicted body weight (PBW) and the high defined as providing 12 mL per kilogram of predicted body weight. The ARDSNetwork researchers conceptually defined "traditional" as those tidal volumes lying in the range of 10 to 15 mL per kilogram PBW. For purposes of the experiment, they operationalized this range by random assignment of ARDS patients to a single value, namely, 12 mL per kilogram PBW - rounding down rather than up from the range midpoint of 12.5 mL per kilogram. The cumulative incidence of death during 180 days and number of days without ventilator use from day 1 to day 28 were the measures of "survival" and "breathing without assistance."
To answer the second question about the efficacy of two investigational drugs interacting with low and high ventilator volumes, the first 234 ARDS patients were simultaneously enrolled in a second clinical trial to compare the effects of ketoconazole compared to placebo; the following 433 patients received no drugs in addition to ventilator volume treatment at the two extremes; and the last 194 patients enrolled received lisofylline compared to placebo (ARDSNetwork, 2002). The fixed-effect factorial design provided data by which to judge the possible interaction effects of these drugs on survival and breathing without assistance. No statistically significant differences for such effects were reported for either drug. The factorial design did not permit assessment of the effects of other investigational treatments, including prone positioning, administered to 15 patients in the low tidal volume group and 12 patients in the high volume group. The results for this non-factorial experiment were not reported.
The ARDSNetwork researchers randomly assigned patients to one of the extreme low and high ventilator volume treatment groups from their prior locations in the normally distributed range of ventilator volumes they had been receiving. As stated by Eichacker, Banks, Natanson (2003), "this (ARDSNetwork) trial compared approximately the 3rd to the 80 percentile of current practice without assessing the most commonly used level of care . . . at the 50th percentile . . ." In consequence, the ARDSNetwork fixed-effect factorial experimental design could not represent survival outcomes between the static 6 mL per kilogram PBW and the static 12 mL per kilogram PBW wherein 77 percent (80th percentile minus 3rd percentile) of clinical practice lies. Nor could it assess the possible interactive effects of ketoconazole and lizofylline on survival outcomes across the range of majority clinical practice.
Clearly, the ARDSNetwork researchers, by the experimental design they selected, were focused on removing clinical equipoise or uncertainty about which of two extreme tidal volumes - 6 mL vs.12 mL per kilogram PBW - singly or interacting with two alternative drugs - ketoconazole or lizofylline - was more efficacious in preventing death among ARDS patients. Their experimental design excluded concern about the efficacy of tidal volumes in the range wherein 77 percent of ARDSNetwork study center practice was occurring prior to random assignment into one or other of the two extremes.
ARDSNetwork statistical analysis.
1. How to resolve problems relating to the conceptualization and operational definition of the independent variable?
First, the use by the ARDSNetwork researchers of the term "traditional" to describe the high volume experimental group is curious insofar as it conveys a meaning that is contrary to the pre-randomization practice at the ARDSNetwork centers. As reported by Eichaker, Banks, and Natanson (2003), less than 3 percent were receiving the low volume of 6 mL per kilogram PBW or lower and 20 percent 12 mL per kilogram PBW or higher. What about the 77 percent lying between these extremes? "Traditional" connotes "typical," "usual" or "customary." Was the ARDSNetwork high volume experimental group meant to serve as a "straw man?" Statistically, the central tendency (50th percentile) of clinical practice even at the ARDSNetwork centers was about 10 mL kilogram PBW, as indicated by Eichaker, Banks and Natanson (2003, Fig. 1). The ARDS Support Center (2003, June 1) website publication by Neil R. MacIntyre, MD, describes typical use as follows:
The tidal breath, in conjunction with the baseline pressure, should be set in such as way that the plateau pressure is < 35 cm H2O (or other index of over distention) does not occur. Generally, this involves tidal volumes (VT) of 8-10 ml/kg although VT as low as 5-6 ml/kg may be needed. Older strategies of using higher tidal volumes arose from a need to prevent atelectasis. Now that PEEP strategies are better understood and the risk of over distention better appreciated, this need has lessened.
Further, Ricard (2003), in an editorial about the ARDSNetwork study, states:
. . . the data of Weinert and colleagues clearly show that tidal volumes had started to be reduced before the completion of the ARDS Network trial (and probably back in the early 1990s), and that publication of the results of the ARDS Network trial did not impact on the approach taken by clinicians in the ventilatory management of patients with ARDS. These observations are in agreement with several other pieces of data.
Second, and more important, is the misconception of the real ARDSNetwork independent variable. It was not low (6 mL per kilogram PBW) vs. high (12 mL per mg. PBW) tidal volume that was manipulated by the ARDSNetwork researchers but rather the CHANGE between the initial volume prescribed by the attending physician on the basis of individual patient need and the static low or high experimental volume defined by the ARDSNetwork research protocol. The difference between the pre- and post-random assignment tidal volumes was substantial for many of the 861 patients who participated in the trial.
Table 1 presents the range of pre- to post-randomization tidal volume differences that could have occurred, given the tidal volumes the 861 patients were receiving before random assignment into either the ARDSNetwork low or high experimental group. The theoretical distribution of pre- to post-random assignment tidal volume differences ranges from an 11.5 point reduction (-65.7 percent) to a 7.5 point increase (166.7 percent). The actual pre-to post-randomization change in tidal volume that the 861 individual subjects experienced can be determined by reanalysis of the ARDSNetwork data. It is for the clinicians to decide how much abrupt changes of these magnitudes may have challenged the physiology of patients who experienced them.
Clearly, the ARDSNetwork statistical analysis both incorrectly conceptualized and operationally defined the independent variable of the trial. Thus, all findings are subject to revision on the basis of reanalysis of the data using the correct operational definition of the study independent variable, along with adequate measurement and statistical control of a range of possible confounding variables. Such reanalysis would permit direct test of the Eichacker, et al (2002) hypothetical model representing the relationship between tidal volumes, resultant plateau airway pressures, and mortality.
Unfortunately, such reanalysis cannot remove clinical equipoise or uncertainty about mortality and other outcomes that were associated with the tidal volumes initially prescribed by the attending physicians on the basis of patient need. The properly designed experiment for doing so, as previously indicated, would have required AT THE OUTSET random assignment of the ARDS patients into a random sample of the range of tidal volumes typically used by attending physicians. Analysis of the outcomes would have properly required use of the analysis of variance model for a random-effect experimental design.
2. How to decide between rival explanations for the ARDSNetwork reported findings?
Randomization into the low and high experimental groups of the ARDSNetwork study is assumed to control for all possible confounding variables - a heroic assumption when ventilator therapy is considered to be an adjunctive rather than the primary intervention. Hence, we cannot rule out a number of rival explanations for the reported difference in outcomes between the low and high volume experimental groups. We cannot ascertain from the ARDSNetwork report whether the data needed to rule out competing explanations were collected. At a minimum, rival explanations for the reported differences in outcomes include:
(A) The length of time the ARDS patient received clinician-prescribed ventilator volume before random assignment into one of the study experimental groups;
(B) The percentage inspired oxygen dialed into the ventilator by the attending physician with the goal of keeping it below 40-50% insofar as higher levels over a longer time create their own toxicity.[2]
(C) Different prognoses before random assignment based on:
1. Pre-existing lung disease
2. Significant blood loss requiring transfusions
3. Prolonged hypotension
4. Presence of renal failure
5. Presence of disseminated intravascular coagulation
In addition, since there is a large possibility that CHANGE from the pre- to post-randomization tidal volume procedures may account for mortality and other outcomes, it would make sense to hold constant statistically via covariate adjustment the very same measures used by the ARDSNetwork researchers to describe ventilator procedures described in Table 1 of their New England Journal of Medicine (2000) report.
The lack of statistical control for length of time before random assignment into one of the two ARDSNetwork experimental groups, as well as associated pre-randomization tidal volume and PEEP values is of particular concern. To reiterate - abrupt change from the initial circumstances of medical care might well provide a rival explanation for the differences in mortality and the other outcomes reported by the ARDSNetwork researchers.
4. How to explain inconsistencies in the reporting of deaths and other data?
Table 2 of the ARDSNetwork (2000) report that provides base-line characteristics of the patients indicates the availability of data available for only 300 (69.4 percent) of the 432 patients in the low tidal volume group and only 290 (67.6) of the 429 patients in the high volume group. What accounted for this large amount of missing data? These large percentages of missing data render meaningless any attempt to compare the two groups on this base-line characteristic.
Figure 2 of the ARDSNetwork (2002) report reports mortality rates for only 267 (62.2 percent) of the 429 patients in the high tidal volume group and only 260 (60.2 percent) of the 432 patients in the low volume group. This high percentage of missing data renders highly suspect the Figure 2 assertion, "The interaction between the study group and the quartile of static compliance at base line was not significant (P = 0.49)."
Table 3 of the ARDSNetwork (2002) report contains very significant amounts of unexplained missing data, lending credence to my calculation of the estimated cumulative death rates on the assumption that the lowest percentage of attrition/missing data in the listed respiratory values for the high and low tidal volume groups reflects mortality and not random data collection failure.
My Table 2 represents the extent of missing data in percentages for ARDSNetwork Table 3 listed respiratory values during the first seven days of treatment in the low and high tidal volume groups.
My Table 3 presents the estimated cumulative death rates for the ARDSNetwork low and high tidal volume groups based on the assumption that the lowest percentage of missing data for the listed respiratory values reflects mortality and not random data collection failure. According to these estimates, the death rate in the low tidal volume group was 4.3 percentage points greater than in the high tidal volume group at the end of day 1; 3.3 percentage points greater at the end of day 2; and 0.6 percentage point less at the end of day 7. The death rate in the low tidal volume group was cumulatively 4.9 percentage points greater than in the high volume group at the end of day 7. This is the opposite to the ARDSNetwork Figure 1 plot of the probability of survival during the first 180 days after randomization. What is the explanation for these inconsistent and contradictory findings?
5. How to explain the lower mortality of ARDSNetwork trial refusers, as reported by Eichacker, Banks, & Natanson (2003)?
Before concluding anything in this regard, it is important to establish the statistical equivalence of the trial refusers and participants before randomization, as well as the course of their medical care after refusal. Why did they, their surrogate decision-makers, or their attending primary care physicians refuse participation? In other words, what patient characteristics, if any, led to refusal and did these characteristics differ significantly from those of the participants? Did they receive significantly different medical treatment for the conditions that precipitated ARDS than did trial participants? Obviously, the equivalence or non-equivalence of the trial refusers compared to the participants will lead to different interpretations of the reported lower mortality among trial refusers. Establishing the statistical equivalence of trial refusers and participants before randomization will require access and reanalysis of the relevant ARDSNetwork data.
6. How to document and statistically control for the influence on mortality and other outcomes of possible adverse events associated with the simultaneous drug trial that was conducted?
The ARDSNetwork researchers report no statistically significant interaction effects in mortality and other outcomes associated with the simultaneous drug trial testing the efficacy of ketoconazole and lisofylline vs. their respective placebos. Nothing is reported about possible drug-related adverse events or whether the two experimental drugs displaced other drugs that were being used before randomization into the low and high tidal volume arms of the ventilator experiment. Given the misconception of the study independent variable and its operational definition, skepticism about the claim of no statistically significant interaction effects is justified. Again, there is need to access and reanalyze the relevant ARDSNetwork data.
7. Conclusion: Conceptual and measurement problems in the ARDSNetwork (2000) report of findings raise basic questions about their trustworthiness as a guide to clinical practice. All study results could well be attributed to (1) the statistically uncontrolled concomitant non-ventilator medical care that was provided for the ARDS-precipitating conditions or (2) the masked effects of abrupt change from the initial tidal volume and medical care regimen prescribed by the attending physician into one or other of the two experimental arms of the ARDSNetwork experiment or (3) the unmeasured interaction effects involving the two simultaneous drug trials and alternative #2 or (4) some combination of alternatives #1 to #3.
1. OHRP should turn the ARDSNetwork data over to an independent body to reanalyze in order to answer the numerous questions that we have been raised here.
2. The New England Journal of Medicine and other journals read by clinicians should publish the results of the reanalysis.
3. NIH should invest in a properly-designed ethical study to remove remaining clinical equipoise about the efficacy of adjunctive ventilator therapy in the range where 77 percent of clinical practice occurs. The Eichacker, et al. hypothesis linking tidal volume to plateau airway pressure changes and mortality needs to be rigorously tested for the sake of all who may ever become ARDS patients.
ARDSNetwork (2000). Ventilation with lower tidal volumes as compared with traditional tidal volumes for acute lung injury and the acute respiratory distress syndrome. New England Journal of Medicine, 342 (18), 1301-1308.
ARDS Support Center (2003, June 1). Mechanical ventilation strategies for lung protection. Presentation by Neil R. MacIntyre, MD, Duke University Medical Center, Durham, NC, at Wisconsin Society for Respiratory Care, May 1999. At: www.ards.org/learnaboutards/treatment/ventilator/ventstrategies.html
Eichacker, P.Q., Gerstenberger, E.P., Banks, S.M., Cui, X., & Natanson, C. (2002). Meta-analysis of acute lung injury and acute respiratory distress syndrome trials testing low tidal volumes. American Journal of Respiratory and Critical Care Medicine, 166(11), 1510-1514.
Eichacker, P.Q., Banks, S.M., & Natanson, C. (2003). Meta-analysis of tidal volumes in ARDS: From the authors. American Journal of Respiratory and Critical Care Medicine, 167, 778-800.
Faul, F., & Erdfeder, E. (1992). GPOWER: Apriori, post-hoc, and compromise power analyses for MS-DOS. Bonn, Germany: Bonn University, Department of Psychology.
Hays, W.L. (1963). Statistics for psychologists. New York: Holt, Rinehart & Winston.
Morganweck, C. (2003). Innovation to research: some transitional obstacles in critical care units. Critical Care Medicine, vol. 31(3)(Suppl), S172-S177.
Nass, M. (2003, June 1). Personal communication.
Noble, J.H. (1974). Peer review: quality control of applied social research. Science, 185 (Sept. 13), 916-921.
Ricard, J-D. (2003). Are we really reducing tidal volume - and should we? American Journal of Respiratory and Critical Care Medicine, 167, 1297-1303.
Seeman, B.T. (2003, May 15). Research dispute prompts debate over patient protection. Washington, DC: Newhouse News Service.
SPSS (2000). SYSTAT 10 Statistics I. Chicago: SPSS.
Weinert, C.R., Gross, C.R., & Marinelli, W.A. (2003). Impact of randomized trial results on acute lung injury ventilator therapy in teaching hospitals. American Journal of Respiratory and Critical Care Medicine, 167, 1304-1309.
footnotes:
[1] Arguing against experimentation with such a critically ill set of patients is their vulnerability and the need to proceed with the greatest caution to protect their lives. Indeed, concerns have been raised about the added risks that a randomized clinical trial protocol may pose for critically ill patients. For example, Dr. Cynthiane Morgenweck in Critical Care Medicine (2003 p. S172) acknowledges: "It can be argued that patients in such critical states cannot be placed in a research study because adherence to the protocol might limit patient improvement. Physicians need the flexibility to act quickly, unhindered by study guidelines." Such critically ill patients require carefully monitored, individualized care, including air ventilation whose tidal volume and airway pressure level is appropriately determined on the basis of the severity of their condition and the presence of complicating conditions. Under the restrictions of a randomized clinical trial protocol with rigid requirements physicians are prevented from providing care that is tailored to that physician's understanding of the patient's best interests. It is one matter for a treating physician to lower tidal volume to 6 ml / kilogram in routine practice because the patient's plateau air pressure rises to a high level. It is quite another matter when a patient is randomly switched to a fixed tidal volume level that countermands the physician's prior assessment of patient needs and optimal medical care for survival.
[2] The percentage inspired oxygen is an independent variable that may be adjusted with the tidal volume but not necessarily.