Elsevier

Academic Pediatrics

Volume 13, Issue 6, Supplement, November–December 2013, Pages S38-S44
Academic Pediatrics

Methods in QI Research
Use of Interrupted Time Series Analysis in Evaluating Health Care Quality Improvements

https://doi.org/10.1016/j.acap.2013.08.002Get rights and content

Abstract

Interrupted time series (ITS) analysis is arguably the strongest quasi-experimental research design. ITS is particularly useful when a randomized trial is infeasible or unethical. The approach usually involves constructing a time series of population-level rates for a particular quality improvement focus (eg, rates of attention-deficit/hyperactivity disorder [ADHD] medication initiation) and testing statistically for a change in the outcome rate in the time periods before and time periods after implementation of a policy/program designed to change the outcome. In parallel, investigators often analyze rates of negative outcomes that might be (unintentionally) affected by the policy/program. We discuss why ITS is a useful tool for quality improvement. Strengths of ITS include the ability to control for secular trends in the data (unlike a 2-period before-and-after t test), ability to evaluate outcomes using population-level data, clear graphical presentation of results, ease of conducting stratified analyses, and ability to evaluate both intended and unintended consequences of interventions. Limitations of ITS include the need for a minimum of 8 time periods before and 8 after an intervention to evaluate changes statistically, difficulty in analyzing the independent impact of separate components of a program that are implemented close together in time, and existence of a suitable control population. Investigators must also be careful not to make individual-level inferences when population-level rates are used to evaluate interventions (though ITS can be used with individual-level data). A brief description of ITS is provided, including a fully implemented (but hypothetical) study of the impact of a program to reduce ADHD medication initiation in children younger than 5 years old and insured by Medicaid in Washington State. An example of the database needed to conduct an ITS is provided, as well as SAS code to implement a difference-in-differences model using preschool-age children in California as a comparison group.

Section snippets

A Brief Description of ITS Analysis

In the context of quality improvement, ITS is best understood as a simple but powerful tool used for evaluating the impact of a policy change or quality improvement program on the rate of an outcome in a defined population of individuals. A time series—repeated observations of a particular event collected over time—is divided into 2 segments in the simplest case. The first segment comprises rates of the event before the intervention or policy, and the second segment is the rates after the

Strengths of ITS Analysis

A notable strength of ITS with respect to evaluating the impact of quality improvement efforts using observational data is that the approach controls for the effect of secular trends in a time series of outcome measures. For example, suppose that an intervention is introduced at a hospital to reduce medication errors. Researchers find that the medication error rate in the year after the intervention is significantly lower than in the year preceding (ie, a t test comparing the postintervention

ITS Example—Hypothetical Impact of a Program to Reduce ADHD Medication Use in Preschoolers

The American Academy of Pediatrics guideline on ADHD treatment recommends behavior therapy as the first-line treatment for ADHD in children.18 Although the 2011 guideline supports use of medications in children as young as 4 years of age, use in this population remains controversial.19 Wolraich et al20 reported that more than 90% of pediatricians start medication at a sometimes or greater frequency in children with ADHD. The Preschool-Age Treatment Study reported that preschool-age children

Segmented Regression

In technical terms, the goal of the regression analysis is to estimate the interaction terms between implementation of a policy/program and time. We also wish to estimate the effects relative to the control population. Fortunately, the somewhat complicated presentation offered in most texts is easily simplified. Table 1 shows each of the data elements needed to fit the regression model. The “Quarter” column is simply the label for each time period. The next 2 rows are the crude (or adjusted

Discussion

As we have demonstrated above, the ITS approach to policy/program evaluation has several advantages. The approach is easy to do and provides powerful, easy-to-understand results. ITS controls for secular trends in the data and therefore reduces bias that might be present in a simple 2-time-period model (ie, simple pre–post measurement and analysis). ITS does not require adjustment for individual-level characteristics.4

There are 3 important threats to validity in ITS analyses. The most serious

Limitations

Although ITS has many strengths, there are important limitations to be aware of. First, estimating the level and slope parameters requires a minimum of 8 observations before and after the policy/program implementation in order to have sufficient power to estimate the regression coefficients. In our example, ITS could not reasonably be used to evaluate the impact of the program on prescribing until 8 quarters (2 years!) after the program began. Estimating monthly or even weekly data may be

Conclusion

ITS is a simple but powerful approach to policy/program evaluation. Although the approach has limitations, few statistical approaches are as elegant in design and powerful in audience impact. ITS is particularly useful when a randomized trial is infeasible or unethical. Because ITS is the strongest quasi-experimental design, its value in quality improvement and program evaluation cannot be understated. Paired with comprehensive qualitative data regarding the implementation of policies/programs,

References (34)

  • S.B. Soumerai et al.

    Payment restrictions for prescription drugs under Medicaid. Effects on therapy, cost, and equity

    N Engl J Med

    (1987)
  • D. Gillings et al.

    Analysis of interrupted time series mortality trends: an example to evaluate regionalized perinatal care

    Am J Public Health

    (1981)
  • D. McDowall et al.

    Interrupted Time Series Analysis

    (1980)
  • B.A. Briesacher et al.

    Medicare part D and changes in prescription drug use and cost burden: national estimates for the Medicare population, 2000 to 2007

    Med Care

    (2011)
  • A.S. Adams et al.

    Prior authorization for antidepressants in Medicaid: effects among disabled dual enrollees

    Arch Intern Med

    (2009)
  • S.B. Soumerai et al.

    Use of atypical antipsychotic drugs for schizophrenia in Maine Medicaid following a policy change

    Health Aff (Millwood)

    (2008)
  • Y. Zhang et al.

    Effects of prior authorization on medication discontinuation among Medicaid beneficiaries with bipolar disorder

    Psychiatr Serv

    (2009)
  • Cited by (0)

    The views expressed in this report are those of the authors and do not necessarily represent those of the US Department of Health and Human Services, the Agency for Healthcare Research and Quality or the American Board of Pediatrics Foundation.

    The authors declare that they have no conflict of interest.

    Publication of this article was supported by the Agency for Healthcare Research and Quality and the American Board of Pediatrics Foundation.

    View full text