Methods in QI ResearchUse of Interrupted Time Series Analysis in Evaluating Health Care Quality Improvements
Section snippets
A Brief Description of ITS Analysis
In the context of quality improvement, ITS is best understood as a simple but powerful tool used for evaluating the impact of a policy change or quality improvement program on the rate of an outcome in a defined population of individuals. A time series—repeated observations of a particular event collected over time—is divided into 2 segments in the simplest case. The first segment comprises rates of the event before the intervention or policy, and the second segment is the rates after the
Strengths of ITS Analysis
A notable strength of ITS with respect to evaluating the impact of quality improvement efforts using observational data is that the approach controls for the effect of secular trends in a time series of outcome measures. For example, suppose that an intervention is introduced at a hospital to reduce medication errors. Researchers find that the medication error rate in the year after the intervention is significantly lower than in the year preceding (ie, a t test comparing the postintervention
ITS Example—Hypothetical Impact of a Program to Reduce ADHD Medication Use in Preschoolers
The American Academy of Pediatrics guideline on ADHD treatment recommends behavior therapy as the first-line treatment for ADHD in children.18 Although the 2011 guideline supports use of medications in children as young as 4 years of age, use in this population remains controversial.19 Wolraich et al20 reported that more than 90% of pediatricians start medication at a sometimes or greater frequency in children with ADHD. The Preschool-Age Treatment Study reported that preschool-age children
Segmented Regression
In technical terms, the goal of the regression analysis is to estimate the interaction terms between implementation of a policy/program and time. We also wish to estimate the effects relative to the control population. Fortunately, the somewhat complicated presentation offered in most texts is easily simplified. Table 1 shows each of the data elements needed to fit the regression model. The “Quarter” column is simply the label for each time period. The next 2 rows are the crude (or adjusted
Discussion
As we have demonstrated above, the ITS approach to policy/program evaluation has several advantages. The approach is easy to do and provides powerful, easy-to-understand results. ITS controls for secular trends in the data and therefore reduces bias that might be present in a simple 2-time-period model (ie, simple pre–post measurement and analysis). ITS does not require adjustment for individual-level characteristics.4
There are 3 important threats to validity in ITS analyses. The most serious
Limitations
Although ITS has many strengths, there are important limitations to be aware of. First, estimating the level and slope parameters requires a minimum of 8 observations before and after the policy/program implementation in order to have sufficient power to estimate the regression coefficients. In our example, ITS could not reasonably be used to evaluate the impact of the program on prescribing until 8 quarters (2 years!) after the program began. Estimating monthly or even weekly data may be
Conclusion
ITS is a simple but powerful approach to policy/program evaluation. Although the approach has limitations, few statistical approaches are as elegant in design and powerful in audience impact. ITS is particularly useful when a randomized trial is infeasible or unethical. Because ITS is the strongest quasi-experimental design, its value in quality improvement and program evaluation cannot be understated. Paired with comprehensive qualitative data regarding the implementation of policies/programs,
References (34)
- et al.
Efficacy and safety of immediate-release methylphenidate treatment for preschoolers with ADHD
J Am Acad Child Adolesc Psychiatry
(2006) - et al.
Second opinions improve ADHD prescribing in a Medicaid-insured community population
J Am Acad Child Adolesc Psychiatry
(2009) - et al.
Methods for estimating confidence intervals in interrupted time series analyses of health interventions
J Clin Epidemiol
(2009) Analysis approaches to community evaluation
Eval Rev
(1997)- et al.
The value of interrupted time-series experiments for community intervention research
Prev Sci
(2000) - et al.
Experimental and Quasi-Experimental Designs for Generalized Causal Inference
(2002) - et al.
Segmented regression analysis of interrupted time series studies in medication use research
J Clin Pharm Ther
(2002) - et al.
How to use an article about quality improvement
JAMA
(2010) - et al.
Effects of a limit on Medicaid drug-reimbursement benefits on the use of psychotropic agents and acute mental health services by patients with schizophrenia
N Engl J Med
(1994) - et al.
Withdrawing payment for nonscientific drug therapy. Intended and unexpected effects of a large-scale natural experiment
JAMA
(1990)
Payment restrictions for prescription drugs under Medicaid. Effects on therapy, cost, and equity
N Engl J Med
Analysis of interrupted time series mortality trends: an example to evaluate regionalized perinatal care
Am J Public Health
Interrupted Time Series Analysis
Medicare part D and changes in prescription drug use and cost burden: national estimates for the Medicare population, 2000 to 2007
Med Care
Prior authorization for antidepressants in Medicaid: effects among disabled dual enrollees
Arch Intern Med
Use of atypical antipsychotic drugs for schizophrenia in Maine Medicaid following a policy change
Health Aff (Millwood)
Effects of prior authorization on medication discontinuation among Medicaid beneficiaries with bipolar disorder
Psychiatr Serv
Cited by (0)
The views expressed in this report are those of the authors and do not necessarily represent those of the US Department of Health and Human Services, the Agency for Healthcare Research and Quality or the American Board of Pediatrics Foundation.
The authors declare that they have no conflict of interest.
Publication of this article was supported by the Agency for Healthcare Research and Quality and the American Board of Pediatrics Foundation.