Our objective was to compare the inter-observer level of agreement in diagnosing pneumonia using the World Health Organization (WHO) guidelines for the interpretation of radiographs. We conducted a prospective study in a pediatric emergency room. Fifteen observers (13 pediatricians, 2 radiologists) interpreted 200 pediatric (<5 years old) chest radiographs using the WHO guidelines. Observers were blinded to the clinical presentation. RESULTS were analyzed for kappa values. Individual readings were compared to two "gold standard" teams: (1) radiologist and pediatrician and (2) two radiologists.
Results: Alveolar pneumonia, non-alveolar pneumonia, and no pneumonia were found (by radiologists) in 12.8%, 2.7%, and 78.6% of readings, respectively. The mean kappa values for alveolar pneumonia, non-alveolar pneumonia, and no pneumonia of observers versus the team consisting of a radiologist and a pediatrician were 0.73, 0.23, and 0.61, respectively. For non-alveolar pneumonia, the mean kappa value was higher for the gold standard consisting of a radiologist and a pediatrician when compared to the two-radiologist team. Pediatricians overdiagnosed "non-alveolar pneumonia" compared with radiologists. In contrast, for the alveolar pneumonia and no-pneumonia diagnoses, no significant differences were found.
Conclusions: The WHO guidelines for interpretation of chest radiographs result in high level of agreement between readers for the definition of "alveolar pneumonia" and "no pneumonia" but poor agreement for non-alveolar pneumonia. The disagreement with regard to the latter was associated with overdiagnosis by pediatricians, which may lead to overtreatment. We believe that radiographic non-alveolar pneumonia should not be an endpoint for clinical trials and research, nor should it be implemented in clinical setting.