Disaster simulations (drills) are widely used throughout the world and are considered a fundamental tool for evaluation and improvement of local disaster response capacity. Despite this, no generally accepted methodology exists for quantitative evaluation of the medical response to a disaster drill. We therefore set out to develop and prospectively test a comprehensive method to assess both medical provider and organizational performance during a disaster simulation. Because disasters disproportionately affect the populations of developing countries, we designed these methods to be sufficiently flexible to be applicable in both the developed and the developing world. Objective outcome measures were identified for each component of disaster medical response and were incorporated into 3 data collection instruments. The derived methods were applied to a multiagency disaster simulation in Guatemala City, Guatemala. On the basis of this pilot study, suggested modifications and recommendations were made. The ability to objectively identify the specific strengths and weaknesses of an emergency medical services systems' medical response to a disaster is an important step toward optimizing system performance. On the basis of our experience, we recommend the incorporation of objective evaluation methods such as these into every disaster simulation.