The effective evaluation of health information technology (HIT) is currently a major challenge. It is essential that applications we develop are usable, meet user information needs and are shown to be safe. Furthermore, to provide appropriate feedback to designers of systems new methods for both formative and summative evaluation are needed as applications become more complex and distributed. To ensure system usability a variety of methods have emerged from the area of usability engineering that have been adapted to healthcare. The authors have applied methods of usability engineering, working with hospitals and other healthcare organizations designing and evaluating a range of HIT applications. We describe how our approach to doing portable low-cost usability testing has evolved to the use of clinical simulations conducted in-situ, within real hospital and clinical units to rapidly evaluate the usability and safety of healthcare information systems both before and after system release. We discuss how this approach was extended to development of methods for conducting in-situ clinical simulations in a range of clinical settings.