Using the Hawthorne effect to examine the gap between a doctor's best possible practice and actual performance

https://doi.org/10.1016/j.jdeveco.2009.11.001Get rights and content

Abstract

Many doctors in developing countries provide considerably lower quality care to their patients than they have been trained to provide. The gap between best possible practice and actual performance (often referred to as the know-do gap) is difficult to measure among doctors who differ in levels of training and experience and who face very different types of patients. We exploit the Hawthorne effect–in which doctors change their behavior when a researcher comes to observe their practices–to measure the gap between best and actual performance. We analyze this gap for a sample of doctors and also examine the impact of the organization for which doctors work on their performance. We find that some organizations succeed in motivating doctors to work at levels of performance that are close to their best possible practice. This paper adds to recent evidence that motivation can be as important to health care quality as training and knowledge.

Introduction

Trained doctors and good medicine are clearly necessary for the delivery of health services in low-income countries. However, there is evidence that these inputs are not sufficient; many doctors who have the capacity to deliver quality care do not in fact do so. 1 In developed countries, the presumption that doctors do not always use their knowledge and skills in their patients' best interests–referred to as imperfect agency–drives much of the research on health care. In such settings, contracts and regulation are seen to improve the quality of care without increasing doctors' capacities. Clearly, most developing countries lag far behind developed countries in the capacities of their health care sectors, and they also fail to adequately regulate the behavior of health care providers. Thus, even in settings where capacity is clearly insufficient, imperfect agency may reduce quality further still.

One important step toward understanding the degree to which doctors underperform and how institutions can reduce or eliminate this behavior is to measure the gap between a doctor's best possible care and the care that he chooses to provide to his or her patients, sometimes referred to as the “know-do gap.” In this paper, we advance an experimental methodology that allows us to measure both best possible and actual care by a doctor performing the same activities with the same types of patients, and therefore to document this gap. We examine the gap between best and actual practice for two key activities in a sample of doctors from the Arusha region in Tanzania and then show how institutional features of these doctor's practices are correlated with the size of this gap.

In our study of health care quality, we discovered that our research team caused a distinct Hawthorne effect, in which the act of being observed alters the subjects' behavior. 2 In particular, when a doctor on our research team arrived to observe doctors in the course of their regular outpatient consultation, the observed doctors changed the way they practiced medicine and significantly improved the quality of care provided. Surprisingly, these same doctors gradually reverted to their normal behavior even while the research team was present. We suggest that the arrival of another doctor creates a sense in the subject doctor of being under high scrutiny because the observed doctor perceives a demand from a fellow doctor. This effect likely depends crucially on the shared training and profession of the researcher and the subject. However, because the researchers are passive and do not provide feedback, and because the subjects have no direct incentive to impress the researcher, the perceived level of scrutiny and therefore the demand for professional behavior both fall over time. The fact that the subject reacts to both high and low levels of scrutiny in the presence of the research team means that we can observe high and low levels of effort with the same quality measurement instrument and with a doctor's normal patients. The high scrutiny implied when the researcher first arrives alters the way the observed doctors treat their patients but it does not alter their capacity to provide care. Thus, the superior quality of care provided in the presence of the research team reveals an achievable, higher level of care that can be compared to the actual level of care provided at other times.

To demonstrate the potential for this research methodology, we examine two distinct measures of quality diagnostic quality (effort exerted to find the correct diagnosis) and whether the doctor ordered a lab test for the patient.

We measure diagnostic quality as the proportion of medically recommended questions and physical examinations actually asked or performed while examining the patient; quality is higher when doctors ask more questions and examine the patient more carefully. We show that the average doctor provides about 50% of the recommended inputs for the average patient but increases his or her provision of effort by approximately 10 percentage points (20%) when our research team first arrives. Importantly, for some organizations in our data, there is almost no gap between best and actual performance, whereas for other organizations, the gap is much higher. We argue that doctors who do not increase their performance significantly in the presence of the research team are those doctors who were already performing at high levels–levels close to their capacity–and who therefore cannot increase the quality of care when subjected to additional scrutiny. On the other hand, doctors who do exhibit large changes in performance upon the arrival of the research team are those who were not performing at levels close to best practice and who therefore can easily increase their input levels in response to additional scrutiny. In other words, doctors who normally work under high levels of scrutiny have already increased their effort, whereas those who normally work under low levels of scrutiny have not.

For the use of lab tests, however, it is more difficult to differentiate high from low quality simply by observing the doctor's activities. Low levels of use might indicate a facility that is not sufficiently careful in diagnosing its patients, but high levels of use may indicate supplier-induced demand. In addition, if patients select doctors according to their condition, one doctor might only see patients requiring lab tests while another doctor only sees patients who do not need tests. Thus, use of the laboratory by itself does not reveal quality. We propose that subjecting doctors to additional scrutiny by a peer may cause them to alter their behavior in favor of professional standards. Thus, the Hawthorne effect reveals whether a doctor believes he is using the laboratory in a professional or ethical manner. We find that, whereas most doctors increase their use of laboratory tests when the research team first arrives (and subsequently decrease their use), doctors in one organization have the opposite pattern: they significantly reduce their use of tests when the research team first arrives, allowing the rate to rise over time. Importantly, this organization is suspected of engaging in supplier-induced demand to the detriment of their patients. Thus, even when we cannot objectively evaluate an organization's activities, the behavior of doctors may suggest that they do not believe they are practicing at the best possible levels of care.

The association between the know-do gap and key institutional characteristics of a doctor's practice confirm the findings of Das and Hammer, 2007b, Leonard and Vialou, 2007 in which ability and practice quality were measured using different instruments. Specifically, these papers show that, when tested on their knowledge of medical protocols with case study patients (vignettes), most doctors exert far more effort than they do with their normal patients and that this know-do gap decreases when doctors have extrinsic motivation to exert effort. Das and Hammer (2007b) proxy for motivation with whether a doctor practices in the private or public sector and show that, when compared to the public sector, doctors in the private sector practice at levels of diagnostic quality that are closer to their ability. Similarly, Das and Hammer (2007b) proxy for motivation with the degree to which authority over fiscal and staffing decisions is decentralized to individual facilities and show that doctors who work under decentralized authority practice at levels closer to their ability than do doctors who work under centralized authority.

The findings in these papers rely on two untested assumptions about the relationship between quality as measured by vignettes and quality as measured by observation with regular patients. First, they assume that two doctors with similar scores on a vignette are, in fact, similar in their ability to diagnose actual patients. Second, they assume that two doctors with different practice quality scores do, in fact, differ in the quality of their practice. Because the vignette measures the ability of doctors to describe rather than implement diagnostic procedures, the first assumption would be violated if some doctors were good at describing procedures but unable to perform them in practice. The second assumption would be violated if the observed ability of a doctor depends on the types of patients he is diagnosing. If either of these two assumptions is violated and if the distribution of vignette-specific skills or patient characteristics is correlated with proxy measures of motivation, then concluding that motivation impacts practice quality is not justified. Consider a public sector doctor and a private sector who have identical vignette-measured ability but demonstrate different behavior with their patients. The differences in observed practice quality could be driven by motivation but it is possible that the patients at public facilities suffer from illnesses that do not require extra diagnostic procedures, whereas patients at private facilities suffer from illnesses that do require these procedures. The practices of these doctors could therefore differ because they see different patients, not because they have different motivation to treat their patients. Thus, it is possible that differences between ability and practice are artifacts of the two instruments used. The use of the Hawthorne effect, on the other hand, allows us to measure ability and practice quality with only one instrument used under normal working conditions with regular patients, comparing doctors to themselves.

In the following section, we review the data on doctor quality and determinants of motivation used in the paper. Section 2 develops the link between the impact of scrutiny implied by the Hawthorne effect and a doctor's motivation, showing that the Hawthorne effect can reveal the existence of gaps between best practice and actual performance. Section 3 examines the association between the proxy measure of motivation and the know-do gap exposed by the Hawthorne effect. In addition, we discuss the significance of these changes in behavior and their implications for patient outcomes.

Section snippets

Data and instruments

The primary data used in this paper were collected over a period of 18 months from October of 2001 through March of 2003. Thirty-nine health facilities in the rural and urban areas of Arusha region of Tanzania were visited at least two times each. Doctors who were present at these facilities during any of the visits were evaluated for competence and performance using case–study patients and direct observation respectively. Direct observation allows us to measure both quality and whether the

The Hawthorne effect as additional scrutiny

The Hawthorne effect refers to a situation in which individuals' behavior changes when they realize they are being observed. It is characterized by a positive but temporary change in some measurable behavior in a situation in which there was no deliberate attempt by the observer to affect behavior (Benson, 2000). The doctors observed in Tanzania were told explicitly that the research would not impact them in any way however, they may have reacted to the mere presence of other doctors as if

Analysis

To assess whether the size of the Hawthorne effect can be explained with our proxy measures of motivation, we examine the impact in two different data sets. We look first at the reaction to the arrival of the research team, and second at the reaction to the continuing presence of the research team. The regressions discussed in the section are probit regression of the probability that an input or lab test was provided. The coefficients reported represent the percentage point increase in the

Conclusion

In a number of medical institutions in the Arusha region of Tanzania, we exploited the Hawthorne effect to measure changes in individual doctors' quality of care as a reaction to both the arrival of a research team and the continued presence of the research team. We show that the average doctors in our sample are capable of providing much higher levels of diagnostic quality than they generally provide when not under intense scrutiny and that this enhanced quality could improve important

Acknowledgments

The data used in this paper were collected under funding from NSF Grant 00-95235 and The World Bank, and with the assistance of R. Darabe, M. Kyande, S. Masanja, H. M. Mvungi and J. Msolla. The authors are solely responsible for the data contained herein. We extend our appreciation to the Commission for Science and Technology (COSTECH) for granting permission to perform this research. The paper has benefitted from the comments of the audiences at The World Bank and the Center for Global

References (32)

  • P.G. Benson

    The Hawthorne Effect

  • Nazmul Chaudhury et al.

    Ghost doctors : absenteeism in Bangladeshi health facilities

    World Bank Economic Review

    (2004)
  • Jishnu Das et al.

    Location, location, location: residence, wealth and the quality of medical care in Delhi, India

    Health Affairs

    (2007)
  • Jishnu Das et al.

    Variations in doctor effort: evidence from Paraguay

    Health Affairs

    (2007)
  • Jishnu Das et al.

    The quality of medical advice in low-income countries

    Journal of Economic Perspectives

    (2008)
  • D. Filmer et al.

    Weak links in the chain: a diagnosis of health policy in poor countries

    World Bank Research Observer

    (2000)
  • Cited by (57)

    • Can patients improve the quality of care they receive? Experimental evidence from Senegal

      2022, World Development
      Citation Excerpt :

      We add to this body of work by showing that provider effort is largely independent of patient effort (i.e. symptom disclosure) – even though provider and patient inputs are complementary to reach high levels of quality of care. In line with previous work (Leonard & Masatu, 2010b, 2010a; Mohanan et al., 2015), we find evidence of the existence of know-do gaps, as providers’ failure to invest effort in consultations is not driven by a lack of knowledge. With a simple vignette experiment, we even show that providers have the necessary clinical skills required to take the history of a hypothetical patient volunteering little information but fail to do this in practice.

    • Infection prevention and control compliance in Tanzanian outpatient facilities: a cross-sectional study with implications for the control of COVID-19

      2020, The Lancet Global Health
      Citation Excerpt :

      Third, by observing infection prevention and control compliance, we might have changed provider behaviour through the Hawthorne effect. However, this measurement method is likely to be more reliable than self-reports of behaviour, and our analysis of the order number of patients observed provided no strong evidence of a Hawthorne effect, although it has been noted in other studies.20 Fourth, we used endline data from a randomised controlled trial of a quality-improvement programme.

    View all citing articles on Scopus
    View full text