Assessing the reliability of household expenditure data: Results of the World Health Survey
Introduction
Household expenditure data has been used extensively for monitoring general household living standards, well being and consumption patterns [1]. More recently, considerable attention has been paid for monitoring household expenditures on health with a view for determining if the need to pay for services prevents some people from seeking or continuing care, and results in severe financial hardship or impoverishment for others [2], [3], [4]. This literature has grown considerably over the last five years, with analysts using available expenditure data, including the Living Standard Measurement Survey (LSMS) supported by the World Bank, Household Budget Surveys (HBS), Income and Expenditure Surveys (IES) and Socio-Economic Surveys (SES) [5], [6], [7], [8].
There has long been a concern with the accuracy of expenditure data reported in household surveys, often linked to concerns about the abilities of households to remember a multitude of different types of expenditures accurately [9], [10], [11]. Measurement error can be introduced at any stage of a survey: design of the survey instrument, data collection, or data entry [12]. This is partly because the household expenditure surveys are among the most difficult and expensive surveys to field [13].
While these concerns are well established, there has been little attempt to understand the extent to which phrasing questions in different ways can influence the response to health expenditure questions, and whether different types of surveys produce consistent results.
We contribute to this literature by comparing two ways of seeking information on health expenditure developed in the World Health Survey (WHS), and then also consider the comparability of the estimated expenditures derived from other surveys undertaken in the same countries.
The WHS was launched by the World Health Organization to strengthen national capacity to monitor critical health inputs, outputs and outcomes [14]. They collected information on total household expenditure with a breakdown that included health expenditures, together with a wide range of indicators on health status, health service utilization, risk factors, and the perceived responsiveness of the health system. This makes the WHS appealing to policy makers and researchers seeking information on diverse topics including the assessment of inequality of health and in intervention coverage across different socio-economic groups. World Health Surveys have been implemented in 72 countries using standard questionnaires and many of the country data sets have recently been put into the public domain (http://www.who.int/healthinfo/survey/en/index.html).
Section snippets
Analysis framework
Reliability refers to the repeatability or consistency of a set of measurements or measuring instrument [15]. A measure is considered reliable if it would give us the same result over and over assuming that what we are measuring is not changing. Reliability could be characterized as either internal or external. Internal reliability compares two sets of data on the same subject using different measures. In World Health Survey we compared the difference between the total reported in response to
External reliability: results from test–retest in the WHS
Fig. 1 reports the ICCs for the test–retest responses for total household expenditure and expenditures on education, food and health. Each vertical bar depicts a country, and the range shows the 95% confidence intervals around the mean estimate of the ICC. For most countries, the average value of the ICC is above 0.6 for all items, which is generally considered to imply good external reliability [19], [20]. The lowest for household expenditure is 0.28, for food is 0.19, for education is 0.39
Discussion
The WHS will be a major source for health and health system related studies. In the area of health financing studies, including out-of-pocket health expenditure, financial catastrophe and impoverishment by health payment, the WHS has great potential to fill in the gaps where no appropriate household surveys exist or where the existing surveys are not up to date. Information on quality and the comparability of the WHS is crucial for researchers in analysing the data and interpreting the results.
Conclusion
The WHS has great applicability to a range of health care financing studies. Countries need timely information to evaluate their health policies, manage their health systems and monitor progress. The WHS may be best viewed as another source of survey data to supplement the information provided by routine national information systems.
In this study we found that the expenditure data in the WHS are reliable based on the test–retest estimates. The aggregated total gives higher non-zero response
Conflict of interest
No conflict of interest declared.
Acknowledgements
We thank Somnath Chatterji for his valuable comments in the early draft. The authors are also grateful to Nirmala Naidoo for her assistance on various questions related to the procedure in conducting the World Health Survey.
References (25)
- et al.
Understanding the impact of eliminating user fees: utilization and catastrophic health expenditures in Uganda
Social Science and Medicine
(2006) - et al.
Effect of payments for health care on poverty estimates in 11 countries in Asia: an analysis of household survey data
Lancet
(2006) - et al.
Household catastrophic health expenditure: a multicountry analysis
Lancet
(2003) Towards good practice for health statistics: lessons from the millennium development goal health indicators
The Lancet
(2007)Case A. Analysis of household expenditures
(1988)- et al.
Catastrophic household expenditure for heath care in a low income society: a study from Nouna District, Burkina Faso
Bulletin of the World Health Organization
(2006) - et al.
Indian community health insurance schemes provide partial protection against catastrophic health expenditure
BMC Health Services Research
(2007) - et al.
Detecting changes in financial protection: creating evidence for policy in Estonia
Health Policy and Planning
(2006) - et al.
Catastrophe and impoverishment in paying for health care: with applications to Vietnam 1993–1998
Health Economics
(2003) The Consumer Expenditure Survey: a comparative analysis
Monthly Labor Review
(1994)