Evaluation of a computer-assisted data entry procedure (including Teleform) for large-scale mailed surveys

doi:10.1016/S0010-4825(03)00012-X

Computers in Biology and Medicine

Volume 33, Issue 5, September 2003, Pages 425-437

https://doi.org/10.1016/S0010-4825(03)00012-X Get rights and content

Abstract

Computer-assisted data entry procedures for self-completion population surveys are previously unreported. We used Teleform for a questionnaire to registrants (aged 50+) with general practices in North Staffordshire, UK (n=8995). The survey achieved a 77% response rate. 98% (n=6647) of returned questionnaires were scannable. These questionnaires were checked (and corrected) before scanning for entries that may be inaccurately read by the software. Three per cent of 1020 scanned questionnaires checked (0.041% of all questionnaire items) had a data entry error. Systematic processes enable accurate survey data transfer to be achieved with Teleform. However, many administrative and technical issues need addressing.

Introduction

Self-completion surveys are a key method in assessing, for example, outcomes of treatment programmes or healthcare needs. When undertaking surveys, medical researchers are concerned with minimising data error. Data error affects the study's internal validity and may compromise the findings and implications for clinical practice or healthcare policy. Recent technological advances have resulted in the development of computer-assisted data entry systems and hence the availability of new methods for processing survey data. Computer-assisted data entry systems offer an alternative to traditional methods of survey data entry (manual and double manual) and can be achieved by installation of scanners and specialist forms processing software for optical mark reading (OMR) and intelligent character recognition (ICR). Paper based questionnaires are processed through a scanner, the data is interpreted by the data entry software, verified by the operator and exported to a database for analysis.

There is limited evidence about the use of computer-assisted data entry for large-scale self-completion population surveys [1], [2]. A review of the medical literature revealed 10 articles that made reference to automatic data entry systems in other research or clinical settings. There is conflicting evidence within these papers about the benefits of computer-assisted data entry over manual data entry regarding, for example, the accuracy of the data. It is also difficult to compare findings as different attributes (number of questionnaires, questionnaire items, or individual variables) have been measured, in different settings with different levels and type of resources. Most published studies in the literature have used Teleform (Cardiff Software). These studies have not, however, used Teleform in a large-scale self-completion community survey. We have used Teleform software (Teleform Standard version 6.1) during a postal survey about knee pain and general health in the general population. We report, therefore, on the error rates found within our scanned data, the pros and cons of using this approach, and on the practical experience of using the new system.

Section snippets

Teleform

Teleform is a software application that enables the creation of forms (e.g. questionnaires) for collecting data and reads the returned data using a scanner. After processing the data, Teleform can automatically export this into other databases for analysis [3].

Teleform has three modules. Firstly, Teleform Designer is used for the questionnaire and database design. Questions can be designed in a number of formats. “Choice” and “Entry” fields enable respondents to tick boxes or fill in bubbles.

Survey response rate

Questionnaires using automatic forms processing can be designed to look very similar to traditionally designed questionnaires. They differ, however, in three ways, as they require:

•
cornerstone markings to enable the software to locate and then read marks from the page,
•
a numeric code in one corner that holds reference information about the form,
•
“constrained print” and “comb fields” to record accurately numerical and string variables.

These features are automatically generated by the software and

Discussion

There are no published studies that have used computer-assisted data entry systems for large-scale self-completion health surveys. Studies that have used computer-assisted data entry systems elsewhere report conflicting evidence on their benefits. We have used a computer-assisted data entry system during a postal survey about general health and knee pain. The pros and cons of implementing this new approach in this setting are discussed below.

Summary

We have implemented a computer-assisted data entry system and used Teleform software to process self-completed questionnaire data. Our questionnaire was of quite complex design and was completed (unassisted) by men and women (aged 50 and over) in the community. We implemented integrated data checking and scanning procedures and found a very low questionnaire item data entry error rate (overall error rate of 0.041%). This error rate is consistent with studies that have used Teleform in other

Acknowledgements

The authors would like to acknowledge the NHS Executive (West Midlands) New Blood Fellowship committee for awarding funding for this project and the Haywood Rheumatism Research and Development Foundation (HRRDF) (North Staffordshire) who supported the survey through a project grant. The authors would like to thank all of the people in Primary Care Sciences Research Centre who helped with the administration of the knee pain survey. Special thanks go to Joanne Bailey for organisation and Paul

References (16)

C.K. Jorgensen et al.
Validation of automated forms processing, A comparison of Teleform with manual data entry
Comput. Biol. Med.
(1998)
A. Bowling
Research Methods in Health
(1997)
J.H. Abramson et al.
Survey Methods in Community Medicine
(1999)
Cardiff Software Inc., Teleform Standard User Guide Version 6.1, San Marcos,...
C. Jinks et al.
A brief screening tool for knee pain in primary care. 1. Validity and reliability
Rheumatology
(2001)
J. Ware et al.
SF 36 Health Survey. Manual and Interpretation Guide
(1993)
A.S. Zigmond et al.
The hospital and anxiety depression scale
Acta Psychiatry Scand.
(1983)
N. Bellamy, WOMAC Osteoarthritis Index, A Users Guide, Health Services Centre, McMaster University, London,...

There are more references available in the full text version of this article.

Cited by (17)

Automated entry of paper-based patient-reported outcomes: Applying deep learning to the Japanese orthopaedic association back pain evaluation questionnaire
2024, Computers in Biology and Medicine
Health-related patient-reported outcomes (HR-PROs) are crucial for assessing the quality of life among individuals experiencing low back pain. However, manual data entry from paper forms, while convenient for patients, imposes a considerable tallying burden on collectors. In this study, we developed a deep learning (DL) model capable of automatically reading these paper forms.
We employed the Japanese Orthopaedic Association Back Pain Evaluation Questionnaire, a globally recognized assessment tool for low back pain. The questionnaire comprised 25 low back pain-related multiple-choice questions and three pain-related visual analog scales (VASs). We collected 1305 forms from an academic medical center as the training set, and 483 forms from a community medical center as the test set. The performance of our DL model for multiple-choice questions was evaluated using accuracy as a categorical classification task. The performance for VASs was evaluated using the correlation coefficient and absolute error as regression tasks.
In external validation, the mean accuracy of the categorical questions was 0.997. When outputs for categorical questions with low probability (threshold: 0.9996) were excluded, the accuracy reached 1.000 for the remaining 65 % of questions. Regarding the VASs, the average of the correlation coefficients was 0.989, with the mean absolute error being 0.25.
Our DL model demonstrated remarkable accuracy and correlation coefficients when automatic reading paper-based HR-PROs during external validation.
Intuitive eating is associated with glycaemic control in adolescents with type I diabetes mellitus
2016, Appetite
While there have been considerable advances in the medical management of type 1 diabetes mellitus (T1DM), for many, glycaemic control remains substandard. Nutrition and eating behaviour are important additional factors to consider with regards to T1DM management and outcomes. Intuitive eating is one such factor, and has not previously been investigated in T1DM. With this in mind, we undertook a study examining the relationship between intuitive eating and glycaemic control in adolescents with T1DM.
A case-control study of adolescents with established T1DM, and age/sex matched controls was conducted. Demographic information, the Intuitive Eating Scale (IES), and HbA₁c were collected. Statistical analysis was undertaken to explore associations between the IES and HbA1c as a marker of glycaemic control.
Data on 38 adolescents with T1DM, and 39 age/sex matched controls were obtained. Those with T1DM had significantly lower (by 0.5 SD) IES scores compared to controls (p = 0.009). Higher values of both total IES and the Eating for physical rather than emotional reasons subscale were associated with lower HbA1c: HbA1c 22% lower/whole unit increase in total IES mean score, HbA1c 11% lower/whole unit increase in Eating for physical rather than emotional reasons mean score, p = 0.017 and p = 0.009 respectively.
In adolescents with T1DM, there appears to be a strong association between intuitive eating, in particular the effect of emotion on eating, and glycaemic control. In addition, those with T1DM have lower scores for their intuitive eating behaviour compared to controls. Emotional eating could be a future target for screening and potentially intervening in those with T1DM, as part of a wider treatment package to improve glycaemic control. Continuing efforts are needed to fully understand the important dynamics of diabetes, adolescence, diet, emotion, and how these factors affect long term outcomes in those with T1DM.
Assessment of the impact of rapid syphilis tests on syphilis screening and treatment of pregnant women in Zambia
2015, International Journal of Gynecology and Obstetrics
Citation Excerpt :
Ten data collectors from Kalomo District were trained and tested in research ethics, standardized data extraction, and management techniques prior to data collection. Data capture was done using TeleForms (Hewlett-Packard, Palo Alto, CA, USA) [14]. Pregnant women attending their first ANC visit during the observation period were randomly selected from the MOH-approved Safe Motherhood (SMH) register using a sequential skip pattern dependent on ANC volume at the facility.
To evaluate the impact of rapid syphilis tests (RSTs) on syphilis testing and treatment in pregnant women in Kalomo District, Zambia.
In March 2012, health workers at all 35 health facilities in Kalomo Distract were trained in RST use and penicillin treatment. In March 2013, data were retrospectively abstracted from 18 randomly selected health facilities and stratified into three time intervals: baseline (6 months prior to RST introduction), midline (0–6 months after RST introduction), and endline (7–12 months after RST introduction).
Data collected on 4154 pregnant women showed a syphilis-reactive seroprevalence of 2.7%. The proportion of women screened improved from baseline (140/1365, 10.6%) to midline (976/1446, 67.5%), finally decreasing at endline (752/1337, 56.3%) (P < 0.001). There was no significant difference in the proportion of syphilis-seroreactive pregnant women who received 1 dose of penicillin before (1/2, 50%) or after (5/48, 10.4%; P = 0.199) RST introduction with low treatment rates throughout.
With RST scale-up in Zambia and other resource-limited settings, same-day test and treatment with penicillin should be prioritized to achieve the goal of eliminating congenital syphilis.
Evaluation of a Teleform-based data collection system: A multi-center obesity research case study
2014, Computers in Biology and Medicine
Citation Excerpt :
Others have investigated the data fidelity resulting from recognition software [2–8]. These studies have identified factors such as illegible handwriting, incomplete or faint markings, and responses placed outside of data entry fields as common issues leading to data inaccuracies [2,4–6]. However, none of these investigations to our knowledge have reported use of customizable scripts.
Utilizing electronic data capture (EDC) systems in data collection and management allows automated validation programs to preemptively identify and correct data errors. For our multi-center, prospective study we chose to use TeleForm, a paper-based data capture software that uses recognition technology to create case report forms (CRFs) with similar functionality to EDC, including custom scripts to identify entry errors. We quantified the accuracy of the optimized system through a data audit of CRFs and the study database, examining selected critical variables for all subjects in the study, as well as an audit of all variables for 25 randomly selected subjects. Overall we found 6.7 errors per 10,000 fields, with similar estimates for critical (6.9/10,000) and non-critical (6.5/10,000) variables—values that fall below the acceptable quality threshold of 50 errors per 10,000 established by the Society for Clinical Data Management. However, error rates were found to widely vary by type of data field, with the highest rate observed with open text fields.
The impact of fine-tuning of optical recognition system on database reliability
2012, Computers in Biology and Medicine
Citation Excerpt :
The rate of errors is related to the specific method and should be investigated in detail before data acquisition. A well setted ORS seems to offer the opportunity to reach a final accuracy comparable to DDE, the rate of error with the use of Teleform [3,4,21] ranging between 0.04% [4] and 0.07% [3], a single study performed with a system developed in Formic reporting an error rate of 0.02% [2]. A 0.1% absolute rate of error is usually considered as acceptable cut-point [24].
Although optical reading systems are important tools to transfer data from a paper form to electronic databases, the impact of system fine-tuning on the final error rate is not usually considered. At the end of a multi-step process involving paper form design training of operators, and fine-tuning procedure, the final rate of error can be reduced from 0.65% to 0.05%. Fine-tuning should be introduced as a standard procedure while using optical reading systems.
A patient-completed and optically read data acquisition system for clinical trials
2011, Contemporary Clinical Trials
Data collection and management within multicentre clinical trials can be challenging. We describe an adaptation of Teleform® technology to enable data recording by patients and their families on teleforms faxed and optically read directly into an electronic database, eliminating the need for case report forms. Preliminary results from a modest study sample size support the use of optically read forms for data collection by patients and their families, requiring only a pen, paper, and fax machine at participating sites.

View all citing articles on Scopus

^☆: This study was funded by the NHS Executive (West Midlands) as part of a New Blood Research Training Fellowship. The Haywood Rheumatism Research and Development Foundation (HRRDF) (North Staffordshire) funded the survey costs.

View full text

Evaluation of a computer-assisted data entry procedure (including Teleform) for large-scale mailed surveys☆

Abstract

Introduction

Section snippets

Teleform

Survey response rate

Discussion

Summary

Acknowledgements

Comput. Biol. Med.

Research Methods in Health

Survey Methods in Community Medicine

A brief screening tool for knee pain in primary care. 1. Validity and reliability

Rheumatology

SF 36 Health Survey. Manual and Interpretation Guide

The hospital and anxiety depression scale

Acta Psychiatry Scand.