Evaluation: Salvation or Nemesis of Medical Informatics?

Dr. Jochen R. Möehr, School of Health Information Science, (University of Victoria)

Abstract

Evaluation components have become a mandatory part of medical informatics projects and a condition for funding. One may wonder, however, what the current information technology landscape would be like, had it been required to pass through the needle's ear of evaluation and proof of effectiveness. A number of questions arise:

• Are we promoting or stifling the advancement of Medical Informatics with insisting on evaluation?
• Is evaluation a prerequisite for progress or a response to the quest for accountability?
• Is evaluation as we currently understand and practice it all there is to it?

The presentation will review some aspects of the history and the current state of evaluation in medical informatics and summarize recent important publications. It will be shown that a comprehensive, albeit not coherent, theory and methodology exist, that their application is lagging, and that the prevailing paradigm for evaluation is stifling. This prevailing paradigm is the objectivist approach, which, in short, assumes that it is necessary and possible to objectively measure the characteristics of information resources and their environment, and that this is the basis for insights and progress based on answering hypotheses formulated at the outset of the study. An alternative is the subjectivist approach, which in contrast seeks to take maximum advantage of the educated subjectivity of the investigator, who uses an approach of immersion into the studied environment in an iterative approach towards increased understanding and insight. The objectivist approach is burdened by its very rigor, by the efforts to ensure objectivity. As a result evaluation efforts are expensive, incur difficulties to recruit subjects, require distorting artificial situations as a consequence of which the generalizability that they purport to facilitate is jeopardized. In addition, the insights gained often appear of little use to stakeholders, and there is a danger that important questions may not get asked, let alone answered. In contrast, the open-endedness of the subjectivist approach and its constant feedback and correction allow to retrace steps that are not productive, and to concentrate on comprehensive answers to important questions in close contact with stake holders. Examples of both approaches from health informatics will be reviewed. It will be argued that the subjectivist approach should guide evaluation projects and determine the appropriate application of objectivist approaches. Important extensions of this combined methodology will be identified. These recent methodological advances signal the budding of an evaluation methodology appropriate to the specific requirements of health informatics. So, while evaluation is justly given a pivotal role in medical informatics, it requires serious attention and methodological extensions in order to be beneficial. As a consequence, we are challenged to not only commit to the use evaluation in medical informatics research and development, but to continue to make evaluation itself the subject of research and education in our field.

About Dr. Jochen Moehr

Dr. Moehr is a physician who spent a decade on clinical research and practice. He has then been involved in research, teaching and the provision of service functions in Medical Informatics since 1970. He joined the School of Health Information Science (HIS) at the University of Victoria in 1986, after 10 years of developing and overseeing the Medical Informatics Program of the University of Heidelberg, the oldest program of this kind in Germany.

In 1992, he was awarded the medal of merit of the faculty of the University of Heidelberg for his achievements in the development of the educational program in medical informatics. From 1977 until 1986 he was professor at the University of Heidelberg and Director of the Division of Medical Informatics with responsibility for the development of a hospital information system for that university's large hospital complex and associated research and teaching facilities. He also conducted research on the use of computers in medical practices since the early seventies.

His current research interests include didactics in medical informatics, information systems in health care, computer based patient records, hospital information systems, information systems for office practice, and community health. At the University of Victoria he has headed up a team developing a cost effective, reliable, secure and confidential wide area network for communication of health data among heterogeneous health care computer systems.

Dr. Moehr is currently Leader of the Informatics Tools Collaboratory of HealNet, a Canadian Network of Centres of Excellence in Health Research. HealNet focuses on evidence based decision support in the clinical area, as well as in administration of health, and unites researchers at universities, from the public, and from the private sector, from across Canada.

Dr. Moehr is a member of GMDS, the German learned society for health informatics, GI, the German learned society for computer science, AMIA, the American Medical Informatics Association, COACH, Canada's Health Informatics Association, and Sigma Xi, the Scientific Research Society. He is a Fellow of the American College of Medical Informatics. He has served in many capacities in these societies, including being a member of the board of GMDS for many years.

Dr. Moehr chaired the Scientific Academic and Research Special Interest Group (SARSIG) of COACH, Canada's Health Informatics Association, from 1991 to 1996. He organized national and international conferences on the use of computers in various fields of medical practice, health care and education. Dr. Moehr is also a member of the editorial board of several journals which include Methods of Information in Medicine, Computer Methods and Programs in Biomedicine, and Computers in Biology and Medicine.

Back to Conference Program

Questions? Email them to conference@interactivehealth.org
© Copyright 1999. Interactive Health.