Background

Health information systems have emerged as a major component in our response to the trends of rising demands in health care. The insight being gained from the evaluation of those systems can critically influence the shaping of the response. Summative or formative evaluation of health information systems assesses their quality, acceptance, and usefulness, creates insight for improvement, discriminates between options, and refines future development strategies. But the evaluation of health information systems can be challenging due to the propagation of their impacts through multiple socio-technological layers till the ultimate recipients, their heterogeneity and fast evolvement, and the complexity of health care settings and systems.

Aim

This thesis tries to explain the challenges of evaluation of health information systems with a narrow down on determining evaluation aspects and to propose relevant solutions. The thesis goes for solutions that mitigate heterogeneity and incomparability, recruit or extend available evaluation models, embrace a wide context of application, and promote automation.

Method

The literature on health information systems evaluation, methods of dealing with heterogeneity in other disciplines of information systems, and ontology engineering were surveyed. Based on the literature survey, the UVON method, based on ontology engineering, was first developed in study 1. The method was applied in FI-STAR, a European Union project in e-Health with 7 use-cases, for summative evaluation of the individual and whole e-health applications. Study 2, extended the UVON method for a formative evaluation during the design phase.

Results

Application of the UVON method resulted in evaluation aspects that were delivered to the seven use-cases of the FI-STAR project in the form of questionnaires. The resulted evaluation aspects were considered sensible and with a confirming overlap with another highly used method in this field (MAST). No significant negative feedback from the FI-STAR use-case owners (n=7) or the respondents (n=87 patients and n=30 health professionals) was received or observed.

Conclusion

In the evaluation of health information systems --possibly also in other similarly characterized systems-- ontology engineering methods, such as the proposed UVON method, can be applied to create a flexible degree of unification across a heterogeneous set of evaluation aspects, import evaluation aspects from other evaluation methods, and prioritize between quality aspects in design phase. Ontologies, through their semantic network structures, can capture the extracted knowledge required for evaluation, facilitate computation of that knowledge, promote automation of evaluation, and accommodate further extensions of the related evaluation methods by adding new features to their network structure.