Health information systems are our technological response to the growing demand for health care. However, their success in their mission can be challenging due to the complexity of evaluating technological interventions in health care. In the series of studies compiled in this dissertation, we looked at the evaluation of these systems. We focused on the evaluation of factors that lead to success, where success is indicated by user satisfaction and can be induced by both intervention-specific and individual-specific factors.

Study 1 developed a method, called UVON, to elicit and organise the user-demanded qualities in the outcomes of the health information system intervention. Through the application of the UVON method in the FI-STAR project, an EU project which developed and deployed seven e-health applications in seven member countries, ten categories of quality and their subcategories were identified. These qualities formed two questionnaires, specific to the patient and health professional users. Through the questionnaires, the patients and health-professionals users evaluated and graded both the occurrence of those demanded qualities in the project outcomes and their general satisfaction.

Study 2 analysed the survey results to find out which of those ten qualities have the highest impact on satisfaction or can predict it better. Two partial least squares structural equation modelling (PLS-SEM) models were constructed, for the patient and health professionals, based on the Unified eValuation using ONtology (UVON) and survey outputs. The models showed that effectiveness is an important quality in creating satisfaction for both user groups. Besides, affordability for the health professionals and efficiency plus safety for the patients were the most influential. A satisfaction index is also introduced for simple and fast inferring of the changes in the outcome qualities.

Study 5 recruited outputs and learnings from studies 1 and 2 to design a system that partially automates the process of evaluating success factors in health information systems, making it continuous and real-time, and replacing hard-to-run surveys with automatically captured indicators and analytics.

Study 3 focused on individual-specific factors in using health information systems, particularly the technophilia personality trait. A short six-items instrument, called TechPH, was designed to measure technophilia in users, tuned for older users. The study recruited empirical data from the Swedish National Study on Aging and Care (SNAC) project. Two factors, labelled techAnxiety and techEnthusiams, are identified by the factor analysis method. A TechPH score was introduced as a scalar measurement of technophilia.

Study 4 elicited and discussed the ethical challenges of evaluating and researching health information systems. Both a scoping review and a novel systematic postulation approach were recruited to identify twenty ethical challenges. The identified ethical challenges were discussed and mapped into a three-dimensional space of evaluation stages, demanded qualities, and major involving entities (stakeholder and artefacts), which fosters further postulation of ethical challenges.