Change search
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • harvard1
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf
Design of a Semi-Automated and Continuous Evaluation System: Customized for Application in e-Health
Blekinge Institute of Technology, Faculty of Engineering, Department of Health.ORCID iD: 0000-0002-0316-548x
Blekinge Institute of Technology, Faculty of Computing, Department of Technology and Aesthetics.
Blekinge Institute of Technology, Faculty of Engineering, Department of Health.ORCID iD: 0000-0003-4312-2246
Blekinge Institute of Technology, Faculty of Engineering, Department of Health.ORCID iD: 0000-0001-9870-8477
Show others and affiliations
(English)Manuscript (preprint) (Other academic)
Abstract [en]

Background and Objectives

Survey-based evaluation of a system, such as measuring user’s satisfaction or patient-reported outcomes, entails a set of burdens that limits the feasibility, frequency, extendability, and continuity of the evaluation. Automating the evaluation process, that is reducing the burden of evaluators in questionnaire curation or minimizing the need for explicit user attention when collecting their attitudes, can make the evaluation more feasible, repeatable, extendible, continuous, and even flexible for improvement. An automated evaluation process can be enhanced to include features, such as the ability to handle heterogeneity in evaluation cases. Here, we represent the design of a system that makes it possible to have a semi-automated evaluation system. The design is presented and partially implemented in the context of health information systems, but it can be applied to other contexts of information system usages as well.

Method

The system was divided into four components. We followed a design research methodology to design the system, where each component reached a certain level of maturity. Already implemented and validated methods from previous studies were embedded within components, while they were extended with improved automation proposals or new features.

Results

A system was designed, comprised of four major components: Evaluation Aspects Elicitation, User Survey, Benchmark Path Model, and Alternative Metrics Replacement. All components have the essential maturity of identification of the problem, identification of solution objectives, and the overall design. In the overall design, the primary flow, process-entities, data-entities, and events for each component are identified and illustrated. Parts of some components have been already verified and demonstrated in real-world cases.

Conclusion

A system can be developed to minimize human burden, both for the evaluators and respondants, in survey-based evaluation. This system automates finding items to evaluate, creating questionnaire based on those items, surveying the users' attitude about those items, modeling the relations between the evaluation items, and incrementally changing the model to rely on automatically collected metrics, usually implicit indicators, collected from the users, instead of requiring their explicit expression of their attitudes. The system provides the possibility of minimal human burden, frequent repetition, continuity and real-time reporting, incremental upgrades regarding environmental changes, proper handling of heterogeneity, and a higher degree of objectivity.

Keywords [en]
Evaluation, Questionnaire, Information Systems, Patient Reported Outcome, User’s Attitude, Automation
National Category
Computer Systems
Identifiers
URN: urn:nbn:se:bth-18795OAI: oai:DiVA.org:bth-18795DiVA, id: diva2:1366417
Available from: 2019-10-29 Created: 2019-10-29 Last updated: 2019-11-18Bibliographically approved
In thesis
1. Evaluating Success Factors of Health Information Systems
Open this publication in new window or tab >>Evaluating Success Factors of Health Information Systems
2019 (English)Doctoral thesis, comprehensive summary (Other academic)
Abstract [en]

Health information systems are our technological response to the growing demand for health care. However, their success in their mission can be challenging due to the complexity of evaluating technological interventions in health care. In the series of studies compiled in this dissertation, we looked at the evaluation of these systems. We focused on the evaluation of factors that lead to success, where success is indicated by user satisfaction and can be induced by both intervention-specific and individual-specific factors.

Study 1 developed a method, called UVON, to elicit and organise the user-demanded qualities in the outcomes of the health information system intervention. Through the application of the UVON method in the FI-STAR project, an EU project which developed and deployed seven e-health applications in seven member countries, ten categories of quality and their subcategories were identified. These qualities formed two questionnaires, specific to the patient and health professional users. Through the questionnaires, the patients and health-professionals users evaluated and graded both the occurrence of those demanded qualities in the project outcomes and their general satisfaction.

Study 2 analysed the survey results to find out which of those ten qualities have the highest impact on satisfaction or can predict it better. Two partial least squares structural equation modelling (PLS-SEM) models were constructed, for the patient and health professionals, based on the Unified eValuation using ONtology (UVON) and survey outputs. The models showed that effectiveness is an important quality in creating satisfaction for both user groups. Besides, affordability for the health professionals and efficiency plus safety for the patients were the most influential. A satisfaction index is also introduced for simple and fast inferring of the changes in the outcome qualities.

Study 5 recruited outputs and learnings from studies 1 and 2 to design a system that partially automates the process of evaluating success factors in health information systems, making it continuous and real-time, and replacing hard-to-run surveys with automatically captured indicators and analytics.

Study 3 focused on individual-specific factors in using health information systems, particularly the technophilia personality trait. A short six-items instrument, called TechPH, was designed to measure technophilia in users, tuned for older users. The study recruited empirical data from the Swedish National Study on Aging and Care (SNAC) project. Two factors, labelled techAnxiety and techEnthusiams, are identified by the factor analysis method. A TechPH score was introduced as a scalar measurement of technophilia.

Study 4 elicited and discussed the ethical challenges of evaluating and researching health information systems. Both a scoping review and a novel systematic postulation approach were recruited to identify twenty ethical challenges. The identified ethical challenges were discussed and mapped into a three-dimensional space of evaluation stages, demanded qualities, and major involving entities (stakeholder and artefacts), which fosters further postulation of ethical challenges.

Place, publisher, year, edition, pages
Karlskrona: Blekinge Tekniska Högskola, 2019. p. 340
Series
Blekinge Institute of Technology Doctoral Dissertation Series, ISSN 1653-2090 ; 14
Keywords
Health Information Systems, Health Information Technology, Health Informatics, eHealth, Digital Health, Evaluation, Information Systems Evaluation, Health Technology Assessment, User Satisfaction, Technophilia, Evaluation and Research Ethics, System Design
National Category
Other Electrical Engineering, Electronic Engineering, Information Engineering
Identifiers
urn:nbn:se:bth-18799 (URN)978-91-7295-387-1 (ISBN)
Public defence
2019-12-17, J1640, Campus Gräsvik, Karlskrona, 14:00 (English)
Opponent
Supervisors
Available from: 2019-10-29 Created: 2019-10-29 Last updated: 2019-12-06Bibliographically approved

Open Access in DiVA

No full text in DiVA

Authority records BETA

Eivazzadeh, ShahryarFiedler, MarkusSanmartin Berglund, JohanAnderberg, PeterLarsson, Tobias

Search in DiVA

By author/editor
Eivazzadeh, ShahryarFiedler, MarkusSanmartin Berglund, JohanAnderberg, PeterLarsson, Tobias
By organisation
Department of HealthDepartment of Technology and AestheticsDepartment of Mechanical Engineering
Computer Systems

Search outside of DiVA

GoogleGoogle Scholar

urn-nbn

Altmetric score

urn-nbn
Total: 19 hits
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • harvard1
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf