Change search
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • harvard1
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf
Toward Patient-centered, Standardized, and Reproducible Approaches of Evaluating the Usability of mHealth Chronic Disease Self-management Systems for Diabetes
Blekinge Institute of Technology, Faculty of Computing, Department of Creative Technologies.ORCID iD: 0000-0002-9854-7690
2018 (English)Doctoral thesis, comprehensive summary (Other academic)
Abstract [en]

Diabetes is a chronic disease affecting 422 million patients worldwide according to World Health Organization data with 30.3 million in the United States and 64 million in Europe. The prevalence speaks to the need for improved ways to support patients in disease self-management. mHealth solutions are increasingly used for this; however, usability is a current challenge affecting patients’ mHealth use. Recent literature emphasizes an increased focus on patient-centeredness in diabetes care, user-centeredness in chronic disease mHealth system design and standardized, systematic approaches for usability evaluation. The aim of this thesis and its individual studies was to incorporate these foci into the evaluation of two mobile health self-management systems for diabetes.

Study I used ISO standard 9241-11 to examine the relationship between selected group characteristics of diabetes patients on specific interaction outcomes to quantitatively identify needed system modifications. Study II utilized a multi-method design to assess diabetes patients’ mHealth usage and combined two novel analytic methods to structure and analyze results. Study III used a modified, user-oriented heuristic evaluation (HE) method, validated tasks and in-depth severity factor ratings to identify critical problems from patients’ point of view. By developing and employing a modified, user-centered cognitive walkthrough method (UC-CW), study IV assessed its effectiveness and efficiency in finding relevant usability problems for users as well as patients’ acceptance. The modified CW was validated against the golden-standard user test with Think Aloud.

Study I emphasized the importance of considering user characteristics in mHealth performance as these influenced interaction outcomes. All patients had difficulties with multiple-step tasks. Patients more recently diagnosed were able to perform tasks more successfully, with fewer errors and at faster times and had higher satisfaction scores; similar outcomes to the more experienced users. Educational level did not, however, seem to influence performance. In study II, the usability test with Think-Aloud (TA), in-depth interviews and questionnaires contributed to 19 consolidated issues, and triangulated on 5 critical usability problems for users. The combined analysis methods resulted in structured, categorized descriptions to aid in problem-solving. In Study III, the disease-related, critical information deficiencies found by expert evaluators using the modified, structured method also converged on and highlighted potentially adverse user concerns. Study IV demonstrated that the UC-CW found more critical user problems compared to the user test with TA despite both methods producing similar major average severity ratings and violations of heuristic categories. The modified method was more efficient per detected problem and experienced as less cognitively demanding and with a higher ease of use.

These studies offer different approaches that include patient-centered, efficient and user-acceptable methods and method modifications to detect critical usability issues for users. Importantly, improved mHealth designs for users could mean improvement in interactions, interaction performance, increased adoption, and long-term perhaps even increased adherence to interventions for chronic conditions.

Place, publisher, year, edition, pages
Karlskrona: Blekinge Tekniska Högskola, 2018. , p. 165
Series
Blekinge Institute of Technology Doctoral Dissertation Series, ISSN 1653-2090 ; 03
Keywords [en]
Chronic Disease, Diabetes, Mobile Health, mHealth, Self-management, Usability Evaluation, User-centered Design
National Category
Computer Sciences
Identifiers
URN: urn:nbn:se:bth-15140ISBN: 978-91-7295-349-9 (print)OAI: oai:DiVA.org:bth-15140DiVA, id: diva2:1178017
Public defence
2018-02-23, J1650, BTH, Valhallavägen 1, 37179, Karlskrona, 09:30 (Swedish)
Opponent
Supervisors
Note

Tillämpad Hälsoteknik

Applied Health Technology

Available from: 2018-01-29 Created: 2018-01-26 Last updated: 2018-04-06Bibliographically approved
List of papers
1. Quantifying usability: an evaluation of a diabetes mHealth system on effectiveness, efficiency, and satisfaction metrics with associated user characteristics
Open this publication in new window or tab >>Quantifying usability: an evaluation of a diabetes mHealth system on effectiveness, efficiency, and satisfaction metrics with associated user characteristics
2016 (English)In: JAMIA Journal of the American Medical Informatics Association, ISSN 1067-5027, E-ISSN 1527-974X, Vol. 23, no 1, p. 5-11Article in journal (Refereed) Published
Abstract [en]

Objective Mobile health (mHealth) systems are becoming more common for chronic disease management, but usability studies are still needed on patients' perspectives and mHealth interaction performance. This deficiency is addressed by our quantitative usability study of a mHealth diabetes system evaluating patients' task performance, satisfaction, and the relationship of these measures to user characteristics. Materials and Methods We used metrics in the International Organization for Standardization (ISO) 9241-11 standard. After standardized training, 10 patients performed representative tasks and were assessed on individual task success, errors, efficiency (time on task), satisfaction (System Usability Scale [SUS]) and user characteristics. Results Tasks of exporting and correcting values proved the most difficult, had the most errors, the lowest task success rates, and consumed the longest times on task. The average SUS satisfaction score was 80.5, indicating good but not excellent system usability. Data trends showed males were more successful in task completion, and younger participants had higher performance scores. Educational level did not influence performance, but a more recent diabetes diagnosis did. Patients with more experience in information technology (IT) also had higher performance rates. Discussion Difficult task performance indicated areas for redesign. Our methods can assist others in identifying areas in need of improvement. Data about user background and IT skills also showed how user characteristics influence performance and can provide future considerations for targeted mHealth designs. Conclusion Using the ISO 9241-11 usability standard, the SUS instrument for satisfaction and measuring user characteristics provided objective measures of patients' experienced usability. These could serve as an exemplar for standardized, quantitative methods for usability studies on mHealth systems.

Place, publisher, year, edition, pages
Oxford University Press, 2016
Keywords
diabetes; mobile health; self-management; usability evaluation; user-centered design
National Category
Endocrinology and Diabetes Other Medical Sciences not elsewhere specified
Identifiers
urn:nbn:se:bth-11378 (URN)10.1093/jamia/ocv099 (DOI)000374179500003 ()
Available from: 2016-01-08 Created: 2016-01-08 Last updated: 2018-01-26Bibliographically approved
2. An evaluation of patients' experienced usability of a diabetes mHealth system using a multi-method approach
Open this publication in new window or tab >>An evaluation of patients' experienced usability of a diabetes mHealth system using a multi-method approach
2016 (English)In: Journal of Biomedical Informatics, ISSN 1532-0464, E-ISSN 1532-0480, Vol. 59, p. 115-129Article in journal (Refereed) Published
Place, publisher, year, edition, pages
Elsevier, 2016
Keywords
Diabetes; Framework analysis; Multi-method evaluation; Usability Problem Taxonomy; mHealth
National Category
Endocrinology and Diabetes
Identifiers
urn:nbn:se:bth-11379 (URN)10.1016/j.jbi.2015.11.008 (DOI)000384703900009 ()
Note

open access

Available from: 2016-01-08 Created: 2016-01-08 Last updated: 2018-01-26Bibliographically approved
3. A Modified User-Oriented Heuristic Evaluation of a Mobile Health System for Diabetes Self-management Support
Open this publication in new window or tab >>A Modified User-Oriented Heuristic Evaluation of a Mobile Health System for Diabetes Self-management Support
2016 (English)In: Computers, Informatics, Nursing, ISSN 1538-2931, E-ISSN 1538-9774, Vol. 34, no 2, p. 77-84Article in journal (Refereed) Published
Abstract [en]

Mobile health platforms offer significant opportunities for improving diabetic self-care, but only if adequate usability exists. Expert evaluations such as heuristic evaluation can provide distinct usability information about systems. The purpose of this study was to complete a usability evaluation of a mobile health system for diabetes patients using a modified heuristic evaluation technique of (1) dual-domain experts (healthcare professionals, usability experts), (2) validated scenarios and user tasks related to patients' self-care, and (3) in-depth severity factor ratings. Experts identified 129 usability problems with 274 heuristic violations for the system. The categories Consistency and Standards dominated at 24.1% (n = 66), followed by Match Between System and Real World at 22.3% (n = 61). Average severity ratings across system views were 2.8 (of 4), with 9.3% (n = 12) rated as catastrophic and 53.5% (n = 69) as major. The large volume of violations with severe ratings indicated clear priorities for redesign. The modified heuristic approach allowed evaluators to identify unique and important issues, including ones related to self-management and patient safety. This article provides a template for one type of expert evaluation adding to the informaticists' toolbox when needing to conduct a fast, resource-efficient and user-oriented heuristic evaluation. 

Place, publisher, year, edition, pages
Lippincott Williams & Wilkins, 2016
Keywords
Diabetes; Heuristic evaluation; Mobile health; Patient self-management; Usability evaluation
National Category
Endocrinology and Diabetes
Identifiers
urn:nbn:se:bth-11380 (URN)10.1097/CIN.0000000000000209 (DOI)000376461500004 ()
Note

Open access

Available from: 2016-01-08 Created: 2016-01-08 Last updated: 2018-01-26Bibliographically approved
4. Using a User-centered Cognitive Walkthrough to Evaluate a mHealth Diabetes Self-management System Including a Case Study and External Validity Test
Open this publication in new window or tab >>Using a User-centered Cognitive Walkthrough to Evaluate a mHealth Diabetes Self-management System Including a Case Study and External Validity Test
(English)Manuscript (preprint) (Other academic)
Abstract [en]

Background: Self-management of chronic diseases with mHealth systems is becoming common. With the move toward patient-centered care, it is vital to incorporate patients in the development and evaluation of these systems. Current methods for usability evaluation such as formal usability testing can be very costly and time-consuming. Other methods may be more efficient but lack a user focus (e.g., Heuristic Evaluation (HE) and Cognitive Walkthrough (CW)). We propose a modified method to address identified deficiencies in the original CW technique and then assess its effectiveness efficiency and user acceptance. In this case study we use the modified, user-centered CW (UC-CW) method with diabetes patient users of a mHealth self-management system and validate it against the “golden-standard” user test with Think Aloud (TA) on the number, types and severity of usability problems, as well as the consumed time, and user experience of the cognitive load.

Materials and methods: A total of 12 diabetes patients were included: 6 in the UC-CW group evaluation session, and 6 in the individual user tests with TA. The setting for the assessments was the Diabetes and Endocrinology Center in Salt Lake City, Utah.  The modified UC-CW method consisted of: making the user the main evaluation contributor in finding the usability problems, a dual domain facilitator with the role of guiding the evaluation session and in rating found usability problems, a task development process resulting in validated tasks, a focus on higher level tasks in evaluation and in streamlining the evaluation in terms of time and resources. Users interacted with the mHealth application for both the modified method and user test with TA. Participants also filled in a pre-test questionnaire of demographic questions and one on their experience, knowledge and perception about information technology. Post-evaluation assessments included the NASA RTLX instrument and a set of brief interview questions about the different methods.

Results: Participants in both methods were a similar mix of males and females with similar types of diabetes. They were equally knowledgeable and experienced in using mobile phones. A total of 26 usability problems were found with the UC-CW and 20 problems with the TA. Participants in both methods gave the application similar severity ratings for all views (UC-CW= 2.7 and TA= 2.6) and a similarly high number of  problems in similar views (Main view [UC-CW=11, TA=10], Carbohydrate Entry view [UC-CW=4, TA= 3] and List view [UC-CW=3, TA=3] and with  similar heuristic violations (Match Between the System and Real World [UC-CW=19, TA=16], Consistency and Standards [UC-CW= 17, TA=15], and Recognition Rather than Recall [UC-CW=13, TA=10]). Both methods converged on 8 usability problems. When comparing the severity of the problems, the UC-CW detected 5 critical issues (severity level 4) while the TA detected two. The distinct issues in the UC-CW compared to the TA were personalized features required for patients’ individual disease needs. In terms of efficiency, the whole evaluation procedure took less time to perform for the TA than the UC-CW. However, when the number of usability problems is considered, the UC-CW was faster. The UC-CW was also faster for the evaluation process. The NASA RTLX scores indicated that participants experienced the UC-CW as half as demanding in terms of cognitive load. Common themes were that the UC-CW was perceived as easy to perform and enjoyable while the TA was considered somewhat awkward and more cognitively demanding.

Conclusions: The modified method proved useful for finding severe and recurring usability issues for users and highlighted the need for personalized user features. The UC-CW also proved efficient with a high user acceptance. These results indicate that the UC-CW can be a useful method to evaluate a mHealth self-management system for diabetes. The external validation that compared the method outcomes provided beginning evidence of the UC-CW’s utility as an effective, as well as efficient and acceptable method.

Keywords
Chronic Disease, Cognitive Walkthrough, Diabetes, Mobile Health, Self-management, Think Aloud, Usability, User Test, User-centered Design
National Category
Computer Sciences
Identifiers
urn:nbn:se:bth-15830 (URN)
Available from: 2018-01-26 Created: 2018-01-26 Last updated: 2018-01-29Bibliographically approved

Open Access in DiVA

fulltext(13217 kB)865 downloads
File information
File name FULLTEXT05.pdfFile size 13217 kBChecksum SHA-512
5efc699c39ea0d0e8cd1bee1738df52d6ff6be8a751abcbb480cd6ace515e9e28d2c9e4c049ad620ec0d0076eb99049be78ef87cf740dfae31fb2b4e32948be7
Type fulltextMimetype application/pdf

Authority records BETA

Georgsson, Mattias

Search in DiVA

By author/editor
Georgsson, Mattias
By organisation
Department of Creative Technologies
Computer Sciences

Search outside of DiVA

GoogleGoogle Scholar
Total: 959 downloads
The number of downloads is the sum of all downloads of full texts. It may include eg previous versions that are now no longer available

isbn
urn-nbn

Altmetric score

isbn
urn-nbn
Total: 9316 hits
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • harvard1
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf