LLM-Based Labelling of Recorded Automated GUI-Based Test Cases
2025 (English)In: 2025 IEEE Conference on Software Testing, Verification and Validation, ICST 2025 / [ed] Fasolino A.R., Panichella S., Aleti A., Mesbah A., Institute of Electrical and Electronics Engineers (IEEE), 2025, p. 453-463Conference paper, Published paper (Refereed)
Abstract [en]
Graphical User Interface (GUI) based testing is a commonly used practice in industry. Although valuable and, in many cases, necessary, it is associated with challenges such as high cost and requirements on both technical and domain expertise. Augmented testing, a novel approach to GUI test automation, aims to mitigate these challenges by allowing users to record and render test cases and test data directly on the GUI of the system under test (SUT). In this context, Scout is an augmented testing tool that captures system states and transitions during manual interaction with the SUT, storing them in a test model that is visually represented in the form of state trees and reports. While this representation provides basic overview of a test suite, e.g. about its size and number of scenarios, it is limited in terms of analysis depth, interpretability, and reproducibility. In particular, without human state labeling, it is challenging to produce meaningful and easily understandable test reports. To address this limitation, we present a novel solution and a demonstrator, integrated into Scout, which leverages large language models (LLMs) to enrich the model-based test case representation by automatically labeling and describing states and describing transitions. We conducted two experiments to evaluate the impact of the solution. First, we compared LLM-enhanced reports with expert-generated reports using embedding distance evaluation metrics. Second, we assessed the usability and perceived value of the enhanced reports through an industrial survey. The results of the study indicate that the plugin can improve readability, actionability, and interpretability of test reports. This work contributes to the automation of GUI testing by reducing the need for manual intervention, e.g. labeling, and technical expertise, e.g. to understand test case models. Although the solution is studied in the context of augmented testing, we argue for the solution's generalizability to related test automation techniques. In addition, we argue that this approach enables actionable insights and lays the groundwork for further research into autonomous testing based on Generative AI.
Place, publisher, year, edition, pages
Institute of Electrical and Electronics Engineers (IEEE), 2025. p. 453-463
Keywords [en]
High costs, Interpretability, Labelings, Language model, Model-based OPC, Systems under tests, Technical expertise, Test Automation, Test case, Test reports, Graphical user interfaces
National Category
Software Engineering
Identifiers
URN: urn:nbn:se:bth-28173DOI: 10.1109/ICST62969.2025.10988984ISI: 001506893900040Scopus ID: 2-s2.0-105007522870ISBN: 9798331508142 (print)OAI: oai:DiVA.org:bth-28173DiVA, id: diva2:1974502
Conference
18th IEEE Conference on Software Testing, Verification and Validation, ICST 2025, Naples, April 31-4, 2025
Part of project
SERT- Software Engineering ReThought, Knowledge FoundationT.A.R.G.E.T. – Testing with AI Reinforced GUI Embedded Technology, Vinnova
Funder
Knowledge Foundation, 20180010Vinnova, 2024-002422025-06-232025-06-232025-09-30Bibliographically approved