Exploring the Use of Large Language Model Assistants in Web GUI Test Automation: A Practitioner Study
2026 (English)Independent thesis Advanced level (degree of Master (Two Years)), 20 credits / 30 HE credits
Student thesis
Abstract [en]
Context. Web GUI test automation couples test scripts to an interface that keeps changing, which makes the scripts expensive to keep working. Large language model (LLM) assistants are now used routinely to write and repair such scripts, but the published evaluations of assistant-generated tests report coverage and execution correctness, not what the generated scripts cost to maintain, how they withstand interface change, or whether they behave consistently. Practitioner evidence on those questions is scarce.
Objectives. This study establishes how practitioners actually use LLM assistants in web GUI test automation, which benefits and challenges they perceive, and how they perceive the effect of generated scripts on robustness, maintainability and flakiness.
Methods. A mixed-methods practitioner study. A questionnaire returned 60 responses, of which 49 form the analysis set, and 7 semi-structured interviews were recorded and transcribed, then analysed by reflexive thematic analysis to yield 9 themes over 213 coded segments. The sample is non-probability, so the quantitative analysis is descriptive throughout: medians, modes and full frequency distributions, with no means on single Likert items, no significance tests and no generalisation of any proportion to a population.
Results. Assistants are used as drafting tools held under supervision: 3 of 49 respondents use generated output unmodified, while 88% use an assistant weekly or more often. Every task in the list is selected by a majority, so the ordering within that narrow band is suggestive rather than measured: the most reported is writing documentation and the least is creating element locators, which inverts the emphasis of the element localisation literature. Reduced time to write scripts is the clearest benefit, at a median of 4. The three highest rated challenges concern the practitioner’s relationship to the output rather than the output itself: over-reliance, difficulty verifying, and the assistant lacking context about the application under test. Flakiness is the only attribute whose modal answer is no noticeable impact, at a median of 3 with 21 of 49 respondents selecting it and the responses either side leaning favourable rather than balanced. Among the interviewees asked about flaky tests this is a reattribution of cause to framework design and review discipline, not inattention, and two of them do attribute flakiness to the assistant directly.
Conclusions. Practitioners supervise these tools closely and locate output quality in the context they themselves supply. The flakiness finding is a considered judgement that agrees with where the flakiness literature places element selection among root causes. Maintainability is the attribute nobody here can measure. It is not measured for a generated suite in the evaluations we cite, and the practitioners doing the work say they cannot judge it either, which makes measuring it the most valuable work that follows from this study.
Place, publisher, year, edition, pages
2026. , p. 68
Keywords [en]
Software Testing, Web GUI, Large Language Models, Test Automation, Graphical User Interface, Practitioner Study, Thematic Analysis, Flakiness, Robustness, Maintainability
National Category
Software Engineering
Identifiers
URN: urn:nbn:se:bth-30635OAI: oai:DiVA.org:bth-30635DiVA, id: diva2:2113323
Subject / course
PA2534 Master's Thesis (120 credits) in Software Engineering; PA2534 Master's Thesis (120 credits) in Software Engineering
Educational program
PAASW Master's Programme in Software Engineering 120,0 hp; PAADA Master Qualification Plan in Software Engineering 120,0 hp
Presentation
2026-09-22, J1650, Blekinge Institute of Technology, Valhallavägen 1, 371 79, Karlskrona, 13:00 (English)
Supervisors
Examiners
2026-11-022026-10-272026-11-02Bibliographically approved