Change search
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf
Exploring the Use of Large Language Model Assistants in Web GUI Test Automation: A Practitioner Study
Blekinge Institute of Technology, Faculty of Computing, Department of Software Engineering.
Blekinge Institute of Technology, Faculty of Computing, Department of Software Engineering.
2026 (English)Independent thesis Advanced level (degree of Master (Two Years)), 20 credits / 30 HE creditsStudent thesis
Abstract [en]

Context. Web GUI test automation couples test scripts to an interface that keeps changing, which makes the scripts expensive to keep working. Large language model (LLM) assistants are now used routinely to write and repair such scripts, but the published evaluations of assistant-generated tests report coverage and execution correctness, not what the generated scripts cost to maintain, how they withstand interface change, or whether they behave consistently. Practitioner evidence on those questions is scarce.

Objectives. This study establishes how practitioners actually use LLM assistants in web GUI test automation, which benefits and challenges they perceive, and how they perceive the effect of generated scripts on robustness, maintainability and flakiness.

Methods. A mixed-methods practitioner study. A questionnaire returned 60 responses, of which 49 form the analysis set, and 7 semi-structured interviews were recorded and transcribed, then analysed by reflexive thematic analysis to yield 9 themes over 213 coded segments. The sample is non-probability, so the quantitative analysis is descriptive throughout: medians, modes and full frequency distributions, with no means on single Likert items, no significance tests and no generalisation of any proportion to a population.

Results. Assistants are used as drafting tools held under supervision: 3 of 49 respondents use generated output unmodified, while 88% use an assistant weekly or more often. Every task in the list is selected by a majority, so the ordering within that narrow band is suggestive rather than measured: the most reported is writing documentation and the least is creating element locators, which inverts the emphasis of the element localisation literature. Reduced time to write scripts is the clearest benefit, at a median of 4. The three highest rated challenges concern the practitioner’s relationship to the output rather than the output itself: over-reliance, difficulty verifying, and the assistant lacking context about the application under test. Flakiness is the only attribute whose modal answer is no noticeable impact, at a median of 3 with 21 of 49 respondents selecting it and the responses either side leaning favourable rather than balanced. Among the interviewees asked about flaky tests this is a reattribution of cause to framework design and review discipline, not inattention, and two of them do attribute flakiness to the assistant directly.

Conclusions. Practitioners supervise these tools closely and locate output quality in the context they themselves supply. The flakiness finding is a considered judgement that agrees with where the flakiness literature places element selection among root causes. Maintainability is the attribute nobody here can measure. It is not measured for a generated suite in the evaluations we cite, and the practitioners doing the work say they cannot judge it either, which makes measuring it the most valuable work that follows from this study.

Place, publisher, year, edition, pages
2026. , p. 68
Keywords [en]
Software Testing, Web GUI, Large Language Models, Test Automation, Graphical User Interface, Practitioner Study, Thematic Analysis, Flakiness, Robustness, Maintainability
National Category
Software Engineering
Identifiers
URN: urn:nbn:se:bth-30635OAI: oai:DiVA.org:bth-30635DiVA, id: diva2:2113323
Subject / course
PA2534 Master's Thesis (120 credits) in Software Engineering; PA2534 Master's Thesis (120 credits) in Software Engineering
Educational program
PAASW Master's Programme in Software Engineering 120,0 hp; PAADA Master Qualification Plan in Software Engineering 120,0 hp
Presentation
2026-09-22, J1650, Blekinge Institute of Technology, Valhallavägen 1, 371 79, Karlskrona, 13:00 (English)
Supervisors
Examiners
Available from: 2026-11-02 Created: 2026-10-27 Last updated: 2026-11-02Bibliographically approved

Open Access in DiVA

Exploring the Use of Large Language Model Assistants in Web GUI Test Automation(734 kB)2 downloads
File information
File name FULLTEXT01.pdfFile size 734 kBChecksum SHA-512
ec4a1860df15ff84b622a5ceaeeaa472ff4a193db2758c96ec612e8d0318eaf689b7bfc960daf4803b961d3db040f406e9ee2fb2e15c1d6f285cc9207b09e8b8
Type fulltextMimetype application/pdf

By organisation
Department of Software Engineering
Software Engineering

Search outside of DiVA

GoogleGoogle Scholar
Total: 2 downloads
The number of downloads is the sum of all downloads of full texts. It may include eg previous versions that are now no longer available

urn-nbn

Altmetric score

urn-nbn
Total: 22 hits
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf