Change search
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf
Towards Platform Agnostic Automated GUI Based Testing: Evaluating Computer Vision Through SCOUT
Blekinge Institute of Technology, Faculty of Computing.
2026 (English)Independent thesis Advanced level (professional degree), 20 credits / 30 HE creditsStudent thesis
Abstract [en]

This thesis addresses the persistent challenges of automated GUI-based testing, a critical component of system testing that verifies software integration across diverse platforms.Despite the prevalence of graphical user interfaces, the reliability of automated testing is often limited by the fragility of test scripts and the difficulty of robustly identifying GUI widgets.While LLMs offer promising capabilities, their practical application is currently hindered by high latency and concerns about generalisability.Building upon SCOUT, a scriptless GUI test automation framework developed within the Vinnova-funded T.A.R.G.E.T. project, this work explores the use of CV as a practical locator strategy to enable more platform-agnostic testing.The research evaluates the robustness and flexibility of state-of-the-art models, specifically YOLO and RT-DETR, in localising GUI elements across diverse web interfaces.The results demonstrate that while CV-based localisation achieves high precision in constrained and domain-specific settings, it is not yet a viable standalone replacement for DOM-based locators due to limitations in generalisation and the loss of structural information.However, the findings suggest that CV can serve as a powerful complementary strategy, combining visual precision with structural reliability to reduce test fragility.This work provides a foundation for developing hybrid localisation frameworks that mitigate the reliance on manual testing and improve the scalability of automated GUI testing.

Place, publisher, year, edition, pages
2026. , p. 66
Keywords [en]
Computer Vision, GUI Element Localisation, Widget Detection, Human-computer interaction, Platform-Agnostic Testing
National Category
Computer Systems
Identifiers
URN: urn:nbn:se:bth-30562OAI: oai:DiVA.org:bth-30562DiVA, id: diva2:2107635
Subject / course
Degree Project in Master of Science in Engineering 30,0 hp
Educational program
DVAMI Master of Science in Engineering: AI and Machine Learning 300 hp
Presentation
2026-09-25, J1650, Sökvägen 10, Karlskrona, 09:00 (English)
Supervisors
Examiners
Available from: 2026-10-22 Created: 2026-10-05 Last updated: 2026-10-22Bibliographically approved

Open Access in DiVA

fulltext(1753 kB)2 downloads
File information
File name FULLTEXT02.pdfFile size 1753 kBChecksum SHA-512
e6cfc56e605228d2028a6e457c6519c7f51b7e926deeb656c8fd8dc2b7d446dd7a71d339fa7790abbb0a183db1f0b8ddff8225e023a70e5909e6a51118005dcb
Type fulltextMimetype application/pdf

By organisation
Faculty of Computing
Computer Systems

Search outside of DiVA

GoogleGoogle Scholar
Total: 2 downloads
The number of downloads is the sum of all downloads of full texts. It may include eg previous versions that are now no longer available

urn-nbn

Altmetric score

urn-nbn
Total: 4 hits
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf