Towards Platform Agnostic Automated GUI Based Testing: Evaluating Computer Vision Through SCOUT
2026 (English)Independent thesis Advanced level (professional degree), 20 credits / 30 HE credits
Student thesis
Abstract [en]
This thesis addresses the persistent challenges of automated GUI-based testing, a critical component of system testing that verifies software integration across diverse platforms.Despite the prevalence of graphical user interfaces, the reliability of automated testing is often limited by the fragility of test scripts and the difficulty of robustly identifying GUI widgets.While LLMs offer promising capabilities, their practical application is currently hindered by high latency and concerns about generalisability.Building upon SCOUT, a scriptless GUI test automation framework developed within the Vinnova-funded T.A.R.G.E.T. project, this work explores the use of CV as a practical locator strategy to enable more platform-agnostic testing.The research evaluates the robustness and flexibility of state-of-the-art models, specifically YOLO and RT-DETR, in localising GUI elements across diverse web interfaces.The results demonstrate that while CV-based localisation achieves high precision in constrained and domain-specific settings, it is not yet a viable standalone replacement for DOM-based locators due to limitations in generalisation and the loss of structural information.However, the findings suggest that CV can serve as a powerful complementary strategy, combining visual precision with structural reliability to reduce test fragility.This work provides a foundation for developing hybrid localisation frameworks that mitigate the reliance on manual testing and improve the scalability of automated GUI testing.
Place, publisher, year, edition, pages
2026. , p. 66
Keywords [en]
Computer Vision, GUI Element Localisation, Widget Detection, Human-computer interaction, Platform-Agnostic Testing
National Category
Computer Systems
Identifiers
URN: urn:nbn:se:bth-30562OAI: oai:DiVA.org:bth-30562DiVA, id: diva2:2107635
Subject / course
Degree Project in Master of Science in Engineering 30,0 hp
Educational program
DVAMI Master of Science in Engineering: AI and Machine Learning 300 hp
Presentation
2026-09-25, J1650, Sökvägen 10, Karlskrona, 09:00 (English)
Supervisors
Examiners
2026-10-222026-10-052026-10-22Bibliographically approved