Background. The increasing complexity of software applications necessitates automated testing to meet time-to-market demands while ensuring quality. Traditional methods like manual testing and scripted automation may struggle to provide continuous testing due to time and budget constraints. Thus, this study aims to analyze the testing efficiency of AT with SCOUT compared to SOP tools like Selenium IDE, striving to validate its credibility and acceptance in the software testing domain.

Objectives. The objectives of this study are to compare the effectiveness between AT with SCOUT and State-of-practice (SOP) test tool in software testing through a quasi-experiment, conduct mutation analysis to objectively assess tool performance, and ensure participant familiarity with testing method.

Methods. The Quasi-experiment was conducted using an open source web application among 10 participants to write the test cases using the Scout and Selenium IDE. In order to evaluate this plugin a quasi-experiment was conducted where the effectiveness, time efficiency in test development and test execution was analyzed. The experiment was conducted against an web application which the authors implemented faults to, also known as mutants. Tool performance when attempting to f ind these faults were documented as the results. These results were analyzed by descriptive statistics using histograms and tables in conjunction with more formal statistics using the Kruskal-Wallis H-test.

Results: The findings reveal Scout’s superior fault-finding capabilities compared to Selenium IDE. Additionally, Scout demonstrates greater efficiency in test development time, compared to Selenium IDE. However, the reverse is observed regarding in test execution efficiency, with Selenium IDE showing faster execution times.

Conclusions. In our study, Scout demonstrates better effectiveness in test case development and bug detection, while Selenium IDE exhibits faster execution times. Scout’s higher true positive rate in mutation analysis underscores its robustness in bug detection, positioning it as a good option for comprehensive web application testing despite participants’ varying perceptions.