Background: Web application security (WAS) testing is crucial for protecting web applications from cyber threats. However, organizations often struggle to select effective WAS testing tools due to the lack of a well-defined set of evaluation criteria. This research aims to address this need by identifying the key metrics for evaluating and comparing WAS testing tools. 

Objectives: The primary objectives of this research are to identify the key metrics for comparing WAS testing tools, validate the significance of these metrics through semi-structured interviews, and perform a comparison between WAS testing tools using the validated metrics. This research aims to find a set of validated metrics for evaluating and comparing WAS testing tools. 

Methods: The research methodology consisted of three main phases: a literature review to compile a comprehensive set of technical and non-technical metrics commonly used for assessing and comparing WAS testing tools, semi-structured interviews with security experts to validate the significance of the identified metrics, and an experiment to compare three WAS testing tools - ZAP, Burp Suite, and Acunetix - using the OWASP Benchmark project. These three tools were selected based on the author’s recommendations in the literature. 

Results: The initial literature review found 37 evaluation metrics for WAS testing tools. Through interviews, experts confirmed some of these were important, but also said some were not very useful. The experts additionally suggested some new metrics that were not in the literature. Incorporating this feedback, the final list was refined down to 35 metrics for evaluating WAS testing tools. An experiment was then conducted to compare three WAS testing tools - ZAP, Burp Suite, and Acunetix with the test subject as the OWASP Benchmark Project and by using the validated set of metrics. The results of this experiment revealed differences in the performance of the tools, with Burp Suite emerging as the best performer. 

Conclusions: This research has provided a valid set of metrics for comparing and evaluating WAS testing tools, empowering organizations to make more informed decisions. Security professionals can optimise their WAS testing tool selection by understanding the key metrics and their relative significance, as established through the literature and interviews. Based on the experimental analysis, Burp Suite performed better than other tools. Therefore, for organizations initiating the selection process of the WAS testing tool, Burp Suite stands out as a good choice.