Context. Software testing is becoming more and more important in software development life-cycle especially for web testing. Selenium is one of the most widely used property-based Graph-User-Interface(GUI) web testing tools. Nevertheless, it also has some limitations. For instance, Selenium cannot test the web components in some specific plugins or HTML5 videos frame. But it is important for testers to verify the functionality of plugins or videos on the websites. Recently, the theory of the image recognition-based GUI testing is introduced which can locate and interact with the components to be tested on the websites by image recognition. There are only a few papers do research on comparing property-based GUI web testing and image recognition-based GUI testing. Hence, we formulated our research objectives based on this main gap.

Objectives. We want to compare these two different techniques with EyeSel which is the tool represents the image recognition-based GUI testing and Selenium which is the tool represents the property-based GUI testing. We will evaluate and compare the strengths and drawbacks of these two tools by formulating specific JUnit testing scripts. Besides, we will analyze the comparative result and then evaluate if EyeSel can solve some of the limitations associated with Selenium. Therefore, we can conclude the benefits and drawbacks of property-based GUI web testing and image recognition-based GUI testing.  

Methods. We conduct an experiment to develop test cases based on websites’ components both by Selenium and EyeSel. The experiment is conducted in an educational environment and we select 50 diverse websites as the subjects of the experiment. The test scripts are written in JAVA and ran by Eclipse.  The experiment data is collected for comparing and analyzing these two tools.

Results. We use quantitative analysis and qualitative analysis to analyze our results. First of all, we use quantitative analysis to evaluate the effectiveness and efficiency of two GUI web testing tools. The effectiveness is measured by the number of components that can be tested by these two tools while the efficiency is measured by the measurements of test cases’ development time and execution time. The results are as follows (1) EyeSel can test more number of components in web testing than Selenium (2) Testers need more time to develop test cases by Selenium than by EyeSel (3) Selenium executes the test cases faster than EyeSel. (4) “Results (1)” indicates the effectiveness of EyeSel is better than Selenium while “Results (2)(3)” indicate the efficiency of EyeSel is better than Selenium. Secondly, we use qualitative analysis to evaluate four quality characteristics (learnability, robustness, portability, functionality) of two GUI web testing tools. The results show that portability and functionality of Selenium are better than EyeSel while the learnability of EyeSel is better than Selenium. And both of them have good robustness in web testing.

Conclusions. After analyzing the results of comparison between Selenium and EyeSel, we conclude that (1) Image recognition-based GUI testing is more effectiveness than property-based GUI web testing (2) Image recognition-based GUI testing is more efficiency than property-based GUI web testing (3) The portability and functionality of property-based GUI web testing is better than Image recognition-based GUI testing (4) The learnability of image recognition-based GUI testing is better than property-based GUI web testing. (5) Both of them are good at different aspects of robustness