Background. With the development and popularization of mobile Internet, smartphones are becoming more and more popular. Android is one of the most popular platforms of smartphones.  And application is one of the most important part of a smartphone. There are a lot of money and human resources spent on Android application development every year. And quiet a big part of them goes to quality assurance of applications. Graphic user interface (GUI) testing is one important part of its quality assurance. Android phones use touch screen as the major I/O method. Therefore, GUI testing on android platform shall be different to conventional software applications that are designed to run on desktop environment.

Objectives. The aim of this research is to assess the performance of two GUI testing approaches (2nd vs 3rd generation) of automated UI testing in terms of testing Android applications. By assessing these approaches, we could hopefully get insights of their advantages and limitations for using them in the context of Android development. And this aim can be divided into three objectives, to compare the time spent on implementing test cases of each tool, to compare the time costed when executing test cases of each tool, to compare the number of defects found by each tool.

Methods. The research methodology we chose is controlled experiment. We have chosen UI Automator and Appium to represent 2nd generation GUI testing approach, EyeAutomate and SikuliX to represent 3rd generation GUI testing approach. We used each tool to implement and execute 120 test cases to compare them on the time spent on implementing test cases of each tool, the time costed when executing test cases of each tool, the number of real defects found by each tool, and the number of false positives found by each tool.

Results. Tools using 3rd generation GUI testing approach take less time to implement test cases than tools using 2nd generation GUI testing approach. And there is no specific pattern when comparing tools using 2nd and 3rd generation GUI testing approaches in terms of time cost on executing test cases. It is different between different test cases. Besides false positive alerts appear at a much higher frequency in tools using 3rd generation GUI testing approach than tools using 2nd generation GUI testing approach. While, real defects found by each tool are the same.

Conclusions. 3rd generation GUI testing approach is more efficient in terms of implementing test cases than 2nd generation GUI testing approach. But 3rd generation GUI testing approach finds much more false positives than 2nd generation approach. To decide if a defect alert is false positive or not requires human effort. In a long term, it may accumulate huge lost on human efforts. Therefore, to maintain test cases, 3rd generation approach consumes lots of human efforts.