Background: The integration of test automation within CAD/PDM environments, such as Scania’s use of Catia and 3DEXPERIENCE, presents unique challenges due to the systems’ complexity and specialized nature. While test automation has the potential to improve testing efficiency and product quality, choosing the right tools and strategies is critical to a successful implementation.

Objectives: The purpose of this study is to identify the key benchmarking crite-ria for evaluating test automation tools, to determine the most appropriate toolsfor automating tests in Scania’s CAD/PDM areas, and to assess the benefits andchallenges of incorporating test automation in this setting.

Methods: A literature review was conducted using a snowballing technique to identify key metrics and KPIs for benchmarking test automation tools. Proof of Concepts (POCs) were carried out using tools such as Tricentis Tosca, UTFOne, Playwright, and RoboCorp, and their performance was assessed against identified criteria. Furthermore, semi-structured interviews with key Scania employees were analyzed thematically to gain insights into the perceived benefits and challenges of implementing test automation.

Results: In this study, several KPIs and metrics, such as execution time, report generation, ease of integration with CI/CD pipelines, learning curve, and implemen-tation cost, were identified and used to evaluate the suitability of four automated testing tools (Tricentis Tosca, UFT One, Playwright, and RoboCorp) for Scania’s CAD/PDM environment. The Proofs of Concept showed that Tricentis Tosca and UTFOne were the best tools for Scania’s CAD/PDM systems, offering strong automation support for complex Catia and 3DEXPERIENCE workflows. The interviews revealed potential benefits such as increased efficiency and faster software releases, but also challenges such as integration complexity, high setup costs, and the learning curve associated with new tools.

Conclusions: Integrating test automation into Scania’s CAD/PDM environment is feasible and provides numerous advantages, including increased testing efficiency and product quality. This research offers an approach for integrating test automation into the DevOps workflow of large software development companies. With regard to quality assurance, efficiency, and alignment with continuous integration and deployment strategies, the study aims to provide a deeper understanding of how test automation can be incorporated into the software development life-cycle.