Background. CI/CD is widely adopted by agile organizations to support rapid iteration and high-quality releases through automation. However, integrating third-party binaries into CI/CD pipelines presents challenges, particularly when source code is not available. This limited visibility complicates verification and integration by restricting the ability to trace functionality, assess code quality, and identify potential risks. This thesis explores how confidence in the software system is influenced when integrating such closed-source third-party components into a larger software system.

Objectives. This research aim to identify key factors that build, indicate, and enhance confidence in integrating third-party binaries into a complex software system. It also provides recommendations to help organizations adapt CI/CD pipelines for reliable and continuous integration.

Methods. This research starts with a literature review to prepare for the case study. The case study involved data collection from document reviews, direct observation, and semi-structured interviews. Insights from document reviews and direct observation informed the design of the semi-structured interviews. A triangulation analysis was conducted to ensure consistency between interview data and findings from document reviews and observations. Finally, the research question was addressed through interview analysis, and the results were validated by interviewees via a questionnaire.

Results. Interview data highlighted acceptance testing, testing with production-representative hardware, and verifying supplier test reports as key factors for ensuring confidence in third-party binary quality. From a process perspective, fast automated supplier feedback and monitoring CI/CD pipeline metrics ranked highest. Integrating these factors, the study formulated eight key recommendations to help organizations manage third-party binaries effectively.

Conclusions. The recommendations focus on three key areas: Testing, Responsibility, and Monitoring & Visualization. This research advocates for systematic automated acceptance testing, which also serves as regression testing. It emphasizes the need for organizations to actively verify and validate third-party binaries beyond supplier validation. Additionally, it underscores the importance of enhancing monitoring and visualization to support data-driven decision-making, ensuring accurate assessment of confidence levels and integration readiness.