Software Configuration Change Impact Analysis without Analysable Bindings
2026 (English)Independent thesis Advanced level (degree of Master (Two Years)), 20 credits / 30 HE credits
Student thesis
Abstract [en]
Background: Change impact analysis identifies which artefacts may be affected by a change. Many established techniques need a link between a configuration value and program state that can be analysed or observed. That link is missing when configuration comes from an externally versioned interface schema, which evolves separately from the implementation while the dependent tests reach the system through a management interface. Engineers fall back on lexical search, whose effectiveness for such changes has not been measured.
Objectives: This thesis identifies the preconditions that established techniques require, checks which hold for schema-level configuration changes in an industrial setting, and compares a lexical candidate-finding rule with an LLM-based candidate classifier on a common benchmark.
Methods: A structured literature review built a framework of technique preconditions and checked it against the 315 records assessed at full text. An embedded single-case study combined schema release documents, test repository analysis and five interviews. A benchmark of 11 real schema changes and 42 authored test scenarios evaluated both conditions against an expert-reviewed reference standard. Initial labels came from a domain engineer and were reviewed by two further domain experts where the required source material was available.
Results: Five of the seven preconditions do not hold in the studied setting. Lexical search returned 30 positives and the classifier returned 15, and all 15 were inside the 30. The two conditions disagreed on 15 scenarios, and every disagreement was a rejection by the classifier. Eleven rejections were correct and four were wrong. Every positive classifier decision was already contained within the lexical positives, so the classifier did not extend the observed positive set. Scoring the eight scenarios it could not process as negative, lexical search reached precision 0.47 and recall 0.70 against 0.67 and 0.50 for the classifier. These values describe this benchmark rather than lexical search in general: the negative scenarios are authored near misses that contain the changed identifier, and lexical search returns any scenario naming it, so its precision follows from how the benchmark was designed.
Conclusions: The classifier removed false positives but missed more affected tests. A pipeline that filters lexical results through it would inherit the lower recall rather than keep the recall of the search. It recovered neither of the two affected scenarios in which the changed element is never named in the test source. Only four unnamed scenarios, from a single case, were classified at all, so this observation is suggestive rather than established; within it, a vocabulary difference and a missing reference behave as different problems. The findings apply to one model configuration, one prompt and one benchmark. The retrieval stage of the approach was not exercised, and a non-generative condition was specified but not evaluated.
Place, publisher, year, edition, pages
2026. , p. 87
Keywords [en]
change impact analysis, software traceability, configuration management, regression test selection, large language models
National Category
Software Engineering
Identifiers
URN: urn:nbn:se:bth-30634OAI: oai:DiVA.org:bth-30634DiVA, id: diva2:2113305
External cooperation
Ericsson AB
Subject / course
PA2534 Master's Thesis (120 credits) in Software Engineering
Educational program
PAADA Master Qualification Plan in Software Engineering 120,0 hp
Presentation
2026-09-22, 09:00
Supervisors
Examiners
2026-11-022026-10-272026-11-02Bibliographically approved