Planned maintenance
A system upgrade is planned for 17/11-2026, at 12:00-13:00. During this time DiVA will be unavailable.
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf
LLM-driven test and test-repair automation for legacy Perl systems: an empirical study at IBM
Helmut Schmidt University Hamburg, Germany.ORCID iD: 0009-0005-1739-6069
Blekinge Institute of Technology, Faculty of Computing, Department of Software Engineering.ORCID iD: 0000-0002-1532-8223
2026 (English)In: Software quality journal, ISSN 0963-9314, E-ISSN 1573-1367, Vol. 34, no 4, article id 59Article in journal (Refereed) Published
Abstract [en]

Automated unit test generation using Large Language Models (LLMs) has shown promise for widely used programming languages such as Java and Python. It may further benefit legacy systems lacking a test base, and the investigation of less widespread legacy languages remains relevant as many companies still operate systems built on them. We develop a modular workflow combining static code analysis, multi-stage prompt engineering with business logic extraction, automated error analysis, and iterative test-fix cycles for legacy systems, evaluated by pass rates, coverage, and repair effectiveness. We evaluate our approach on four productive IBM Perl repositories (186 functions) and three CPAN distributions (139 functions) using two open-source LLMs: Llama-3.3-70B-Instruct and Mistral-Small-3.1-24B-Instruct-2503. A key methodological contribution is a defensive eval wrapper that records a crash in the code under test as a failed assertion rather than terminating the run. A git-diff mode enables selective test generation for changed functions, supporting incremental maintenance workflows. The larger model achieves initial pass rates of 66-82% (industrial) and 74-81% (CPAN), improving to 74-86% and 76-89% after repair, with statement coverage of 34-92% and 65-100%. The smaller model performs substantially worse on both sets. Repair reduces structural errors on both (no_output from 0-44% to 0-6% and 0-7% to 0-5%) and improves pass rates for both models in most repositories. We show the transferability of LLM-based test generation to low-resource legacy languages, establishing Perl as a viable target for automated testing pipelines.

Place, publisher, year, edition, pages
Springer, 2026. Vol. 34, no 4, article id 59
Keywords [en]
Large language models, Automated test generation, Perl, Legacy code, Software testing, CI/CD integration
National Category
Computer Sciences
Identifiers
URN: urn:nbn:se:bth-30708DOI: 10.1007/s11219-026-09790-3ISI: 001908531300001Scopus ID: 2-s2.0-105054164101OAI: oai:DiVA.org:bth-30708DiVA, id: diva2:2114520
Available from: 2026-11-02 Created: 2026-11-02 Last updated: 2026-11-09Bibliographically approved

Open Access in DiVA

fulltext(2950 kB)74 downloads
File information
File name FULLTEXT01.pdfFile size 2950 kBChecksum SHA-512
51726ef93df3a8e62941e3f6c6a928cf043d7254c7eebb0bcf111dc73f59b253f7141bb27bdd85facb1ccc4a019c11efd880c6058afdc1849839451954acdcd8
Type fulltextMimetype application/pdf

Other links

Publisher's full textScopus

Authority records

Petersen, Kai

Search in DiVA

By author/editor
Leunig, SimonPetersen, Kai
By organisation
Department of Software Engineering
In the same journal
Software quality journal
Computer Sciences

Search outside of DiVA

GoogleGoogle Scholar
Total: 74 downloads
The number of downloads is the sum of all downloads of full texts. It may include eg previous versions that are now no longer available

doi
urn-nbn

Altmetric score

doi
urn-nbn
Total: 13918 hits
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf