Code Generation from Large API Specifications with Open Large Language Models: Increasing Relevance of Code Output in Initial Autonomic Code Generation from Large API Specifications with Open Large Language Models
2024 (English)Independent thesis Basic level (degree of Bachelor), 180 HE credits
Student thesis
Abstract [en]
Background. In software systems defined by extensive API specifications, auto- nomic code generation can streamline the coding process by replacing repetitive, manual tasks such as creating REST API endpoints. The use of large language models (LLMs) for generating source code comprehensively on the first try requires refined prompting strategies to ensure output relevancy, a challenge that grows as API specifications become larger.
Objectives. This study aims to develop and validate a prompting orchestration solution for LLMs that generates more relevant, non-duplicated code compared to a single comprehensive prompt, without refactoring previous code. Additionally, the study evaluates the practical value of the generated code for developers at Ericsson familiar with the target application that uses the same API specification.
Methods. Employing a prototyping approach, we develop a solution that produces more relevant, non-duplicated code compared to a single prompt with local-hosted LLMs for the target API at Ericsson. We perform a controlled experiment running the developed solution and a single prompt to collect the outputs. Using the results, we conduct interviews with Ericsson developers about the value of the AI-generated code.
Results. The study identified a prompting orchestration method that generated 427 relevant lines of code (LOC) on average in the best-case scenario compared to 66 LOC with a single comprehensive prompt. Additionally, 66% of the developers interviewed preferred using the AI-generated code as a starting point over starting from scratch when developing applications for Ericsson, and 66% preferred starting from the AI-generated code over code generated from the same API specification via Swagger CodeGen.
Conclusions. Increasing the extent locally hosted LLMs can generate relevant code from large API specifications without refactoring the generated code in comparison to a single comprehensive prompt is possible with the right prompting orchestration method. The value of the generated code is that it can currently be used as a good starting point for further software development.
Place, publisher, year, edition, pages
2024. , p. 37
Keywords [en]
Large Language Models, Autonomic Code Generation, API Specifica- tions, Artificial Intelligence, Prompt Engineering
National Category
Other Engineering and Technologies
Identifiers
URN: urn:nbn:se:bth-26582OAI: oai:DiVA.org:bth-26582DiVA, id: diva2:1877570
External cooperation
Ericsson
Subject / course
PA1445 Kandidatkurs i Programvaruteknik; PA1445 Kandidatkurs i Programvaruteknik
Educational program
PAGPT Software Engineering; PAGWE Web Programming
Supervisors
Examiners
2024-07-012024-06-252025-02-10Bibliographically approved