Background: Generative AI has achieved rapid and widespread acclaim over a short period since the inception of recent models that have opened up opportunities not possible before. Large Language Models (LLMs), a subset of generative AI, have become an essential part of code generation for software development. However, there is always a risk that the generated code does not fulfill the programmer's intent and contains faults or bugs that can go unnoticed. To that end, we propose that verification of generated code should increase its quality and trust.

Objectives: This thesis aims to research generation of code that is both functionally correct and verifiable by implementing and evaluating four prompting approaches and a reinforcement learning solution to increase robustness within code generation, using unit-test and verification rewards.

Methods: We used a Rapid Literature Review (RLR) and Design Science methodology to get a solid overview of the current state of robust code generation. From the RLR and related works, we evaluated the following four prompting approaches: Base prompt, Documentation prompting, In-context learning, and Documentation + In-context learning on the two datasets: MBPP and HumanEval. Moreover, we fine-tuned one model using Proximal Policy Optimization (PPO) for the novel task.

Results: We measured the functional correctness and static verification success rates, amongst other metrics, for the four proposed approaches on eight model configurations, including the PPO fine-tuned LLM. Our results show that for the MBPP dataset, on average, In-context learning had the highest functional correctness at 29.4% pass@1, Documentation prompting had the highest verifiability at 8.48% verfiable@1, and finally, In-context learning had the highest functionally correct verifiable code at 3.2% pass@1 & verifiable@1. Moreover, the PPO fine-tuned model showed an overall increase in performance across all approaches compared to the pre-trained base model.

Conclusions: We found that In-context learning on the PPO fine-tuned model yielded the best overall results across most metrics compared to the other approaches. The PPO fine-tuned with In-context learning resulted in 32.0% pass@1, 12.8% verifiable@1, and 5.0% pass@1 & verifiable@1. Documentation prompting was better for verifable@1 on MBPP. However, it did not perform as well for the other metrics. Documentation prompting + In-context learning was performance-wise between Documentation prompting and In-context learning, while Base prompt performed the worst overall. For future work, we envision several improvements to PPO training, including but not limited to training on Nagini documentation and utilizing expert iteration to create supervised fine-tuning datasets to improve the model iteratively. 

Classifying e-mails into distinct labels can have a great impact on customer support. By using machine learning to label e-mails, the system can set up queues containing e-mails of a specific category. This enables support personnel to handle request quicker and more easily by selecting a queue that match their expertise. This study aims to improve a manually defined rule-based algorithm, currently implemented at a large telecom company, by using machine learning. The proposed model should have higher F1-score and classification rate. Integrating or migrating from a manually defined rule-based model to a machine learning model should also reduce the administrative and maintenance work. It should also make the model more flexible. By using the frameworks, TensorFlow, Scikit-learn and Gensim, the authors conduct a number of experiments to test the performance of several common machine learning algorithms, text-representations, word embeddings to investigate how they work together. A long short-term memory network showed best classification performance with an F1-score of 0.91. The authors conclude that long short-term memory networks outperform other non-sequential models such as support vector machines and AdaBoost when predicting labels for e-mails. Further, the study also presents a Web-based interface that were implemented around the LSTM network, which can classify e-mails into 33 different labels. © 2020, The Author(s).

[Context:] Causal relations (e.g., If A, then B) are prevalent in functional requirements. For various applications of AI4RE, e.g., the automatic derivation of suitable test cases from requirements, automatically extracting such causal statements are a basic necessity. [Problem:] We lack an approach that is able to extract causal relations from natural language requirements in fine-grained form. Specifically, existing approaches do not consider the combinatorics between causes and effects. They also do not allow to split causes and effects into more granular text fragments (e.g., variable and condition), making the extracted relations unsuitable for automatic test case derivation. [Objective Contributions:] We address this research gap and make the following contributions: First, we present the Causality Treebank, which is the first corpus of fully labeled binary parse trees representing the composition of 1,571 causal requirements. Second, we propose a fine-grained causality extractor based on Recursive Neural Tensor Networks. Our approach is capable of recovering the composition of causal statements written in natural language and achieves a F1 score of 74% in the evaluation on the Causality Treebank. Third, we disclose our open data sets as well as our code to foster the discourse on the automatic extraction of causality in the RE community. © 2021 IEEE.

Background: The detection and extraction of causality from natural language sentences have shown great potential in various fields of application. The field of requirements engineering is eligible for multiple reasons: (1) requirements artifacts are primarily written in natural language, (2) causal sentences convey essential context about the subject of requirements, and (3) extracted and formalized causality relations are usable for a (semi-)automatic translation into further artifacts, such as test cases. Objective: We aim at understanding the value of interactive causality extraction based on syntactic criteria for the context of requirements engineering. Method: We developed a prototype of a system for automatic causality extraction and evaluate it by applying it to a set of publicly available requirements artifacts, determining whether the automatic extraction reduces the manual effort of requirements formalization. Result: During the evaluation we analyzed 4457 natural language sentences from 18 requirements documents, 558 of which were causal (12.52%). The best evaluation of a requirements document provided an automatic extraction of 48.57% cause-effect graphs on average, which demonstrates the feasibility of the approach. Limitation: The feasibility of the approach has been proven in theory but lacks exploration of being scaled up for practical use. Evaluating the applicability of the automatic causality extraction for a requirements engineer is left for future research. Conclusion: A syntactic approach for causality extraction is viable for the context of requirements engineering and can aid a pipeline towards an automatic generation of further artifacts from requirements artifacts. © 2020 ACM.

The purpose of COVAX is to analyze and create the technical solutions required to provide access through the Internet to homogeneously-encoded document descriptions of archive, library, and museum collections based in the application of XML. This paper describes the development of the project, and the main achievements in building an information system of distributed XML databases containing document descriptions from libraries, archives, and museums emphasizing the conversion processes needed to transform legacy data to an XML environment.

Background: Natural Language Processing (NLP) is a field within artificial intelligence (AI) that focuses on enabling computers to understand, interpret, and generate human language. It includes methods such as tokenization, part-of-speech tagging, parsing, named entity recognition, semantic analysis, machine translation, and text generation. NLP allows computers to learn from text, interact with people more effectively, and automate language-related tasks, improving human-computer interaction.

Objectives: To analyze the High Performance ECU feature elements and convert them into comprehensive test case specifications. Then, evaluate the accuracy and efficiency of the generated test case specifications.

Methods:The study focuses on automating the generation of test case specifications from feature element documents stored in Polarion. It evaluates around 400 feature elements using rule-based and Named Entity Recognition (NER) natural language processing techniques, comparing them against manual methods.

Results:The rule-based approach achieves 95% accuracy for single-signal feature elements. SVM outperformed other algorithms in Named Entity Recognition and the rule-based approach dominated the NER method as well as manual methods.

Conclusions: The Rule-Based Method and NER methods were more efficient and accurate than the manual method for generating test case specifications, demonstrating the potential of NLP-based automation to improve software testing.

The Rule-Based Method out-performed both the NER and manual methods, particularly for less complex requirements. Further refinement of the NER approach is needed to match the performance of the Rule-Based Method, especially for more complex feature elements.

Opinion mining has witnessed significant advancements in well-resourced languages. However, for low-resource languages, this landscape remains relatively unexplored. This paper addresses this gap by conducting a comprehensive investigation into sentiment analysis in the context of Hausa, one of the most widely spoken languages within the Afro-Asiatic family. To resolve the problem, three different models based on Convolutional Neural Network (CNN), Recurrent Neural Network (RNN), and Hierarchical Attention Network (HAN), all tailored to the unique linguistic characteristics of Hausa have been proposed. Additionally, we have developed the first dedicated lexicon dictionary for Hausa sentiment analysis and a customized stemming method to enhance the accuracy of the bag of words approach. Our results indicate that CNN and HAN achieved significantly higher performance compared to other models such as RNN. While the experimental results demonstrate the effectiveness of the developed deep learning models in contrast to the bag of words approach, the proposed stemming method was found to significantly improve the performance of the bag of words approach. The findings from this study not only enrich the sentiment analysis domain for Hausa but also provide a foundation for future research endeavors in similarly underrepresented languages. © 2023 IEEE.

Emotive speech is a non-invasive and cost-effective biomarker in a wide spectrum of neurological disorders with computational systems built to automate the diagnosis. In order to explore the possibilities for the automation of a routine speech analysis in the presence of hard to learn pathology patterns, we propose a framework to assess the level of competence in paralinguistic communication. Initially, the assessment relies on a perceptual experiment completed by human listeners, and a model called the Aggregated Ear is proposed that draws a conclusion about the level of competence demonstrated by the patient. Then, the automation of the Aggregated Ear has been undertaken and resulted in a computational model that summarizes the portfolio of speech evidence on the patient. The summarizing system has a classical emotion recognition system as its central component. The code and the medical data are available from the corresponding author on request. IEEE