Background We explore key concepts including IT Service Management (ITSM) and Incident Management, highlighting the importance of incident data in improving service operations. We also cover the significance of Data Analytics in extracting insights and its relevance to ITSM. The capabilities of Large Language Models (LLMs) in generating code for data analysis are discussed, alongside various prompting techniques used to refine LLM responses. This foundational understanding sets the stage for evaluating how effectively LLMs can generate code for complex data analytics tasks.

Objectives The Objectives of this thesis are threefold. First, we aim to evaluate the ability of various LLMs to generate accurate and relevant code for data analytics using ITSM data. Second, we seek to identify and assess the effectiveness of different prompting techniques in guiding LLMs to produce structured and task-relevant outputs for ITSM-related tasks. Finally, we explore how different LLMs can facilitate conversational generation to enhance and streamline ITSM data analysis, ensuring more effective and interactive data-driven decision-making.

Method We conducted a comprehensive literature review to identify key prompting strategies—Zero-Shot Prompting, Priming, and Chain-of-Thought Prompting. These strategies were experimentally validated using three LLMs: LLAMA3.1-8B, GEMMA29B, and PHI3-14B. The models were tested on four categories of tasks: Descriptive, Statistical, Graph Plot, and Data Cleaning. The effectiveness of each prompting strategy and LLM was measured through correctness, relevance, and clarity across multiple task domains.

Results The findings demonstrate that Priming emerged as the most effective prompting strategy, significantly enhancing the models’ ability to generate task-relevant and accurate code. Chain-of-Thought Prompting improved logical reasoning but still struggled with data handling. Zero-Shot Prompting, though useful for general tasks, consistently failed in complex multi-step tasks. Among the models, GEMMA2-9B outperformed others, particularly in Descriptive and Data Cleaning tasks, providing accurate, relevant, and clear outputs based on user feedback. PHI3-14B excelled in Graph Plotting, while all models showed significant weaknesses in handling Statistical tasks such as hypothesis testing.

Conclusions This thesis demonstrates the critical role of structured prompting strategies, particularly Priming, in improving the performance of LLMs in code generation for data analytics. The experimental results show that while GEMMA2-9B performs best in Descriptive and Data Cleaning tasks, PHI3-14B performs best in Graphical Analysis. However, there remains a need for further development in handling statistical reasoning across all models.