Different Hallucinations calls for Different Solutions - A Categorisation of LLM Transcription Mistakes
2025 (English) In: CEUR Workshop Proceedings / [ed] Nowaczyk S., Vettoruzzo A., Technical University of Aachen , 2025, Vol. 4037, p. 39-52Conference paper, Published paper (Refereed)
Abstract [en]
This paper presents a contribution to better interpretation of the results we get from GenAI models, more specifically, better interpretation of the mistakes that they make. We have conducted an analysis of 644 (from GPT-4o) + 4858 (from ARIA) mistakes made by two models on a key-value extraction task, and found that they may be categorised into three mutually exclusive groups. These groups are; i problems identifying the requested information, p problems presenting the correct information, and s skewed training data. These categories could be used to indicate which action a user could take to reduce the number of mistakes. Further, we have found a strong correlation between the suggested categories and the Ratcliff/Obershelp pattern recognition score between the generated result and the expected result; all faulty results containing minor mistakes are more than 60% similar to the expected result. Only mistakes based on lack of identifying what was requested had less than 60% similarity to the expected result.
Place, publisher, year, edition, pages Technical University of Aachen , 2025. Vol. 4037, p. 39-52
Series
CEUR Workshop Proceedings, E-ISSN 1613-0073
Keywords [en]
Document analysis, Generative AI, LLM, Verification, Artificial intelligence, Errors, Human engineering, Information retrieval systems, Pattern recognition, Documents analysis, Key values, Strong correlation, Training data
National Category
Artificial Intelligence
Identifiers URN: urn:nbn:se:bth-28782 Scopus ID: 2-s2.0-105017736795 OAI: oai:DiVA.org:bth-28782 DiVA, id: diva2:2007260
Conference 2025 Swedish AI Society Workshop, SAIS 2025, Halmstad, June 16-17, 2925
2025-10-172025-10-172025-10-17 Bibliographically approved