Change search
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf
When Multilingual Embeddings Collapse: Representation Failure and Metric (Un)reliability for Hausa and Fongbe Machine Translation
Blekinge Institute of Technology, Faculty of Computing, Department of Computer Science.ORCID iD: 0009-0006-1284-5563
North-West University, South Africa.
Victoria University of Wellington, New Zealand.
2026 (English)In: Proceedings of the 6th Workshop on Multilingual Representation Learning (MRL 2026), Association for Computational Linguistics , 2026, p. 377-385Conference paper, Published paper (Refereed)
Abstract [en]

Neural evaluation metrics for machine translation inherit their behaviour from the multilingual encoders beneath them, yet how those encoders represent low-resource African languages is rarely examined. We evaluate four large language models on English-to-Hausa and English-to-Fongbe translation across scales from 500 to 10,000 sentences, validate automatic metrics against native-speaker judgment, and then probe the multilingual representation space that neural metrics depend on. The central finding is a representation failure: XLM-R assigns near-identical vectors to semantically different texts in both languages (within-language cosine similarity above 0.99), so BERTScore cannot separate systems of visibly different quality, and COMET produces a ranking anomaly for Fongbe. This representation collapse is a central reason embeddingbased metrics are unreliable here, rather than any surface property of the languages. We further show that metric–human rank agreement is language-dependent, that in our samples system rankings stabilise by about 2,500 sentences, and that inclusion of a language in an encoder’s pretraining set does not guarantee usable representations. We frame these results as evidence that multilingual evaluation should be checked at the representation level before it is trusted.

Place, publisher, year, edition, pages
Association for Computational Linguistics , 2026. p. 377-385
National Category
Natural Language Processing
Identifiers
URN: urn:nbn:se:bth-30643ISBN: 9798891764699 (print)OAI: oai:DiVA.org:bth-30643DiVA, id: diva2:2113408
Conference
6th multilingual representation learning (MRL) workshop 2026, Budapest, Oct 24-29, 2026
Available from: 2026-10-28 Created: 2026-10-28 Last updated: 2026-10-28Bibliographically approved

Open Access in DiVA

fulltext(219 kB)21 downloads
File information
File name FULLTEXT01.pdfFile size 219 kBChecksum SHA-512
667317106d4eb8473e96ba1583b7bf4848da437f0259a0d8f5292a59743d7c2aa21ed17251493531a37a50875cf4528001a2b9e550fd9a4b039bea7696624567
Type fulltextMimetype application/pdf

Authority records

Adjovi, Mahounan Pericles

Search in DiVA

By author/editor
Adjovi, Mahounan Pericles
By organisation
Department of Computer Science
Natural Language Processing

Search outside of DiVA

GoogleGoogle Scholar
Total: 21 downloads
The number of downloads is the sum of all downloads of full texts. It may include eg previous versions that are now no longer available

isbn
urn-nbn

Altmetric score

isbn
urn-nbn
Total: 14443 hits
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf