SkMTEB: Slovak Massive Text Embedding Benchmark and Model Adaptation

Summary

SkMTEB is introduced as the first comprehensive MTEB-style text embedding benchmark for the Slovak language, consisting of 31 datasets spanning 7 task types. Alongside the benchmark, the authors develop two Slovak-adapted embedding models, e5-sk-small (45M parameters) and e5-sk-large (365M parameters), by applying vocabulary trimming and fine-tuning to the Multilingual E5 models. Despite model size reductions of up to 62%, the resulting models achieve performance competitive with proprietary embedding APIs while remaining fully deployable on local hardware. The work, authored by Marek Šuppa, Andrej Ridzik, Daniel Hládek, Natália Kňažeková, and Viktória Ondrejová, was released on arXiv (2606.13647) and provides a foundational evaluation resource for Slovak NLP, enabling standardized comparison of text embedding models and promoting efficient, locally deployable alternatives to commercial services for low-resource languages.