Abstract
Understanding the representations of different languages in multilingual language models is essential for comprehending their cross-lingual properties, predicting their performance on downstream tasks, and identifying any biases across languages. In our study, we analyze the geometry of three multilingual language models in Euclidean space and find that all languages are represented by unique geometries. Using a geometric separability index we find that although languages tend to be closer according to their linguistic family, they are almost separable with languages from other families. We also introduce a Cross-Lingual Similarity Index to measure the distance of languages with each other in the semantic space. Our findings indicate that the low-resource languages are not represented as good as high resource languages in any of the models.
The problem
Multilingual transformers such as mBERT and XLM-R are trained on many languages in a single model, and a common assumption is that they form an interlingua: translations of the same sentence land near each other in a shared space. Whether that actually holds, and whether it holds equally for high- and low-resource languages, matters for cross-lingual transfer and for fairness across languages.
Measuring this is not straightforward. Models like XLM-R are highly anisotropic (any two random embeddings already have high cosine similarity), so raw cosine similarity between languages is hard to interpret.
Approach
We take 300 parallel sentences from XNLI-15way across 15 languages (English, French, Spanish, German, Greek, Bulgarian, Russian, Turkish, Arabic, Vietnamese, Thai, Chinese, Hindi, Swahili, Urdu) and embed them with three multilingual models: bert-base-multilingual-cased (mBERT), Multilingual-MiniLM-L12-H384, and xlm-roberta-base. We study the geometry with three tools:
- PCA visualisation. The top three principal components of the embeddings for a group of languages, plotted in 3D.
- Cross-lingual Similarity Index (Gamma). For a language pair, the mean cosine similarity between embeddings of parallel sentences, divided by the model’s anisotropy (the average cosine similarity between randomly sampled sentence pairs). Values near 1/Anisotropy mean the languages share a region of the space; values at or below 1 mean the pair is no closer than random sentences.
- Language Separability (Phi). Treating each language as a cluster, the pairwise Geometric Separability Index, which measures how often a point’s nearest neighbour has the same language label.
Results
- Distinct geometries. In mBERT the languages occupy clearly different orientations in PCA space. MiniLM and XLM-R embeddings are more compact because of high anisotropy, but languages still sit on different affine regions. In XLM-R, low-resource languages such as Urdu and Swahili are noticeably more dispersed than high-resource ones.
- Similarity follows resources and families. Gamma should be close to 1/Anisotropy for perfect translations, but it is not: it is higher for high-resource languages and for languages in the same family, consistent with a model that first represents a language in isolation and only contextualises it against others as it sees more text. Pre-training also matters; English has high Gamma in XLM-R, plausibly because of its cross-lingual pre-training.
- Near-separable subspaces. For mBERT and XLM-R, Phi between language clusters is extremely high, so languages form near-isolated vector spaces rather than an interlingua. MiniLM is the partial exception: Germanic, Romance, Slavic, and Hellenic languages are assimilated (low Phi), but other families are not.
- Fairness implication. Low average Gamma for low-resource languages points to a shortcoming in cross-lingual capability and a heavy dependence on data volume. High Phi within families suggests that separable language subspaces are easy to learn, but some intersection is needed for cross-lingual transfer, in line with prior findings that language distance in embedding space correlates with transfer performance.
| Language (family) | Gamma with English | Phi vs. English |
|---|---|---|
| German (Germanic) | 1.596 | 0.90 |
| Spanish (Romance) | 1.591 | 0.90 |
| French (Romance) | 1.574 | 0.91 |
| Russian (Slavic) | 1.496 | 0.97 |
| Vietnamese (Vietic) | 1.474 | 0.95 |
| Bulgarian (Slavic) | 1.410 | 0.97 |
| Greek (Hellenic) | 1.351 | 0.97 |
| Turkish (Turkic) | 1.283 | 0.95 |
| Arabic (Arabic) | 1.272 | 0.99 |
| Hindi (Hindustani) | 1.237 | 0.99 |
| Urdu (Hindustani) | 1.174 | 0.99 |
| Thai (Tai) | 1.096 | 0.97 |
| Swahili (Niger-Congo) | 1.048 | 0.97 |
| Chinese (Chinese) | 0.607 | 0.98 |
Source: Figure 1 (b, c) and Table 1 of the paper, mBERT. Gamma is the mean cosine similarity of parallel sentences divided by the model's anisotropy; its ceiling for mBERT is 1/Anisotropy = 2.456, and a value at or below 1 means the pair is no closer than random sentences. Phi is the Geometric Separability Index between the two language clusters (1 = fully separable). Higher Gamma and lower Phi indicate a more shared representation.
| Model | 1/Anisotropy (ceiling of Gamma) |
|---|---|
| mBERT (bert-base-multilingual-cased) | 2.456 |
| Multilingual-MiniLM-L12-H384 | 1.12 |
| XLM-R (xlm-roberta-base) | 1.019 |
Source: diagonal of the Gamma heatmaps in Figures 1 and 2 of the paper. A ceiling close to 1 means the model is highly anisotropic: random sentence pairs already have cosine similarity near 1, which is why Gamma rather than raw cosine similarity is used to compare languages.
Resources
- Paper: OpenReview · arXiv:2305.07839 · Hugging Face papers (the arXiv version carries the subtitle “An Equality Lens”)
- Data: XNLI (Conneau et al., 2018)
- Models: bert-base-multilingual-cased, xlm-roberta-base, Multilingual-MiniLM-L12-H384
- Related: WADER, our SemEval-2023 work on multilingual tweet intimacy with XLM-RoBERTa
- All publications