Post

Low-Resource NLP: Transfer Learning and Data Augmentation for African Languages

Posted
Transfer learning diagram showing knowledge flowing from multilingual model to low-resource language
Transfer learning diagram showing knowledge flowing from multilingual model to low-resource language
By ML Kenya
8 min read
Low-Resource NLP: Transfer Learning and Data Augmentation for African Languages

The Low-Resource Reality

Africa is home to over 2,000 languages — roughly one-third of all living languages on Earth. Yet when we look at language technology coverage, the picture is stark:

  • High-resource: English, French, Arabic (300M+ African speakers, well-covered)
  • Medium-resource: Swahili, Hausa, Yoruba, Amharic (datasets exist, models are possible)
  • Low-resource: Kikuyu (8M speakers), Wolof (10M speakers), Lingala (20M speakers) — very limited data
  • Extremely low-resource: Igbo (44M speakers but very little digital text), Oromo (37M speakers, written in multiple scripts), Twi (11M speakers) — virtually no usable datasets for most NLP tasks
  • Zero-resource: Hundreds of languages with no digital presence at all

For most of these languages, collecting enough annotated data for traditional supervised learning is prohibitively expensive. A single human-annotated NER dataset for a new language costs $20,000-$50,000 — often more than an entire research group’s annual budget.

This is where low-resource NLP techniques become essential. The rest of this post covers practical methods you can use when you have little to no labeled data for an African language.

Transfer Learning from Multilingual Models

The Multilingual Model Revolution

Models trained on 100+ languages simultaneously have been game-changers. The key insight: a model that’s seen dozens of languages learns a language-agnostic representation space. When you fine-tune on a specific language, it can leverage patterns from related languages.

ModelLanguagesParametersNotes for African NLP
mBERT110178MGood baseline, small vocabulary per language
XLM-R100278M-550MStronger on low-resource languages
NLLB-200200600M-54.5BBest for translation, includes many African languages
LaBSE109471MExcellent for cross-lingual sentence retrieval
RemBERT100580MBetter per-language capacity than XLM-R

Practical Transfer Strategy

The most effective approach for a low-resource African language follows this hierarchy:

Level 1: Zero-shot cross-lingual transfer Fine-tune on a high-resource language (e.g., English), then evaluate directly on the target language. This requires no target-language labels. Works surprisingly well for related languages.

1
2
3
4
5
6
7
8
9
10
11
12

from transformers import pipeline

# Zero-shot sentiment analysis on Kikuyu using English-trained model
classifier = pipeline(
    "text-classification", 
    model="xlm-roberta-base-sentiment-en"
)

# Test on Kikuyu text
kikuyu_text = "Nĩndĩmwendete mũno"  # "I love you very much"
result = classifier(kikuyu_text)
print(result)  # May get: [{'label': 'POSITIVE', 'score': 0.87}]

Level 2: Few-shot fine-tuning Collect 50-500 labeled examples in the target language. Fine-tune the multilingual model on this small dataset. With careful training (low learning rate, heavy regularization), this often matches or exceeds a model trained on 10K+ English examples.

Level 3: Continued pre-training (language adaptation) Before fine-tuning, continue pre-training the multilingual model on unlabeled text in the target language (if available). Even 10MB of raw text helps the model adapt its subword representations to the target language’s morphology.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44

from transformers import (
    AutoModelForMaskedLM, 
    AutoTokenizer, 
    DataCollatorForLanguageModeling,
    Trainer, 
    TrainingArguments
)
from datasets import Dataset

# Collect unlabeled text in your target language
raw_texts = load_unlabeled_text("yoruba")  # Hypothetical function

dataset = Dataset.from_dict({"text": raw_texts})
tokenizer = AutoTokenizer.from_pretrained("xlm-roberta-base")

def tokenize_fn(examples):
    return tokenizer(examples["text"], truncation=True, max_length=128)

tokenized = dataset.map(tokenize_fn, batched=True, remove_columns=["text"])

data_collator = DataCollatorForLanguageModeling(
    tokenizer=tokenizer, mlm=True, mlm_probability=0.15
)

model = AutoModelForMaskedLM.from_pretrained("xlm-roberta-base")

training_args = TrainingArguments(
    output_dir=f"./xlmr-lang-adapted-yoruba",
    per_device_train_batch_size=8,
    num_train_epochs=3,
    learning_rate=5e-5,
    save_steps=5000,
)

trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=tokenized,
    data_collator=data_collator,
)
trainer.train()

# Save the language-adapted model
model.save_pretrained("./xlmr-yoruba-adapted")

Level 4: Cross-lingual transfer from related languages For a language like Kikuyu (E40, Bantu branch), fine-tune on Swahili (E20, Bantu) or Zulu data, then evaluate on Kikuyu. Bantu languages share agglutinative morphology, noun class systems, and significant vocabulary. This is often more effective than zero-shot from English.

Data Augmentation Techniques

When labeled data is the bottleneck, augmentation creates synthetic examples:

Back-Translation

The most reliable augmentation method: take your target-language text, translate it to a high-resource language (English), then translate it back. The round-trip introduces natural variation.

1
2
3

Original (Hausa): "Ina son karanta littattafai" (I like reading books)
→ English: "I like reading books"
→ Back to Hausa: "Ina jin daɗin karanta littattafai" (slight word choice change, same meaning)

Use the NLLB-200 model for both translation directions:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25

from transformers import pipeline

# Use NLLB for back-translation
translator_en_to_ha = pipeline(
    "translation", model="facebook/nllb-200-distilled-600M", 
    src_lang="eng_Latn", tgt_lang="hau_Latn"
)

translator_ha_to_en = pipeline(
    "translation", model="facebook/nllb-200-distilled-600M",
    src_lang="hau_Latn", tgt_lang="eng_Latn"
)

hausa_sentences = [
    "Ina son karanta littattafai",
    "Yana zuwa makaranta kowace rana",
]

augmented = []
for sent in hausa_sentences:
    # Translate to English
    en = translator_ha_to_en(sent)[0]["translation_text"]
    # Translate back to Hausa (multiple variations possible)
    back_ha = translator_en_to_ha(en)[0]["translation_text"]
    augmented.append(back_ha)

Caveat: Back-translation quality depends on the translation model’s strength in both languages. For extremely low-resource languages, the NLLB model may produce poor translations. Always validate a sample manually.

Word-Level Substitution with Multilingual Embeddings

Replace words in the original sentence with semantically similar words from the same language, using contextual word embeddings:

1
2
3
4
5
6
7
8
9
10
11
12

import numpy as np
from sentence_transformers import SentenceTransformer

model = SentenceTransformer("sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2")

def get_word_embedding(word, context_sentence):
    """Get contextual embedding for a word"""
    # Simplified: use sentence-level embedding as proxy
    return model.encode(context_sentence)

# For a real system, use context-aware substitution
# e.g., replace "good" with "excellent" in sentiment-preserving way

Synonym Replacement via Machine Translation

For languages without thesauri: translate the word to English, get synonyms, translate back:

1
2
3
4

Hausa word: "kyakkyawa" (beautiful)
→ English: "beautiful"
→ Synonyms: "gorgeous", "lovely", "stunning"
→ Back to Hausa: "kyakkyawa", "kyau sosai", "mai ban sha'awa"

MixUp for Text

Create interpolated examples by mixing two training examples. For text, this means combining the embeddings:

1
2
3
4
5
6
7

def text_mixup(input_ids_1, input_ids_2, label_1, label_2, alpha=0.3):
    """Create mixed example from two training samples"""
    lam = np.random.beta(alpha, alpha)
    # Mix embeddings (requires embedding layer access)
    mixed_input = lam * input_ids_1 + (1 - lam) * input_ids_2
    mixed_label = lam * label_1 + (1 - lam) * label_2
    return mixed_input, mixed_label

Few-Shot Prompting for Low-Resource NLP

Large language models (LLMs) can perform NLP tasks in low-resource languages via in-context learning — even when they haven’t been explicitly fine-tuned.

Zero-Shot Prompting

1
2
3
4

Classify the sentiment of this Wolof sentence as positive, negative, or neutral.

Wolof: "Dama faayantu, xale yi rekk bëgg nañu leen."
Sentiment:

Few-Shot Prompting (with examples in target language)

1
2
3
4
5
6
7
8
9
10

Classify the sentiment of Wolof sentences.

Wolof: "Dama faayantu lool."  
Sentiment: POSITIVE

Wolof: "Lu bon lii, xale yi duñu ko bëgge."  
Sentiment: NEGATIVE

Wolof: "Dama faayantu, xale yi rekk bëgg nañu leen."  
Sentiment:

The key insight from Masakhane’s 2024 study: few-shot prompting works best when the examples use the exact same dialect and register as the test sentence. A single example using standard Wolof in a formal register improves performance on formal test sentences by 15%, but provides zero benefit for informal/urban Wolof sentences.

Practical Workflow for a New Low-Resource Language

Here’s the step-by-step process we recommend:

  1. Assess resources: Check HuggingFace, Masakhane repo, and Lacuna Fund for existing data. Even if none exists for your task, check for related data (e.g., unlabeled text, parallel corpora).

  2. Choose a base model: XLM-R is the safest choice. If you need translation, use NLLB-200.

  3. Collect unlabeled text: Even 5,000 sentences of raw text helps. Sources: Bible translations (available for 1,500+ African languages), religious tracts, government websites, local news blogs.

  4. Language-adapt the model: Continued pre-training on unlabeled target-language text (Level 3 above).

  5. Collect 100-500 labeled examples: Work with native speakers. Lacuna Fund provides grants for this. The Masakhane community can connect you with speakers.

  6. Fine-tune with augmentation: Use back-translation (3x-5x augmentation factor) and train with heavy regularization.

  7. Evaluate on human-curated test set: Automatic metrics can mislead for low-resource languages. Manual evaluation by 2+ native speakers is essential.

  8. Iterate: The most common failure mode is domain mismatch. If your training data is formal news text but your deployment target is social media, augment with real in-domain data.

The Ethical Dimension

Low-resource NLP carries risks:

  • Performance disparities: A model that works at 92% accuracy in English but only 71% in Hausa creates a two-tier service. Who gets the worse experience? Almost always the less economically powerful population.

  • Data colonialism: Don’t parachute into a community, extract data, publish a paper, and leave. Follow the Masakhane model: involve local researchers as co-authors, share models and data back to the community, and ensure the technology serves the speakers.

  • Language extinction risk: Building “good enough” NLP for a minority language can reduce the incentive to maintain the language. Any technology we build should support revitalization, not replacement.

The Masakhane community’s principle is worth repeating: “Nothing about us without us.” Every low-resource NLP project should have native speakers as partners, not just subjects.

Next in the series: Deploying AI on mobile devices — why TensorFlow Lite is the most important ML framework in Africa.

Machine Learning, AI in Africa
africa low-resource natural-language-processing transfer-learning data-augmentation
This post is licensed under CC BY 4.0 by the author.