Fine-Tuning LLMs on African Language Datasets

Why Fine-Tune?

A foundation model like Llama 3 70B or Mistral generates fluent English, French, and Arabic. Ask it to write in Swahili or translate a sentence to Yoruba, and quality drops off a cliff.

The reason is straightforward: training data distribution. The largest models are trained on corpora where English comprises 50-80% of tokens. African languages account for fractions of a percent, even widely spoken ones like Swahili (200M speakers). The model simply hasn’t seen enough examples to learn the grammar, idioms, and vocabulary of these languages.

Fine-tuning changes that. With as little as 10,000-50,000 high-quality examples in a target language, you can dramatically improve a model’s performance on African language tasks: translation, classification, summarisation, and generation. And thanks to Parameter-Efficient Fine-Tuning (PEFT), this is achievable on hardware you probably already have.

This builds on our earlier deep dives into Swahili NLP and Low-Resource NLP Techniques. Fine-tuning is the next logical step: once you know the challenges and the transfer strategies, you need a practical pipeline to adapt models to your specific language.

Available datasets

The African NLP community (particularly the Masakhane project) has done the hard work of curating and annotating datasets. Here are the key resources for fine-tuning:

• Masakhane datasets: A growing collection of annotated datasets for translation, NER, sentiment, and news classification across 50+ African languages. The Masakhane GitHub hosts the primary repositories.

• MasakhaNER (2021, updated 2024): Named Entity Recognition annotations for 20+ African languages including Yoruba, Hausa, Igbo, Swahili, Amharic, and Wolof. Each language has ~10K-15K annotated sentences.

• AfriQA: A question-answering dataset covering 10 African languages, built by translating and culturally adapting the SQuAD format. Useful for instruction-tuning LLMs to answer questions in African languages.

• MasakhaNEWS: News topic classification (5 categories) for 15 African languages, with ~10K articles per language.

• SERAKWAYE: A benchmark for African language understanding covering 42 languages across 3 tasks (sentiment, topic classification, NER). Designed specifically to evaluate multilingual models on African languages.

• JW300: A parallel corpus of ~3.5M sentence pairs between English and 300+ languages, including extensive African language coverage. Ideal for translation fine-tuning.

For Swahili specifically, the datasets covered in our Swahili NLP post (CC-100, OSIAN, GlobalVoices, NLLB) provide additional pre-training material.

Three key techniques

1. LoRA and QLoRA (Parameter-Efficient Fine-Tuning)

LoRA (Low-Rank Adaptation) freezes the base model weights and injects trainable low-rank matrices into each transformer layer. Instead of updating 70B parameters, you train a few million. A 7B model fine-tunes on 16GB of GPU memory with LoRA. Training runs 2-4x faster because gradients only flow through the LoRA weights, and the adapter is a ~25MB file you can share independently of the base model.

QLoRA goes further by quantizing the base model to 4-bit (NF4 format) before applying LoRA. That brings a 70B parameter model within reach of a single RTX 4090 (24GB), a job that would otherwise need 8× A100s.

2. Continued Pre-Training with tokenizer extension

Base model tokenizers rarely include African language vocabulary efficiently. Our Swahili example from the earlier post showed one word (hawakuwasomea) fragmenting into 7 subword tokens. This wastes context window and increases compute.

The solution is tokenizer extension:

1. Train a new BPE tokenizer on a large African language corpus (e.g., OSIAN Swahili)
2. Merge the new tokens into the base model’s tokenizer
3. Run continued pre-training (masked language modeling or causal LM) on the extended vocabulary

This adds only the tokens that the original tokenizer handled poorly, usually a few thousand new tokens.

3. Instruction tuning on translated or adapted data

For task-specific fine-tuning (translation, QA, summarisation), convert your datasets into instruction format:

### Instruction:
Translate the following sentence from English to Swahili.

### Input:
The agricultural sector employs 60% of the workforce in East Africa.

### Response:
Sekta ya kilimo inaajiri asilimia 60 ya wafanyakazi katika Afrika Mashariki.

This format works with chat-tuned models (Llama, Mistral, Qwen) and produces dramatically better results than standard fine-tuning.

Practical pipeline

Here’s an end-to-end pipeline using HuggingFace TRL (Transformer Reinforcement Learning) for QLoRA fine-tuning of a translation model:

import torch
from transformers import AutoTokenizer, BitsAndBytesConfig
from trl import SFTTrainer
from peft import LoraConfig, get_peft_model
from datasets import load_dataset

# ===== 1. Dataset Preparation =====
# Load JW300 Swahili-English parallel data
dataset = load_dataset("masakhane/jw300", "sw-en", split="train")

def format_instruction(example):
    return {
        "text": f"### Instruction:\nTranslate English to Swahili.\n\n### Input:\n{example['en']}\n\n### Response:\n{example['sw']}"
    }

dataset = dataset.map(format_instruction)

# ===== 2. Tokenizer Extension (if needed) =====
tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-3.2-3B")

# Check tokenizer efficiency on Swahili text
test_word = "hawakuwasomea"
tokens = tokenizer.tokenize(test_word)
print(f"Base tokenizer: {len(tokens)} tokens for '{test_word}'")
# If token count is high, extend the tokenizer here
# new_tokens = train_new_bpe(swahili_corpus)
# tokenizer.add_tokens(new_tokens)

tokenizer.chat_template = None  # Use default

# ===== 3. QLoRA Configuration =====
bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16,
)

lora_config = LoraConfig(
    r=16,           # rank of the LoRA matrices
    lora_alpha=32,  # scaling factor
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj"],
    lora_dropout=0.05,
    bias="none",
    task_type="CAUSAL_LM",
)

# ===== 4. Training =====
trainer = SFTTrainer(
    model="meta-llama/Llama-3.2-3B",
    train_dataset=dataset,
    tokenizer=tokenizer,
    args=TrainingArguments(
        output_dir="./swahili-llm-lora",
        per_device_train_batch_size=4,
        gradient_accumulation_steps=4,
        warmup_steps=100,
        num_train_epochs=3,
        learning_rate=2e-4,
        fp16=True,
        logging_steps=25,
        save_strategy="epoch",
    ),
    peft_config=lora_config,
    dataset_text_field="text",
    max_seq_length=1024,
)

trainer.train()

# ===== 5. Merge and Export =====
model = trainer.model.merge_and_unload()
model.save_pretrained("./swahili-llm-merged")
tokenizer.save_pretrained("./swahili-llm-merged")

Hardware reality check

The pipeline above uses Llama 3.2 3B as an example. It fine-tunes in about 2-3 hours on a single RTX 4090 (24GB) or RTX 3090.

For larger models, here’s the QLoRA footprint:

• 3B: 8-10 GB (RTX 3060 / any 8GB+), roughly 2 hours (3 epochs, 50K examples)
• 7-8B: 14-16 GB on an RTX 3090/4090, roughly 4 hours
• 13B: 18-20 GB on an RTX 4090, roughly 6-7 hours
• 70B: 22-24 GB on an RTX 4090 (just barely), roughly 20-24 hours

This is the core message of our constrained-environment ML theme: you don’t need a cluster. A single consumer GPU, careful use of 4-bit quantization, and LoRA adapters let you produce production-quality African language models on hardware that fits under a desk.

The limiting factor is data, not compute. Investing in building high-quality African language datasets (via Masakhane, local universities, or community annotation drives) delivers far more impact than buying more GPUs.

Testing your fine-tuned model

After merging, evaluate on the SERAKWAYE benchmark or run a quick qualitative test:

from transformers import pipeline

pipe = pipeline("text-generation", model="./swahili-llm-merged")

prompt = "### Instruction:\nTranslate to Swahili.\n\n### Input:\nI would like to buy maize and beans.\n\n### Response:\n"
result = pipe(prompt, max_new_tokens=50)[0]["generated_text"]
print(result)
# Expected: "Ningependa kununua mahindi na maharagwe."

Compare the output against the base model without fine-tuning. The difference is dramatic for a weekend’s work and a single GPU.

Summary

Fine-tuning LLMs on African languages is one of the highest-leverage activities in NLP today. The community datasets exist, the techniques (LoRA/QLoRA, tokenizer extension, instruction tuning) are mature, and the hardware requirements are modest. The gap between base model performance and what’s possible after fine-tuning is enormous. Those base models were never trained on your target language to begin with.

If you’ve been reading through this series (from Swahili NLP through Low-Resource NLP to this post), you now have the full toolkit: understanding the linguistic challenges, transfer learning strategies, and a practical fine-tuning pipeline. The next step is picking a language and running the experiment.

Next in the series, Evaluating African Language LLMs: benchmarks, human evaluation protocols, and building trust in model outputs for production deployment.