The Challenge of Transliterating Abjad Scripts

Transliterating scripts based on the Arabic Abjad system (such as Modern Standard Arabic, Ottoman Turkish, or Persian) into Latin characters presents unique NLP challenges. Unlike alphabets, Arabic script primarily writes consonants and long vowels, suppressing short vowels (tashkeel or harakat). Additionally, agglutinative prefixes, suffixes, and syntactic connectors—like the Persian Ezafe (-e/-i)—mean that a word's Latin transliteration heavily depends on surrounding context and implicit vocalization.

When building Sequence-to-Sequence (Seq2Seq) models for transliteration, choosing the right tokenization approach for the source script (encoder) is critical. Here is a comprehensive guide to evaluating BPE vs. Word-level vs. Character-level tokenization for Arabic script transliteration.

Why a 16K BPE Tokenizer Might Be Hindering Your Model

Using a standard Byte-Pair Encoding (BPE) tokenizer with a 16,000-symbol vocabulary on the Arabic encoder often creates significant problems:

  • Over-Segmentation vs. Overspecialization: A 16K vocabulary is often too large unless your training corpus contains tens of millions of sentence pairs. The tokenizer will memorize long, rare word forms instead of learning structural subwords, leading to severe out-of-vocabulary (OOV) or sub-optimal split issues during inference.
  • Ignoring Morphological Boundaries: Native BPE algorithms split tokens based purely on character pair frequencies rather than linguistic boundaries. In Arabic and Persian, splitting across root letters (radicals) or clitics degrades the model's ability to infer missing short vowels.

Comparing Tokenization Approaches for Arabic/Persian Encoders

1. Character-Level Tokenization (Recommended with Contextual Encoders)

Character-level tokenization (vocabulary size ~50–100 symbols) is standard practice for target decoders, but it is also remarkably effective for encoders when paired with attention-based architectures or modern Transformer/GRU setups.

  • Pros: Zero OOV tokens, tiny vocabulary size, and allows deep neural network layers to learn custom contextual embeddings for implicit vowels.
  • Cons: Longer sequence lengths requiring higher memory during training.

2. Morphologically Aware Subword Tokenization (Best Overall Accuracy)

Instead of pure BPE or raw word-level tokenization, the optimal strategy for Arabic script transliteration involves morphological segmentation followed by character or small subword tokenization.

  • For Arabic: Use tools like Farasa or CAMeL Tools to segment clitics (e.g., و+, ال+) before tokenization.
  • For Persian: Use tools like Hazm or Parsivar to standardize half-spaces (ZWNJ) and separate prefixes/suffixes (e.g., می‌شود into می + شود).

3. Word-Level Tokenization (Not Recommended)

Full word-level tokenization requires huge vocabularies (100k+ tokens) due to morphological rich features, yet it still fails on unseen word inflections. It is strongly discouraged for transliteration tasks.

Recommended Solution Architecture & Workflow

To achieve state-of-the-art results for custom transliteration standards, follow this pipeline:

  1. Preprocessing/Normalization: Strip non-standard characters and normalize Unicode variants (e.g., unifying Persian ی / ک and Arabic variants).
  2. Automatic Diacritization (Optional but High Impact): Run the source text through an automatic diacritizer (e.g., mishkal or shakkelha for Arabic) to restore short vowels prior to tokenization.
  3. Encoder Tokenization: Use character-level or a small BPE vocabulary (1,000 to 2,000 tokens) on morphologically pre-segmented text.
  4. Decoder Tokenization: Character-level (~40–60 tokens).

Implementation Example: Pre-Segmentation with Python

Here is an example demonstrating how to pre-segment Persian and Arabic text using popular tools before feeding them to your Seq2Seq tokenizer: