Azerbaijani Handwritten Text Recognition

The problem

Evler.az receives handwritten documents — rental applications, ownership forms, client notes — that have to become searchable text. Transcribing them by hand is slow and introduces its own errors, and the documents keep arriving.

Standard OCR does not solve this. Off-the-shelf systems are trained overwhelmingly on English and a handful of other well-resourced languages, and handwriting is the hard case even in those. Azerbaijani adds a specific difficulty: the alphabet is Latin-based but carries ə, ğ, ı, ö, ş, ü, ç, and a missed diacritic does not merely misspell a word — it can change which word it is.

Underneath all of it sat the real obstacle. There was no large annotated corpus of handwritten Azerbaijani text lines to train on. The data problem had to be solved before the model problem could be.

Approach

Stage one — synthetic data

I trained on the LocalDoc Azerbaijani synthetic handwritten OCR dataset, a public synthetic corpus of handwritten Azerbaijani text lines in the Latin script. I did not generate it, and that is worth saying plainly: at the start of this project no real handwritten Azerbaijani dataset of usable size existed, publicly or at Evler.az, and a public synthetic set was the only way to get a model off the ground at all.

I scaled the training set deliberately rather than starting at full size, so that a pipeline bug would surface on a cheap run instead of an expensive one:

Images went in as RGB through the TrOCR processor; transcriptions were tokenised and truncated to 128 tokens, with padding replaced by -100 so it was skipped in the loss.

One thing that did not work the first time: preprocessing the whole dataset up front, via a map over the Hugging Face dataset, cached so many transformed samples that RAM became the binding constraint before GPU memory did. Moving to lazy preprocessing — transforming each batch during training instead — is what made the 25,000-sample run stable.

Good enough to prove the approach, not good enough to deploy. Synthetic handwriting is too regular; real handwriting has habits no generator reproduces.

Stage two — real handwriting

This is the part that did not exist before the project. I collected and annotated a 5,000-line dataset of real Azerbaijani handwriting, matching the schema of the synthetic set — an image and its ground-truth transcription — and used it for a second fine-tuning stage on top of the stage-one weights.

The ratio is the interesting part. Going from 5,000 to 25,000 synthetic lines — twenty thousand more images — improved CER by about 1.89×. Adding 5,000 real lines improved it by a further 1.84×. Four times fewer images, for the same proportional gain. Where the data comes from mattered about as much as how much of it there was.

Results in context

A number alone means nothing without a baseline. I trained two:

That is roughly a fivefold reduction in character error against the CNN–BiLSTM–CTC baseline, and about sixfold against TrOCR-small.

One caveat I would rather state than have someone find: the baselines were trained on 10,000 samples and the final model on 30,000. Part of that gap is architecture and part of it is data budget, and this comparison does not separate them. The architecture claim I am confident in is narrower — at equal data, 5,000 lines, TrOCR-base reached 12.1% CER where CNN–BiLSTM–CTC on twice that reached 17.23%.

Put in practical terms: at 21% CER roughly one character in five is wrong and a human re-reads every line anyway. At 3.47% most lines come out clean and the rest are correctable in a review pass. That is the difference between a demo and a tool.

What the errors actually are

Aggregate error rates hide the shape of the failure, so I aligned every prediction against its ground truth with Levenshtein and counted the operations:

Substitutions dominate by roughly seven to one. The model is not losing its place in the line or hallucinating extra text — it gets the length and the order right and confuses individual characters.

Which characters was predictable in hindsight. Azerbaijani-specific letters carried error rates between about 3.9% and 9.8%, with the most frequent confusions being ə ↔ a and ı ↔ i — pairs that differ by a diacritic — alongside visual near-twins like r/n, m/n and l/d. The remaining error is concentrated in exactly the characters that make the language what it is.

What didn't work

I tried rule-based post-processing on the model's output: normalising whitespace, stripping invalid characters, and correcting Azerbaijani-specific mistakes with hand-written rules. It made things worse.

The final system uses the raw model output. The experiment still earned its place: it is what told me the residual errors are not the kind a lookup table can fix. A rule that corrects ə → a in the cases where the model was wrong also corrupts the cases where it was right, and no hand-written rule can tell those apart. Fixing this properly needs a language model over the output, not a substitution table.

What I'd do differently

More writers, not more lines. The 5,000 real images bought a disproportionate improvement, which suggests the model is still short of handwriting variety rather than volume. If I ran this again I would spend the same annotation budget across more people rather than collecting more lines from the same hands.

Give the baselines an equal data budget. Training the comparison models on 10,000 samples while the final model saw 30,000 made the headline number look better than the evidence strictly supports. It costs a few more GPU hours to do it properly and it would have made the architecture claim airtight.

Target the diacritics directly. Knowing that ə↔a and ı↔i are most of the remaining error, an obvious next attempt is weighting those characters in the loss, or augmenting the training data specifically for them, rather than hoping general scale fixes it.

How it would ship

Not as an oracle. The intended integration at Evler.az is human-in-the-loop: a document is scanned, text regions are detected or cropped, the model transcribes, and a person confirms or corrects before anything is written to a record. At 3.47% CER that is a large time saving on a task that was fully manual. At any CER above zero, on documents that carry names, addresses and legal terms, it is the only responsible way to deploy it.

Stack

PyTorch · Hugging Face Transformers · TrOCR (Transformer encoder–decoder) · Python

Trained on a laptop GPU, which shaped the implementation more than the architecture did: small batch sizes with gradient accumulation to simulate a larger effective batch, lazy preprocessing, periodic checkpointing to survive interruptions, and dataloader workers set to zero to keep memory pressure down.