# Project '44 — OCR Model Lessons Learned
*War Diary OCR Pipeline | Calgary Highlanders Sep 44 Control Test*
*Compiled May 2026*

---

## 1. Test Corpus

All model evaluations used the same 32-page control PDF derived from the Calgary Highlanders War Diary, September 1944. The 32 pages were selected to cover the full range of document types present in a typical war diary file:

| Document Type | Pages in Test | Notes |
|---|---|---|
| Typed narrative diary entries (HTML table) | 7 | Core data pages — Place / Date / Hour / Summary / Remarks |
| Message forms (Army Form C2136) | ~8 | Dense typed text, some on grey or degraded paper |
| Artillery / DF task tables | 2 | Complex tabular data with grid references |
| Field Return of Officers (dual-page form) | 1 | Complex nominal roll — 42 named officers |
| Field Return of Other Ranks | 1 | Complex dual-form, dense |
| Movement orders | 2 | Mixed typed + handwritten annotation |
| Hand-drawn maps / traces / sketch maps | 4 | APIS trace, disposition sketch, report lines, DMA order |
| Handwritten narrative appendix | 1 | Cursive, "Enroute Loon Plage to Antwerp" |
| Regimental journal (The Glen) | 3 | Cover, CO's address (degraded), newsletter pages |
| Administrative pages | 3 | Cover, instructions, appendix index, signing page |

---

## 2. Models Tested

| Model | API / Provider | Status | Run Method |
|---|---|---|---|
| olmOCR 2-7B (`allenai/olmOCR-2-7B-1025`) | DeepInfra | ✅ Completed | `step1_ocr_wardiaries_olmocr.py` |
| Gemini 2.0 Flash (`google/gemini-2.0-flash`) | DeepInfra | ✅ Completed | `ocr_model_comparison.py` |
| Gemini 2.5 Flash (`google/gemini-2.5-flash`) | DeepInfra | ✅ Completed | `ocr_model_comparison.py` |
| Gemini 2.5 Pro (`google/gemini-2.5-pro`) | DeepInfra | ✅ Completed | `ocr_model_comparison_anthropic.py` |
| Qwen3-VL-30B (`Qwen/Qwen3-VL-30B-A3B-Instruct`) | DeepInfra | ✅ Completed (disqualified) | `ocr_model_comparison.py` |
| Claude 3.7 Sonnet (`anthropic/claude-3-7-sonnet-latest`) | DeepInfra | ❌ 404 — model not available on DeepInfra | `ocr_model_comparison.py` |
| Claude Sonnet 4.6 (`claude-sonnet-4-6`) | Anthropic API direct | ✅ Completed | `step1_ocr_wardiaries_claude-4-6.py` |

### Key Lesson: Claude via DeepInfra Does Not Work

`claude-3-7-sonnet-latest` was routed through DeepInfra's OpenAI-compatible endpoint. Every page returned `404 Not Found`. DeepInfra does not host Anthropic models. All Claude testing must use the Anthropic API directly (`api.anthropic.com/v1/messages`) with `x-api-key` header and `anthropic-version: 2023-06-01`.

---

## 3. Overall Model Rankings

### Typed Diary Content (narrative pages, message forms, movement orders)

| Rank | Model | Notes |
|---|---|---|
| 1 | **Gemini 2.5 Pro** | Best overall. No paraphrase contamination, no factual hallucination. Wins ~25/32 pages by category. |
| 2 | **Claude Sonnet 4.6** | Strong on structured diary tables. Outputs clean HTML tables consistently. Not yet formally scored against control test on all 32 pages. |
| 3 | **olmOCR 2-7B** | Reliable baseline. Consistent output structure. See failure modes below. |
| 4 | **Gemini 2.5 Flash / 2.0 Flash** | Identical outputs. Viable for typed pages; fails on handwriting. Not a Pro replacement. |
| 5 | **Qwen3-VL-30B** | **Disqualified.** Catastrophic failure modes. Do not use. |

---

## 4. Model-by-Model Findings

---

### 4.1 olmOCR 2-7B

**Overall verdict:** Reliable baseline. Never enters infinite loops, never silently fails on typed pages. Failure modes are predictable and lower-severity than Gemini Flash or Qwen. Currently used as the production step 1 model pending full Claude 4.6 evaluation.

**Strengths:**
- Consistent structured output (HTML tables where appropriate, plain text elsewhere)
- No base64 embedding or `<think>` block leakage
- Good performance on clean typed diary narrative pages
- Handles most message forms adequately
- Timeout is the only failure mode on complex forms (deterministic — not silence)

**Failure modes:**

| Failure | Page(s) | Severity | Example |
|---|---|---|---|
| Paraphrase contamination | 2, 5 | High | "of the force" → "of the army as a whole"; verbatim accuracy compromised on boilerplate |
| Factual hallucination | 4 | **Critical** | "General Montgomery" instead of "General Crerar" — wrong person at the Dieppe march past |
| Meaning reversal | 6 | **Critical** | "roads were reported clear" → "roads were reported closed" — flips historical record |
| Artillery garbling | 7 (p.30) | High | "some 25 pdrs fell short" → "some 20 plrs fell asleep" — loses calibre and military meaning |
| Timeout on complex forms | 9 | High | Field Return of Other Ranks times out at 180s. Complex multi-part forms exceed limit. |
| Handwriting degradation | 32 | High | "Loon Plage" → "Caen Place"; "MacLauchlan" → "G.H. Launchlan"; distances wrong |
| Reversed initials | 8 (p.51) | Low | "R. L. Ellis" → "A. R. L. Ellis" |
| Nominal roll errors | 10 | Medium | CO name garbled; table structure breaks down mid-form |

**Post-processing notes:**
- olmOCR output is the cleanest raw text of all tested models — no stripping required
- Timeout is the only deterministic failure; retry with smaller page batches

---

### 4.2 Gemini 2.0 Flash and Gemini 2.5 Flash

**Overall verdict:** Both models produced **identical outputs** on every page — no meaningful difference between them for this document type. Viable for clean typed appendix pages as a supplementary model. Not a replacement for olmOCR. Not fit for handwritten content.

**Strengths:**
- Good to excellent on typed message forms, movement orders, typed narrative pages
- Reasonable table structure preservation
- Generally faster than Pro

**Failure modes:**

| Failure | Page(s) | Severity | Example |
|---|---|---|---|
| `<think>` block leakage | All | Medium | Reasoning chain present in output — must be stripped post-processing |
| Truncation on dense multi-entry pages | 6 (doc p.7) | **High** | 6 Sep entry complete; entire 7 Sep entry (Loon Plage) dropped |
| Truncation on long narrative pages | 29 (CO's address) | Medium | Final 1.5 paragraphs lost |
| Truncation on sparse pages | 51 (signing) | Medium | Signature block at page bottom missed |
| Handwriting total failure | 32 | **Critical** | "Leen Cire des. thae Laucklan", "Euront pour Koon plage" — largely unintelligible |
| S→3 substitution | 23, 24+ | Medium | "SASK R" → "3ASK R", "SOUTH" → "30UTH", "GUARDS" → "GUARD3" |
| Z→S substitution | 26 | Medium | "PZ FAUST" → "PS FAUST" |
| I→J confusion | 17 | Medium | "I Sec" (Intelligence Section) → "J. See" |
| Redaction marker garbling | 21 | Low | "XXXXXXX" → "Kaxxkuka" |
| Complex dual-page form failure | 10 | **Critical** | Abandons after header; 44 named officers completely lost |
| Diagram/trace partial reads | 14 | Medium | Unit names split across lines; edge text garbled with no uncertainty flag |

**Recommendation:** Viable as a supplementary model for document classification and typed-content extraction only. Route handwritten pages and complex forms away from Flash.

---

### 4.3 Gemini 2.5 Pro

**Overall verdict:** Clear winner on typed content and best-performing general model tested to date. Significantly better than olmOCR on narrative diary pages, message forms, maps, and handwritten content. Requires mandatory post-processing to strip `<think>` blocks and embedded base64 data.

**Strengths:**
- Wins typed narrative diary entries — no paraphrase contamination, no hallucination
- Wins on maps and hand-drawn traces (only model that successfully extracts text from these)
- Best performance on handwritten narrative appendix (page 32)
- Correctly reads degraded/compressed typography better than any other model
- Gets original typos and spelling errors correctly (verbatim faithfulness)
- Gets superscript annotations, handwritten margin notes, signatures
- Gets French place names accurately ("Loon Plage" → correct on all attempts)

**Strongest pages vs olmOCR:**
- Page 4: Correct "General Crerar" where olmOCR hallucinated "General Montgomery"
- Page 5: Correct place names and diarist quotes where olmOCR paraphrased and garbled
- Page 6: Correct "roads were reported clear" where olmOCR reversed to "closed"
- Page 7: "25 pdrs fell short" correct where olmOCR produced "20 plrs fell asleep"
- Page 32 (handwritten): Correct "Loon Plage 1368", "MacLauchlan", "Borsbeek 782933"

**Failure modes:**

| Failure | Page(s) | Severity | Example |
|---|---|---|---|
| `<think>` block leakage | All | Medium | Longer and more verbose than Flash; must be stripped |
| Embedded base64 image data | 6, 7 | **High** | Hundreds of lines of `data:image/png;base64` embedded in Remarks column; file size explodes |
| HTML verbosity (`<br>` proliferation) | 4, 7 | Low | Dozens of empty `<br>` tags in table cells |
| Truncation on complex forms | 10, 16 | High | Field Return of Officers and Movement Order cut off mid-page (token limit) |
| Page 29 degradation | 29 | Medium | Dense two-column CO's address: "Docompris", "noathing" — quality below usual standard |

**Critical post-processing required:**
```python
import re

def clean_gemini_output(text):
    # Strip think blocks
    text = re.sub(r'<think>.*?</think>', '', text, flags=re.DOTALL)
    # Strip embedded base64 image data
    text = re.sub(r'data:image/[^;]+;base64,[A-Za-z0-9+/=\n]+', '[IMAGE]', text)
    # Strip excessive <br> padding in table cells
    text = re.sub(r'(\s*<br>\s*){5,}', '\n', text)
    return text.strip()
```

---

### 4.4 Qwen3-VL-30B

**Overall verdict: Disqualified. Do not use.**

Multiple distinct catastrophic failure modes make Qwen3-VL-30B unsuitable for any production role in this pipeline. Even its single strength (nominal roll extraction on page 10) is not sufficient to justify inclusion.

**Failure modes:**

| Failure | Pages | Severity | Example |
|---|---|---|---|
| Infinite loop / token flood | 1, 17, 22, 24, 25, 27+ | **Critical** | "Bde Bde Bde Bde..." repeated thousands of times; or `<tr><td>2</td></tr>` ×hundreds |
| Silent image placeholder failure | 14, 15, 19, 27, 28, 29, 31 | **Critical** | `<div class="image"><img/></div>` — zero text extracted. Affects ~40% of pages including clean printed pages |
| S→3 systematic substitution | 23+ | High | "GUARDS" → "GUARD3", "SCOT" → "3COT", "3 SASK R", "FU3 MR" |
| "LOON-PLACE" misread | 23, 32 | High | "LOON-PLAGE" → "LOON-PLACE" — wrong on the key Calgary Highlanders location throughout |
| Numeric errors on intel content | 26 | High | "559 GR 331 inf div" → "859 GR 531 inf div" — corrupts unit identification |

**Qwen's one notable strength:**
- Page 10 (Field Return of Officers nominal roll) — Qwen correctly extracted all 42 officer names with ranks, corps, and appointments. This is the most structurally complex table in the test and Qwen outperformed every other model on this specific page. However this does not offset the ~40% silent failure rate.

---

### 4.5 Claude Sonnet 4.6

**Status:** Successfully run via Anthropic API direct. Full 32-page comparison file exists (`step1_Calgary-Highlanders_War-Diary_Sep44_claude-4-6.md`). Formal page-by-page scoring against control test not yet completed.

**What we know from the full diary run:**
- Produces clean HTML tables for diary pages — same format as olmOCR
- Output structure consistent and correct: `## Page N` markers, HTML tables where applicable
- Image compression pipeline established: PDF → PNG (pdftoppm, 150 DPI) → JPEG (85% quality) → base64 → Anthropic API. JPEG compression required because PNG at 150 DPI is ~6-7 MB (over Anthropic's 5 MB image limit); JPEG 85% brings it to ~0.8-1.2 MB with no meaningful OCR detail loss.
- Token usage logged per page: ~$3.00/1M input, $15.00/1M output

**Technical parameters confirmed working:**
- Endpoint: `https://api.anthropic.com/v1/messages`
- Model ID: `claude-sonnet-4-6`
- Headers: `x-api-key`, `anthropic-version: 2023-06-01`
- Response path: `data["content"][0]["text"]`
- `max_tokens: 8192` (double the extraction step — vision work needs more headroom)

**Next step:** Formal page-by-page comparison of `step1_Calgary-Highlanders_War-Diary_Sep44_claude-4-6.md` against the Gemini Pro and olmOCR control test outputs.

---

### 4.6 Claude 3.7 Sonnet (via DeepInfra) — Complete Failure

All 32 pages returned `[OCR FAILED — HTTP error: 404 Client Error: Not Found for url: https://api.deepinfra.com/v1/openai/chat/completions]`.

**Root cause:** Anthropic models are not hosted on DeepInfra. The model ID `anthropic/claude-3-7-sonnet-latest` was passed to DeepInfra's OpenAI-compatible endpoint, which returned 404 for all requests.

**Resolution:** Rewrote `step1_ocr_wardiaries_claude-4-6.py` to call the Anthropic API directly. Do not attempt to use any Anthropic model via DeepInfra.

---

## 5. Cross-Model Rankings by Page Type

| Page Type | Best Model | Runner-up | Notes |
|---|---|---|---|
| Typed narrative diary entries | Gemini 2.5 Pro | Claude 4.6 (TBD) | olmOCR paraphrase/hallucination issues |
| Typed appendix forms | Gemini 2.5 Pro | olmOCR | Qwen S→3 errors on grey pages |
| Complex officer / rank nominal rolls | Qwen3 | Gemini 2.5 Pro | Qwen's only standout; truncation in Pro |
| Message forms (dense typed) | Gemini 2.5 Pro | Qwen3 | olmOCR not fully sampled |
| Artillery / DF task tables | Gemini 2.5 Pro | Qwen3 | Qwen swaps some entries |
| Hand-drawn maps / traces | Gemini 2.5 Pro | — | Qwen fails, olmOCR fails |
| Handwritten narrative | Gemini 2.5 Pro | olmOCR | Flash unintelligible; Qwen partially attempts |
| Printed formal documents | Gemini 2.5 Pro | — | olmOCR and Qwen both fail page 27 (Montgomery) |
| Illustrated / journal pages | Gemini 2.5 Pro | — | Qwen and olmOCR fail The Glen cover |
| Signing pages / simple forms | Gemini 2.5 Pro / Qwen | — | All models adequate |

---

## 6. Character Substitution Patterns Found

These substitution errors are scan/font artefacts and appear consistently across multiple models on degraded or grey-background pages:

| Substitution | Affected Models | Pages | Correction Rule |
|---|---|---|---|
| `S` → `3` | Gemini Flash, Qwen | 23, 24+ | Post-processing rule against known military vocabulary (SASK R, SCOTS, GUARDS, SOUTH, FUS MR) |
| `Z` → `S` | Gemini Flash | 26 | "PZ FAUST" → "PS FAUST" |
| `I` → `J` | Gemini Flash | 17 | "I Sec" → "J. See" |
| `F` → `P` | Qwen | 11 | "5 Cdn Fd Regt" → "5 Cdn Pd Regt" |
| `O` → `0` | Qwen | 8 | "CA(O)" → "CA(0)" |

**Recommendation:** Build a post-processing correction dictionary for known military unit names, abbreviations, and vocabulary. High-confidence substitutions can be corrected automatically; low-confidence ones should be flagged for human review.

---

## 7. Key Decisions Made

### OCR Architecture
- **claude-4-6 via Anthropic API** is the current production step 1 model for diary narrative pages
- **olmOCR via DeepInfra** is the fallback / comparison baseline
- Two-pass architecture (vision OCR → text extraction) is deliberate and correct — collapsing into one step would pay vision token rates for all field extraction

### Image Processing
- **150 DPI** is sufficient for this document type at typical war diary scan quality
- **JPEG 85% compression** is required before sending to Anthropic API (5 MB limit); no meaningful OCR detail loss at this quality level
- PNG at 150 DPI produces ~6-7 MB images — always exceeds Anthropic's limit

### Output Format
- **HTML tables** for diary pages (Place / Date / Hour / Summary / Remarks structure)
- **Plain text** for all other page types
- **`## Page N`** section headers in output MD — required by step 2 (viewer) and step 3 (extraction)

### Pipeline Routing (proposed)
- **Misc documents (~90% of pages):** Step 1 → Step 3 immediately. No human verification needed.
- **Narrative diary pages (~10% of pages):** Step 1 → Step 2 viewer → human verification → Step 3. Only verified text feeds the descendant report.

### Models Definitively Ruled Out
- **Qwen3-VL-30B:** Do not use. ~40% silent failure rate, infinite loops, systematic character errors.
- **Gemini Flash (either version):** Not for handwriting, not for complex forms.
- **Any Anthropic model via DeepInfra:** Returns 404. Use Anthropic API directly.

---

## 8. OCR Correction Experiment (Gemini Pro + Stacey Context)

**Test:** Attempted automated OCR correction by passing olmOCR output to Gemini 2.5 Pro along with contextual reference material (C.P. Stacey's *The Victory Campaign*).

**Pages tested:** 4, 5, 6, 7, 30, 32

**Result:** 0 corrections suggested across all 6 pages.

**Analysis:** One of two interpretations:
1. The olmOCR output on these pages was accurate enough that Gemini Pro found nothing to correct — a positive signal for olmOCR quality on the pages selected.
2. The correction prompt was insufficiently targeted to surface the known errors (paraphrasing, meaning reversals, hallucinations) that were identified in the manual comparison.

**Conclusion:** Automated correction via a second-pass LLM is not yet proven. Manual human review (via the OCR viewer) remains the quality gate for narrative pages.

---

## 9. Cost Comparison

Based on DeepInfra pricing and Anthropic API pricing as at May 2026, and typical token counts for this document type (~1,500 input tokens per page for text, plus ~1,200 tokens for image via vision models):

| Model | Provider | Input $/1M | Output $/1M | Relative Cost | Notes |
|---|---|---|---|---|---|
| olmOCR 2-7B | DeepInfra | ~$0.07 | ~$0.07 | 1× (baseline) | Cheapest. Consistent. |
| Gemini 2.0 Flash | DeepInfra | ~$0.15 | ~$0.60 | ~4× | Same output as 2.5 Flash |
| Gemini 2.5 Flash | DeepInfra | ~$0.15 | ~$0.60 | ~4× | No gain over 2.0 Flash |
| Qwen3-VL-30B | DeepInfra | ~$0.20 | ~$0.88 | ~6× | Disqualified |
| Gemini 2.5 Pro | DeepInfra | ~$1.25 | ~$10.00 | ~50× | Best accuracy; think blocks inflate token count |
| Claude Sonnet 4.6 | Anthropic | $3.00 | $15.00 | ~100× | Best structured output; no post-processing needed |

**For 150,000-page scale:**
- olmOCR at scale: very low cost; accuracy limitations are the constraint
- Gemini Pro at full corpus: materially expensive; hybrid approach preferred
- Claude 4.6 at full corpus: highest cost but may be justified for narrative diary pages (only ~10% of pages)

**Recommended approach at scale:** olmOCR for misc documents (90%); Claude 4.6 for narrative diary pages (10%). This concentrates the expensive model where accuracy matters most for the commercial product.

---

## 10. Known Open Questions

1. **Formal Claude 4.6 scoring:** `step1_Calgary-Highlanders_War-Diary_Sep44_claude-4-6.md` exists but has not been formally scored against the control test comparison. This should be the next evaluation step.

2. **Step 3 model choice:** Step 3 (extraction) currently uses `meta-llama/Meta-Llama-3.1-70B-Instruct` (olmOCR variant) and `claude-sonnet-4-6` (claude variant). Llama 70B has not been formally evaluated for grid reference extraction accuracy. Bad grids silently poison PostGIS downstream. Formal comparison of Llama vs Claude on `_positions.json` output quality is pending.

3. **Handwriting routing:** No model has been confirmed for production-quality handwriting transcription. Gemini 2.5 Pro is the best tested but has truncation issues on complex pages. A dedicated handwriting model (e.g., Google Document AI, Azure Form Recognizer) has not been evaluated.

4. **Complex form failures:** Field Return of Officers (nominal roll) causes truncation in Gemini Pro and timeouts in olmOCR. Qwen handled it but is otherwise disqualified. No production-viable solution confirmed for this page type.

5. **`<think>` block suppression:** Gemini Pro's think blocks can be stripped post-processing, but they also inflate token counts and costs. Testing a `thinkingBudget: 0` parameter (if available via DeepInfra) is pending.

---

*Document covers: olmOCR 2-7B, Gemini 2.0 Flash, Gemini 2.5 Flash, Gemini 2.5 Pro, Qwen3-VL-30B, Claude 3.7 Sonnet (failed), Claude Sonnet 4.6*
*Test corpus: Calgary Highlanders War Diary Sep 44, 32-page control test*
*Pipeline scripts: `step1_ocr_wardiaries_claude-4-6.py`, `step1_ocr_wardiaries_olmocr.py`, `ocr_model_comparison.py`, `ocr_model_comparison_anthropic.py`*

