# War Diary OCR — Lessons Learned
*Project '44 | Running log of model tests, findings, and decisions*
*Control test: Calgary Highlanders War Diary Sep 44 (32-page control PDF)*

---

## Table of Contents

1. [Models Tested](#1-models-tested)
2. [Control Test Setup](#2-control-test-setup)
3. [Model-by-Model Findings](#3-model-by-model-findings)
    - [olmOCR (allenai/olmOCR-2-7B-1025)](#31-olmocr)
    - [Gemini 2.5 Flash](#32-gemini-25-flash)
    - [Gemini 2.5 Pro](#33-gemini-25-pro)
    - [Qwen3-VL-30B](#34-qwen3-vl-30b)
    - [Claude Sonnet 4.6](#35-claude-sonnet-46)
4. [Cross-Model Comparison by Page Type](#4-cross-model-comparison-by-page-type)
5. [Known Failure Modes (All Models)](#5-known-failure-modes-all-models)
6. [Cost Analysis](#6-cost-analysis)
7. [Architecture Decisions](#7-architecture-decisions)
8. [Key Principles](#8-key-principles)
9. [Open Questions & Next Tests](#9-open-questions--next-tests)

---

## 1. Models Tested

| Model | Provider | API Endpoint | Vision | Notes |
|-------|----------|-------------|--------|-------|
| `allenai/olmOCR-2-7B-1025` | DeepInfra | OpenAI-compatible | ✅ | Fine-tuned specifically for document OCR |
| `google/gemini-2.5-flash` / `google/gemini-2.0-flash` | DeepInfra | OpenAI-compatible | ✅ | Both versions produced identical output |
| `google/gemini-2.5-pro` | DeepInfra | OpenAI-compatible | ✅ | Best general-purpose performer tested |
| `Qwen/Qwen3-VL-30B-A3B-Instruct` | DeepInfra | OpenAI-compatible | ✅ | Disqualified — critical failure modes |
| `claude-sonnet-4-6` | Anthropic API (direct) | `api.anthropic.com` | ✅ | Best overall performer; not available via DeepInfra |
| `anthropic/claude-3-7-sonnet-latest` | DeepInfra | OpenAI-compatible | ❌ | Vision/image requests return 404 via DeepInfra |

> **Note:** `claude-sonnet-4-6` is only available through the Anthropic API directly. DeepInfra's OpenAI-compatible endpoint does not support multimodal input for Claude models. Claude Sonnet 4 is on DeepInfra as `anthropic/claude-4-sonnet` but has not been tested.

---

## 2. Control Test Setup

**Source document:** Calgary Highlanders War Diary, September 1944 — 32-page control PDF  
**Pages tested:** 7–32 (skipping admin/cover pages)  
**Image format:** PNG rendered at 150 DPI via Poppler, then compressed to JPEG 85% to stay under Anthropic's 5MB per-image limit  
**API pattern:** All DeepInfra models use the same OpenAI-compatible endpoint with base64-encoded image_url in the messages array

**Document types in the control test:**

| Page(s) | Document Type | Notes |
|---------|--------------|-------|
| 7–8, 11–13, 16, 22–26 | Typed narrative diary entries (HTML table format) | Core diary — Place / Date / Hour / Summary / Remarks columns |
| 9–10 | Field Return of Officers / Other Ranks | Complex dual-page nominal roll form, densest page in test |
| 14–15, 17–18, 20–21 | Movement orders, typed appendices | Mixed typed content with occasional handwritten annotation |
| 19 | Hand-drawn trace / map overlay | Sparse — report line names with grid markers |
| 27 | Printed formal letter (Montgomery) | Clean typeset, should be easy |
| 28–29 | C.O.'s address (typed narrative) | Long-form typed, degraded scan |
| 30–31 | Regimental journal pages (Tumbleweed letter) | Typed narrative, page 30 was the loop-trigger for olmOCR |
| 32 | Handwritten narrative appendix | Cursive handwriting — hardest page in the test |

---

## 3. Model-by-Model Findings

### 3.1 olmOCR

**Model:** `allenai/olmOCR-2-7B-1025`  
**Cost (control test):** ~$0.000388/page  
**Full corpus estimate (150,000 pages):** ~$60

#### Strengths
- Designed specifically for document OCR — performs best on clean typed text
- Reliable on typed narrative diary pages (the main diary HTML table entries)
- Good on handwritten content relative to general VLMs — best of all tested models on page 32 (handwritten appendix)
- No base64 image dumps, no think blocks, no padding loops on most pages
- Predictable failure modes — when it fails, it fails consistently enough to detect
- Output is clean, directly usable HTML tables

#### Failures & Known Issues

| Issue | Page | Severity | Description |
|-------|------|----------|-------------|
| Hallucination from training priors | 4 | **Critical** | Substitutes "General Montgomery" for "General Crerar" — does not misread the letters, pulls the statistically more probable name from training weights. Completely plausible, undetectable without ground truth. |
| Repetition loop | 30 | **Critical** | Enters infinite repetition loop starting at the Tumbleweed letter ("A follow came out...") and repeats for the rest of its token budget. Same failure mode as Qwen but triggered less frequently. |
| Paraphrase contamination | Multiple | High | Rewrites sentences in its own words rather than transcribing verbatim. Not a hallucination but not a transcription either — corrupts the historical record. |
| Handwriting misreads | 32 | Medium | "Evacout from Leuven Plage" vs correct "Enroute from Loon Plage". Key place name wrong, though close enough to recognise. |
| "Lieut" → "Heard" | Multiple | Medium | Consistent misread of the abbreviation "Lieut" in cursive as "Heard". Pattern is consistent enough to post-process. |

#### Verdict
olmOCR was the right initial choice vs Adobe OCR — it is meaningfully better on structured historical documents. But its ceiling is now clear. The Montgomery/Crerar hallucination and the loop failure are architectural limitations of a 7B model, not tuning problems. No prompt change will fix them.

---

### 3.2 Gemini 2.5 Flash

**Models:** `google/gemini-2.5-flash`, `google/gemini-2.0-flash`  
**Both versions produced identical output on all 32 pages.**

#### Strengths
- Good to excellent on typed appendix pages — message forms, movement orders
- Handles moderately complex forms
- Fast

#### Failures & Known Issues

| Issue | Page(s) | Severity | Description |
|-------|---------|----------|-------------|
| Dense page truncation | 6 (doc p.7) | High | Truncates after the 6 Sep entry, drops the entire 7 Sep entry including Loon Plage operational passage. |
| Sparse page truncation | 51 | Medium | Misses entire signature block at bottom of page — processes empty space, stops before reaching content. |
| Long narrative truncation | 29 | Medium | Good accuracy but drops final 1.5 paragraphs. Truncation pattern is not purely length-dependent. |
| Complete failure on complex forms | 10 | High | Abandons transcription of the Field Return of Officers after the header — no table data produced. |
| Handwriting — fundamental failure | 32 | **Critical** | Largely unintelligible on cursive handwriting. "Leen Cire des. thae Laucklan", "Euront pour Koon plage". Not fit for purpose on handwritten pages. |
| `<think>` block leakage | All | Medium | Reasoning chains appear in output throughout. Requires post-processing to strip. |
| `I`/`J` confusion | 17 | Medium | "I Sec" (Intelligence Section) misread as "J. See". Pattern likely recurs across corpus. |
| `S`→`3` substitution | 23, 24+ | Medium | Systematic: "3ASK R" (SASK R), "30UTH" (SOUTH), "GUARD3" (GUARDS). Amenable to post-processing correction for known unit names. |
| `Z`→`S` substitution | 26 | Medium | "PS FAUST" → "PZ FAUST" (Panzerfaust). Corrupts weapons identification. |

#### Handwriting Ranking (page 32)
**olmOCR > Claude Sonnet 4.6 > Flash** — Flash is not fit for purpose on handwritten pages.

#### Verdict
Viable for typed appendix pages. Not viable as primary OCR engine. Potential role: supplementary typed-content extraction or document classification, not as olmOCR replacement.

---

### 3.3 Gemini 2.5 Pro

**Model:** `google/gemini-2.5-pro`  
**Cost (input/output):** $1.25 / $10.00 per 1M tokens  
**Output volume:** ~4,144 lines for 32 pages — ~3× olmOCR, driven by think-block overhead and verbose HTML

#### Strengths
- Best general-purpose performer before Claude 4.6 was tested
- Correct on "General Crerar" — no Montgomery substitution
- No repetition loops (page 30 complete)
- Gets the Montgomery letter (page 27) correct in full
- Handles maps and traces — gets all report line names on page 19
- Best handwriting performance of the non-Claude models
- Strong on complex forms — message forms, DF task tables, movement orders

#### Failures & Known Issues

| Issue | Page(s) | Severity | Description |
|-------|---------|----------|-------------|
| Base64 image injection | 6, 7 (Remarks column) | High | Embeds `<img src="data:image/png;base64,...">` garbage strings in Remarks column when it encounters handwritten initials/circled marks. Post-processing strip required. |
| Token exhaustion on complex forms | 9, 10 | Medium | Page 9 cuts off mid-table in the tradesmen section. Fix: increase `max_tokens` to 8192. Not a capability failure. |
| `<think>` block leakage | All | Medium | Verbose reasoning chains throughout output. Requires post-processing strip. |

#### Verdict
Significantly better than olmOCR on typed content. The base64 injection and token exhaustion issues are tractable post-processing problems, not fundamental model failures. Recommended as a replacement for olmOCR on typed content if cost is acceptable.

---

### 3.4 Qwen3-VL-30B

**Model:** `Qwen/Qwen3-VL-30B-A3B-Instruct`  
**Status: DISQUALIFIED — do not use in production**

#### Failures & Known Issues

| Issue | Page(s) | Severity | Description |
|-------|---------|----------|-------------|
| Infinite repetition loop | Multiple | **Critical** | Triggered on ~40% of complex pages. Repeats the same phrase endlessly for the rest of the token budget. |
| Silent failure / empty output | Multiple | **Critical** | ~40% silent-fail rate — produces empty or near-empty output on complex pages without any error signal. |
| Infinite image loop | 17, 19 | **Critical** | On map/trace pages: outputs hundreds of `<div class="image"><img/></div>` elements. |
| Systematic `S`→`3` substitution | 23, 25+ | High | "3ASK R", "30UTH", "GUARD3". More severe than Flash. |
| Numeric digit errors | 26 | High | "404 BLANDES" vs "484", "559 GR 331" vs "531 inf div". Corrupts unit identification in intelligence reports. |
| Handwriting | 32 | High | Partially attempts but degrades badly on cursive. Below olmOCR. |
| Padding loop | 22, 25 | Medium | Outputs hundreds of empty `<tr><td></td></tr>` rows after content ends. |

**One genuine strength:** Nominal roll forms (page 10 — Field Return of Officers). Qwen completed the 44-officer roll where Flash completely failed. This single strength is not sufficient to justify inclusion given the catastrophic failures elsewhere.

#### Verdict
**Do not use.** Infinite loops, ~40% silent-fail rate, and systematic character substitutions make Qwen3-VL-30B unsuitable for any production role.

---

### 3.5 Claude Sonnet 4.6

**Model:** `claude-sonnet-4-6`  
**API:** Anthropic direct (`api.anthropic.com`) — not available via DeepInfra  
**Image format:** 150 DPI PNG → JPEG 85% compression (to stay under Anthropic's 5MB image limit)  
**max_tokens:** 8192 (required — pages 9 and 10 exceeded 4096)  
**Cost:** $3.00 / $15.00 per 1M tokens (input/output)  
**Control test cost (pages 7–32):** $0.49 total, avg $0.0195/page  
**Full corpus estimate (150,000 pages):** $2,300–$2,900

#### Strengths
- **Best overall performer** — only model to pass all key validation tests cleanly
- No repetition loops — page 30 (Tumbleweed letter) completes correctly
- No hallucination substitution — "General Crerar" correct on page 4
- No base64 image injection — clean HTML tables throughout
- Completes page 10 Field Return of Officers (44-officer nominal roll) in full
- Gets "Loon Plage" correct on page 32 (handwritten appendix)
- Gets Montgomery letter (page 27) complete and accurate
- Gets all report line names on trace map (page 19)
- Clean output format — no think blocks, no padding artifacts, no inline CSS

#### Failures & Known Issues

| Issue | Page(s) | Severity | Description |
|-------|---------|----------|-------------|
| Token truncation on dense forms | 9, 10 | Medium | Truncated at 4096 default `max_tokens`. Fixed by setting 8192. Not a capability failure. |
| Minor name misreads (handwriting) | 32 | Low | "D.F. MacLauchlan" vs "D.G.", "Hutchell" vs "Mitchell", "Wiegils" vs "Megill". Operational content intact. |
| Superscript misreads | 7 | Low | "wome 25 pdrs fell short" → "some"; "morep at 2317" → missing superscript "hrs". |

#### Key Validation Checks (Pass/Fail)

| Check | Source of Failure on Other Models | Result |
|-------|----------------------------------|--------|
| Page 4 — "General Crerar" | olmOCR hallucination | ✅ Pass |
| Page 10 — 44-officer nominal roll complete | Flash abandons after header | ✅ Pass |
| Page 19 — trace map report lines | Qwen infinite image loop | ✅ Pass |
| Page 27 — Montgomery letter complete | olmOCR fails | ✅ Pass |
| Page 30 — no repetition loop | olmOCR and Qwen loop | ✅ Pass |
| Page 32 — "Loon Plage" in header | olmOCR "Leuven", Flash "Koon" | ✅ Pass |
| No base64 injection in Remarks | Gemini Pro failure | ✅ Pass |
| No `<think>` block leakage | Flash and Pro failure | ✅ Pass |

#### Verdict
**Best model tested. Current recommendation for production OCR.** The cost ($2,300–$2,900 for full corpus) is within budget ceiling. The only structural issue is that it requires direct Anthropic API access — it cannot be run via DeepInfra's OpenAI-compatible endpoint for vision tasks.

---

## 4. Cross-Model Comparison by Page Type

| Page Type | Winner | Runner-up | Avoid |
|-----------|--------|-----------|-------|
| Typed narrative diary entries | Claude 4.6 | Gemini Pro | Qwen |
| Typed appendix forms | Claude 4.6 | Gemini Pro | Qwen |
| Complex dual-page forms (nominal rolls) | Claude 4.6 | Qwen (only strength) | Flash |
| Message forms (dense typed, grey paper) | Claude 4.6 | Gemini Pro | Qwen (S→3 errors) |
| Artillery / DF task tables | Claude 4.6 | Gemini Pro | Qwen (swaps entries) |
| Hand-drawn maps / traces | Claude 4.6 | Gemini Pro | Qwen (image loop) |
| Handwritten narrative | olmOCR | Claude 4.6 | Flash (critical failure) |
| Printed formal documents | Claude 4.6 | Gemini Pro | olmOCR |
| Regimental journal / illustrated pages | Claude 4.6 | Gemini Pro | olmOCR (loops) |
| Signing pages / simple forms | All adequate | — | — |

---

## 5. Known Failure Modes (All Models)

### 5.1 Repetition / Infinite Loop
**Affected models:** olmOCR (rare), Qwen3 (frequent — ~40% of complex pages)  
**Trigger:** Ambiguous multi-column text, complex layouts, or anything that breaks the model's parsing flow  
**Symptom:** Model gets stuck on a phrase and repeats it for the rest of the token budget  
**Detection:** Output length disproportionately long; repeated n-grams in output  
**Fix:** Post-processing loop detection (count repeated 10-gram occurrences); flag for human review

### 5.2 Hallucination from Training Priors
**Affected models:** olmOCR confirmed; presumed risk on all small/medium models  
**Example:** "General Montgomery" substituted for "General Crerar" — statistically more probable completion pulled from training weights  
**Danger:** Output looks completely plausible; undetectable without ground truth or image comparison  
**Mitigation:** Use larger models with stronger instruction-following; provide Stacey/reference context to enable contradiction detection

### 5.3 Token Truncation
**Affected models:** Flash (systematic), Pro (on complex forms), Claude 4.6 (at default 4096 limit)  
**Trigger:** Dense pages, long narrative entries, complex forms with many rows  
**Fix:** Set `max_tokens: 8192` for all models; detect truncation by checking if output ends mid-sentence

### 5.4 Base64 Image Injection
**Affected models:** Gemini Pro  
**Trigger:** Handwritten marks in Remarks column (circled initials, check marks)  
**Symptom:** `<img src="data:image/png;base64,...">` garbage strings embedded in HTML table cells  
**Fix:** Post-processing regex strip of any `<img>` tags containing base64 data

### 5.5 `<think>` Block Leakage
**Affected models:** Gemini Flash, Gemini Pro  
**Symptom:** Model reasoning chains appear in output (verbose internal monologue about how to interpret the page)  
**Fix:** Post-processing strip of `<think>...</think>` blocks

### 5.6 Systematic Character Substitution on Degraded Scans
**Affected models:** Qwen3 (severe), Flash (moderate)  
**Known substitution clusters:**
- `S` → `3` (e.g., "3ASK R" for "SASK R", "GUARD3" for "GUARDS")
- `Z` → `S` (e.g., "PS FAUST" for "PZ FAUST")
- `I` → `J` (e.g., "J. See" for "I Sec")
  **Fix:** Post-processing correction dictionary for known military vocabulary and unit names

### 5.7 Handwriting — Fundamental Failure
**Affected models:** Gemini Flash (critical), Qwen3 (severe)  
**Safe models for handwriting:** olmOCR (best), Claude Sonnet 4.6 (good)  
**Recommendation:** Route handwritten pages to olmOCR or Claude; do not use Flash or Qwen

---

## 6. Cost Analysis

| Model | Input (per 1M tokens) | Output (per 1M tokens) | Avg cost/page | Full corpus (150k pages) |
|-------|-----------------------|------------------------|---------------|--------------------------|
| olmOCR (DeepInfra) | ~$0.20 | ~$0.20 | ~$0.00039 | **~$60** |
| Gemini Flash (DeepInfra) | $0.10 | $0.40 | ~$0.003 | **~$450** |
| Gemini Pro (DeepInfra) | $1.25 | $10.00 | ~$0.018 | **~$2,700** |
| Claude Sonnet 4.6 (Anthropic) | $3.00 | $15.00 | ~$0.0195 | **~$2,900** |
| Qwen3-VL-30B (DeepInfra) | Variable | Variable | N/A | **Disqualified** |

**Budget ceiling:** $2,000–$5,000 (confirmed acceptable)  
**olmOCR was the original baseline** — $60 for the full corpus, essentially unlimited for prototype work with $20 loaded  
**Claude Sonnet 4.6** sits comfortably within budget at ~$2,900 for full corpus re-OCR

### Image Rendering
- PDF pages rendered at **150 DPI** via Poppler (`pdftoppm`)
- PNG output compressed to **JPEG 85%** to stay under Anthropic's 5MB per-image limit
- Quality tested at page 32 (handwriting) — 150 DPI JPEG 85% produced the Claude results above
- No quality regression observed vs PNG at this compression level

---

## 7. Architecture Decisions

### 7.1 Two-Pass Pipeline is Correct
**Question raised:** Is it wasteful to OCR in step 1 then run an LLM again in step 3?  
**Answer:** No. Step 1 is vision work (image → text). Step 3 is reasoning work (text → structured JSON). Different tasks, different models, different costs.
- Step 1 sends PNG images — expensive per page, requires visual understanding
- Step 3 reads clean OCR text — cheap input, no image involved
- Keeping them separate means if the extraction schema changes, you re-run step 3 only against already-OCR'd MD files. You never re-OCR.

### 7.2 Why Not Collapse Into One Step
Collapsing step 1 and step 3 would mean paying vision token rates for every field extraction on every page. It would also make re-runs of the extraction schema (which will happen as the project matures) require re-OCRing everything. The two-pass is the economical choice.

### 7.3 Gemini Pro vs olmOCR Hybrid
Three options were evaluated:

**Option A — Gemini Pro everywhere**  
Run Gemini Pro on all pages. Post-process to strip `<think>` blocks and base64 injections. Best accuracy, moderate cost.

**Option B — Hybrid routing**
- Classifier identifies page type first
- Gemini Pro (or Claude) for: diary narrative pages, message forms, typed appendices, maps/traces
- olmOCR as fallback for pages where Gemini Pro is known to inject base64 (handwritten marks in Remarks column)

**Option C — olmOCR + LLM correction pass**  
Run olmOCR first (fast, cheap). Flag low-confidence pages. LLM correction pass using Gemini Pro or Claude with Stacey/reference context injected.
- Input to correction is text-only (no image) — costs ~$1.25/1M tokens, not vision rates
- Stacey's Victory Campaign (~330,000 tokens) can be injected in full given Gemini Pro's 1M context window
- Correction model can catch loops, garbled text, broken sentences — but cannot catch plausible hallucinations (e.g., Montgomery for Crerar) without seeing the image

**Current decision:** Claude Sonnet 4.6 directly is the cleanest option given it outperforms the hybrid on all page types tested.

### 7.4 pgvector over Chroma DB
Deliberate architectural choice. Chroma is a pure vector store — semantic similarity only. pgvector inside Postgres enables a single SQL query combining:
- Semantic similarity (vector search)
- Date range filters
- Unit ID filters
- PostGIS spatial queries (`ST_DWithin` — "find events within 5km of this grid reference")

This spatial query capability is the architectural moat. Chroma cannot do this. Adding PostGIS to a Chroma pipeline would require a separate system and application-layer joins.

### 7.5 Document Classification
Two-pass classification approach:

**Pass 1 — Free, no API (OCR quality scoring):**  
Score each page's OCR output for alpha-ratio and recognisable text patterns. Classifies ~80% of pages without any API call.

**Pass 2 — Vision LLM on ambiguous pages (~20%):**  
Llama 3.2 Vision or Qwen2.5-VL on DeepInfra. One API call per ambiguous page, tiny output (classification label + confidence JSON).  
Estimated cost: $2–5 for full 150,000-page corpus.

---

## 8. Key Principles

- **Sources problem vs prompt problem:** Low-quality OCR output from war diary narrative pages is partly a sources problem — the diary itself is officer-centric. The solution is broader document type coverage (patrol reports, sitreps, message logs, Part 1 Orders), not better prompting.
- **olmOCR hallucinations cannot be fixed by prompting.** A 7B OCR-fine-tuned model does not reliably follow "transcribe verbatim" instructions when it encounters statistically probable alternative completions.
- **Handwriting routing is essential.** Flash and Qwen are not fit for purpose on cursive handwritten pages. olmOCR and Claude 4.6 must handle these.
- **Bad grids silently poison PostGIS.** Extraction accuracy in step 3 matters more than cost — a wrong grid reference goes into the database looking correct.
- **The viewer MD and the JSON outputs serve different consumers.** Viewer MD = display artifact for Jonesy QA and Project '44 frontend. Step 3 JSONs = database inputs. Both are necessary and neither replaces the other.
- **Transkribus is not viable at scale.** ~€10,000–12,000 for 150,000 pages vs ~$60 for olmOCR or ~$2,900 for Claude. The accuracy premium does not justify the cost delta.

---

## 9. Open Questions & Next Tests

| Question | Priority | Notes |
|----------|----------|-------|
| Claude 4.6 vs Llama 70B for step 3 extraction | High | Step 3 currently uses Llama for narrative/position extraction. Claude likely more accurate on ambiguous table rows and grid references. Run same diary pages through both and compare `_positions.json` output. |
| `anthropic/claude-4-sonnet` via DeepInfra for vision | Medium | Untested — Claude 4 (not 4.6) is available on DeepInfra. May support image input unlike Claude 3.7 Sonnet. Would enable single-API approach if it matches Claude 4.6 quality. |
| LLM correction pass on olmOCR output | Medium | Gemini Pro or Claude as post-processor on olmOCR text, with Stacey context injected. Can catch loops and garbled text but not plausible hallucinations. Worth quantifying the accuracy delta vs direct Claude OCR. |
| 100 DPI vs 150 DPI JPEG 85% | Low | Test page 32 (handwriting) at 100 DPI PNG vs current 150 DPI JPEG 85%. Determine if quality regression is measurable. 100 DPI PNG would remove Pillow dependency and eliminate compression artifacts. |
| Transkribus small-batch test | Low | Test 50 handwritten appendix pages to quantify accuracy vs olmOCR. Not viable at corpus scale but useful data point for handwriting quality ceiling. |