testin out different ocr outputs

notes/ocr-viewer-prompt.md

@@ -1,46 +0,0 @@
-# P44 OCR Viewer — Copilot Build Prompt
-
-## Context
-A desktop OCR review tool for verifying war diary OCR output page by page.
-Single HTML file, no build step, runs by double-clicking.
-
-## If Copilot times out — build in this order:
-
-### Step 1: Shell + layout
-> Build the shell of the P44 OCR Viewer. Two panels side by side using CSS Grid.
-> Dark background (#1a1a2e), gold accent (#c8a84b). Left panel: image display with
-> scroll. Right panel: scrollable text area. Top toolbar with three file inputs:
-> (1) page image, (2) OCR text file, (3) word-position JSON sidecar. No interactivity
-> yet — just layout and file loading that populates each panel.
-
-### Step 2: Word overlay on image
-> Add a word bounding box overlay layer to the left panel. The JSON sidecar format is:
-> [{ "word": "string", "id": "w001", "bbox": { "x": 0, "y": 0, "w": 50, "h": 20 } }]
-> Render transparent absolutely-positioned divs over the image for each word,
-> using the bbox coordinates scaled to the rendered image dimensions.
-> Each div gets a data-word-id attribute matching the JSON id.
-
-### Step 3: Word spans in text panel
-> Parse the OCR text in the right panel and wrap each whitespace-delimited token
-> in a <span> with a data-word-id attribute. IDs are assigned sequentially (w001,
-> w002...) matching the order in the JSON sidecar.
-
-### Step 4: Bidirectional hover highlight
-> On hover over any word span (right) or bounding box div (left), highlight the
-> matching element on the other side with a yellow background/border. Use the
-> data-word-id to find the match. Clear highlight on mouseout.
-
-### Step 5: Load example button
-> Add a "Load example" button to the toolbar that pre-populates all three panels
-> with bundled inline sample data (a small image placeholder, 2-3 lines of sample
-> OCR text, and a matching JSON sidecar with ~10 words). Tool should be fully
-> demonstrable without real files.
-
-## Full one-shot prompt (use if Copilot has capacity)
-Build a desktop OCR review tool called P44 OCR Viewer for reviewing war diary OCR output. The interface has two panels side by side:
-Left panel — the source document image. Load a JPEG or PNG page scan. Render it at full panel height, scrollable. Each word in the image should be represented by a transparent, hoverable bounding box overlay (coordinates come from a JSON sidecar file, format: [{ "word": "string", "id": "w001", "bbox": { "x": 0, "y": 0, "w": 50, "h": 20 } }]).
-Right panel — the OCR text output. Load the corresponding plain text or Markdown file. Each word in the text should be individually wrapped in a <span> with a matching data-word-id attribute that maps to the same ID in the JSON sidecar.
-Hover behaviour: Hovering over a word on either side highlights the corresponding word on the other side — a yellow highlight on the image bounding box overlay, and a yellow background on the text span. Bidirectional. Highlight clears on mouse-out.
-File loading: A toolbar at the top has three file inputs: (1) Page image, (2) OCR text file, (3) Word-position JSON sidecar. Files are loaded locally, no server required. A "Load example" button pre-populates with bundled sample data so the tool works without real data.
-Stack: Single HTML file, vanilla JS, no build step required. Designed to run by double-clicking — no localhost server, no npm. All layout via CSS Grid. Colour scheme: dark background (#1a1a2e), muted gold accent (#c8a84b) consistent with a WWII archival aesthetic.
-The word-alignment JSON sidecar format is designed to be produced by olmOCR (Allen Institute), which outputs bounding box data alongside transcribed text. The viewer is the QA interface a historian uses to verify OCR accuracy page by page.

notes/wardiary_claude_lessons-learned.md

@@ -0,0 +1,381 @@
+# War Diary OCR — Lessons Learned
+*Project '44 | Running log of model tests, findings, and decisions*
+*Control test: Calgary Highlanders War Diary Sep 44 (32-page control PDF)*
+
+---
+
+## Table of Contents
+
+1. [Models Tested](#1-models-tested)
+2. [Control Test Setup](#2-control-test-setup)
+3. [Model-by-Model Findings](#3-model-by-model-findings)
+    - [olmOCR (allenai/olmOCR-2-7B-1025)](#31-olmocr)
+    - [Gemini 2.5 Flash](#32-gemini-25-flash)
+    - [Gemini 2.5 Pro](#33-gemini-25-pro)
+    - [Qwen3-VL-30B](#34-qwen3-vl-30b)
+    - [Claude Sonnet 4.6](#35-claude-sonnet-46)
+4. [Cross-Model Comparison by Page Type](#4-cross-model-comparison-by-page-type)
+5. [Known Failure Modes (All Models)](#5-known-failure-modes-all-models)
+6. [Cost Analysis](#6-cost-analysis)
+7. [Architecture Decisions](#7-architecture-decisions)
+8. [Key Principles](#8-key-principles)
+9. [Open Questions & Next Tests](#9-open-questions--next-tests)
+
+---
+
+## 1. Models Tested
+
+| Model | Provider | API Endpoint | Vision | Notes |
+|-------|----------|-------------|--------|-------|
+| `allenai/olmOCR-2-7B-1025` | DeepInfra | OpenAI-compatible | ✅ | Fine-tuned specifically for document OCR |
+| `google/gemini-2.5-flash` / `google/gemini-2.0-flash` | DeepInfra | OpenAI-compatible | ✅ | Both versions produced identical output |
+| `google/gemini-2.5-pro` | DeepInfra | OpenAI-compatible | ✅ | Best general-purpose performer tested |
+| `Qwen/Qwen3-VL-30B-A3B-Instruct` | DeepInfra | OpenAI-compatible | ✅ | Disqualified — critical failure modes |
+| `claude-sonnet-4-6` | Anthropic API (direct) | `api.anthropic.com` | ✅ | Best overall performer; not available via DeepInfra |
+| `anthropic/claude-3-7-sonnet-latest` | DeepInfra | OpenAI-compatible | ❌ | Vision/image requests return 404 via DeepInfra |
+
+> **Note:** `claude-sonnet-4-6` is only available through the Anthropic API directly. DeepInfra's OpenAI-compatible endpoint does not support multimodal input for Claude models. Claude Sonnet 4 is on DeepInfra as `anthropic/claude-4-sonnet` but has not been tested.
+
+---
+
+## 2. Control Test Setup
+
+**Source document:** Calgary Highlanders War Diary, September 1944 — 32-page control PDF  
+**Pages tested:** 7–32 (skipping admin/cover pages)  
+**Image format:** PNG rendered at 150 DPI via Poppler, then compressed to JPEG 85% to stay under Anthropic's 5MB per-image limit  
+**API pattern:** All DeepInfra models use the same OpenAI-compatible endpoint with base64-encoded image_url in the messages array
+
+**Document types in the control test:**
+
+| Page(s) | Document Type | Notes |
+|---------|--------------|-------|
+| 7–8, 11–13, 16, 22–26 | Typed narrative diary entries (HTML table format) | Core diary — Place / Date / Hour / Summary / Remarks columns |
+| 9–10 | Field Return of Officers / Other Ranks | Complex dual-page nominal roll form, densest page in test |
+| 14–15, 17–18, 20–21 | Movement orders, typed appendices | Mixed typed content with occasional handwritten annotation |
+| 19 | Hand-drawn trace / map overlay | Sparse — report line names with grid markers |
+| 27 | Printed formal letter (Montgomery) | Clean typeset, should be easy |
+| 28–29 | C.O.'s address (typed narrative) | Long-form typed, degraded scan |
+| 30–31 | Regimental journal pages (Tumbleweed letter) | Typed narrative, page 30 was the loop-trigger for olmOCR |
+| 32 | Handwritten narrative appendix | Cursive handwriting — hardest page in the test |
+
+---
+
+## 3. Model-by-Model Findings
+
+### 3.1 olmOCR
+
+**Model:** `allenai/olmOCR-2-7B-1025`  
+**Cost (control test):** ~$0.000388/page  
+**Full corpus estimate (150,000 pages):** ~$60
+
+#### Strengths
+- Designed specifically for document OCR — performs best on clean typed text
+- Reliable on typed narrative diary pages (the main diary HTML table entries)
+- Good on handwritten content relative to general VLMs — best of all tested models on page 32 (handwritten appendix)
+- No base64 image dumps, no think blocks, no padding loops on most pages
+- Predictable failure modes — when it fails, it fails consistently enough to detect
+- Output is clean, directly usable HTML tables
+
+#### Failures & Known Issues
+
+| Issue | Page | Severity | Description |
+|-------|------|----------|-------------|
+| Hallucination from training priors | 4 | **Critical** | Substitutes "General Montgomery" for "General Crerar" — does not misread the letters, pulls the statistically more probable name from training weights. Completely plausible, undetectable without ground truth. |
+| Repetition loop | 30 | **Critical** | Enters infinite repetition loop starting at the Tumbleweed letter ("A follow came out...") and repeats for the rest of its token budget. Same failure mode as Qwen but triggered less frequently. |
+| Paraphrase contamination | Multiple | High | Rewrites sentences in its own words rather than transcribing verbatim. Not a hallucination but not a transcription either — corrupts the historical record. |
+| Handwriting misreads | 32 | Medium | "Evacout from Leuven Plage" vs correct "Enroute from Loon Plage". Key place name wrong, though close enough to recognise. |
+| "Lieut" → "Heard" | Multiple | Medium | Consistent misread of the abbreviation "Lieut" in cursive as "Heard". Pattern is consistent enough to post-process. |
+
+#### Verdict
+olmOCR was the right initial choice vs Adobe OCR — it is meaningfully better on structured historical documents. But its ceiling is now clear. The Montgomery/Crerar hallucination and the loop failure are architectural limitations of a 7B model, not tuning problems. No prompt change will fix them.
+
+---
+
+### 3.2 Gemini 2.5 Flash
+
+**Models:** `google/gemini-2.5-flash`, `google/gemini-2.0-flash`  
+**Both versions produced identical output on all 32 pages.**
+
+#### Strengths
+- Good to excellent on typed appendix pages — message forms, movement orders
+- Handles moderately complex forms
+- Fast
+
+#### Failures & Known Issues
+
+| Issue | Page(s) | Severity | Description |
+|-------|---------|----------|-------------|
+| Dense page truncation | 6 (doc p.7) | High | Truncates after the 6 Sep entry, drops the entire 7 Sep entry including Loon Plage operational passage. |
+| Sparse page truncation | 51 | Medium | Misses entire signature block at bottom of page — processes empty space, stops before reaching content. |
+| Long narrative truncation | 29 | Medium | Good accuracy but drops final 1.5 paragraphs. Truncation pattern is not purely length-dependent. |
+| Complete failure on complex forms | 10 | High | Abandons transcription of the Field Return of Officers after the header — no table data produced. |
+| Handwriting — fundamental failure | 32 | **Critical** | Largely unintelligible on cursive handwriting. "Leen Cire des. thae Laucklan", "Euront pour Koon plage". Not fit for purpose on handwritten pages. |
+| `<think>` block leakage | All | Medium | Reasoning chains appear in output throughout. Requires post-processing to strip. |
+| `I`/`J` confusion | 17 | Medium | "I Sec" (Intelligence Section) misread as "J. See". Pattern likely recurs across corpus. |
+| `S`→`3` substitution | 23, 24+ | Medium | Systematic: "3ASK R" (SASK R), "30UTH" (SOUTH), "GUARD3" (GUARDS). Amenable to post-processing correction for known unit names. |
+| `Z`→`S` substitution | 26 | Medium | "PS FAUST" → "PZ FAUST" (Panzerfaust). Corrupts weapons identification. |
+
+#### Handwriting Ranking (page 32)
+**olmOCR > Claude Sonnet 4.6 > Flash** — Flash is not fit for purpose on handwritten pages.
+
+#### Verdict
+Viable for typed appendix pages. Not viable as primary OCR engine. Potential role: supplementary typed-content extraction or document classification, not as olmOCR replacement.
+
+---
+
+### 3.3 Gemini 2.5 Pro
+
+**Model:** `google/gemini-2.5-pro`  
+**Cost (input/output):** $1.25 / $10.00 per 1M tokens  
+**Output volume:** ~4,144 lines for 32 pages — ~3× olmOCR, driven by think-block overhead and verbose HTML
+
+#### Strengths
+- Best general-purpose performer before Claude 4.6 was tested
+- Correct on "General Crerar" — no Montgomery substitution
+- No repetition loops (page 30 complete)
+- Gets the Montgomery letter (page 27) correct in full
+- Handles maps and traces — gets all report line names on page 19
+- Best handwriting performance of the non-Claude models
+- Strong on complex forms — message forms, DF task tables, movement orders
+
+#### Failures & Known Issues
+
+| Issue | Page(s) | Severity | Description |
+|-------|---------|----------|-------------|
+| Base64 image injection | 6, 7 (Remarks column) | High | Embeds `<img src="data:image/png;base64,...">` garbage strings in Remarks column when it encounters handwritten initials/circled marks. Post-processing strip required. |
+| Token exhaustion on complex forms | 9, 10 | Medium | Page 9 cuts off mid-table in the tradesmen section. Fix: increase `max_tokens` to 8192. Not a capability failure. |
+| `<think>` block leakage | All | Medium | Verbose reasoning chains throughout output. Requires post-processing strip. |
+
+#### Verdict
+Significantly better than olmOCR on typed content. The base64 injection and token exhaustion issues are tractable post-processing problems, not fundamental model failures. Recommended as a replacement for olmOCR on typed content if cost is acceptable.
+
+---
+
+### 3.4 Qwen3-VL-30B
+
+**Model:** `Qwen/Qwen3-VL-30B-A3B-Instruct`  
+**Status: DISQUALIFIED — do not use in production**
+
+#### Failures & Known Issues
+
+| Issue | Page(s) | Severity | Description |
+|-------|---------|----------|-------------|
+| Infinite repetition loop | Multiple | **Critical** | Triggered on ~40% of complex pages. Repeats the same phrase endlessly for the rest of the token budget. |
+| Silent failure / empty output | Multiple | **Critical** | ~40% silent-fail rate — produces empty or near-empty output on complex pages without any error signal. |
+| Infinite image loop | 17, 19 | **Critical** | On map/trace pages: outputs hundreds of `<div class="image"><img/></div>` elements. |
+| Systematic `S`→`3` substitution | 23, 25+ | High | "3ASK R", "30UTH", "GUARD3". More severe than Flash. |
+| Numeric digit errors | 26 | High | "404 BLANDES" vs "484", "559 GR 331" vs "531 inf div". Corrupts unit identification in intelligence reports. |
+| Handwriting | 32 | High | Partially attempts but degrades badly on cursive. Below olmOCR. |
+| Padding loop | 22, 25 | Medium | Outputs hundreds of empty `<tr><td></td></tr>` rows after content ends. |
+
+**One genuine strength:** Nominal roll forms (page 10 — Field Return of Officers). Qwen completed the 44-officer roll where Flash completely failed. This single strength is not sufficient to justify inclusion given the catastrophic failures elsewhere.
+
+#### Verdict
+**Do not use.** Infinite loops, ~40% silent-fail rate, and systematic character substitutions make Qwen3-VL-30B unsuitable for any production role.
+
+---
+
+### 3.5 Claude Sonnet 4.6
+
+**Model:** `claude-sonnet-4-6`  
+**API:** Anthropic direct (`api.anthropic.com`) — not available via DeepInfra  
+**Image format:** 150 DPI PNG → JPEG 85% compression (to stay under Anthropic's 5MB image limit)  
+**max_tokens:** 8192 (required — pages 9 and 10 exceeded 4096)  
+**Cost:** $3.00 / $15.00 per 1M tokens (input/output)  
+**Control test cost (pages 7–32):** $0.49 total, avg $0.0195/page  
+**Full corpus estimate (150,000 pages):** $2,300–$2,900
+
+#### Strengths
+- **Best overall performer** — only model to pass all key validation tests cleanly
+- No repetition loops — page 30 (Tumbleweed letter) completes correctly
+- No hallucination substitution — "General Crerar" correct on page 4
+- No base64 image injection — clean HTML tables throughout
+- Completes page 10 Field Return of Officers (44-officer nominal roll) in full
+- Gets "Loon Plage" correct on page 32 (handwritten appendix)
+- Gets Montgomery letter (page 27) complete and accurate
+- Gets all report line names on trace map (page 19)
+- Clean output format — no think blocks, no padding artifacts, no inline CSS
+
+#### Failures & Known Issues
+
+| Issue | Page(s) | Severity | Description |
+|-------|---------|----------|-------------|
+| Token truncation on dense forms | 9, 10 | Medium | Truncated at 4096 default `max_tokens`. Fixed by setting 8192. Not a capability failure. |
+| Minor name misreads (handwriting) | 32 | Low | "D.F. MacLauchlan" vs "D.G.", "Hutchell" vs "Mitchell", "Wiegils" vs "Megill". Operational content intact. |
+| Superscript misreads | 7 | Low | "wome 25 pdrs fell short" → "some"; "morep at 2317" → missing superscript "hrs". |
+
+#### Key Validation Checks (Pass/Fail)
+
+| Check | Source of Failure on Other Models | Result |
+|-------|----------------------------------|--------|
+| Page 4 — "General Crerar" | olmOCR hallucination | ✅ Pass |
+| Page 10 — 44-officer nominal roll complete | Flash abandons after header | ✅ Pass |
+| Page 19 — trace map report lines | Qwen infinite image loop | ✅ Pass |
+| Page 27 — Montgomery letter complete | olmOCR fails | ✅ Pass |
+| Page 30 — no repetition loop | olmOCR and Qwen loop | ✅ Pass |
+| Page 32 — "Loon Plage" in header | olmOCR "Leuven", Flash "Koon" | ✅ Pass |
+| No base64 injection in Remarks | Gemini Pro failure | ✅ Pass |
+| No `<think>` block leakage | Flash and Pro failure | ✅ Pass |
+
+#### Verdict
+**Best model tested. Current recommendation for production OCR.** The cost ($2,300–$2,900 for full corpus) is within budget ceiling. The only structural issue is that it requires direct Anthropic API access — it cannot be run via DeepInfra's OpenAI-compatible endpoint for vision tasks.
+
+---
+
+## 4. Cross-Model Comparison by Page Type
+
+| Page Type | Winner | Runner-up | Avoid |
+|-----------|--------|-----------|-------|
+| Typed narrative diary entries | Claude 4.6 | Gemini Pro | Qwen |
+| Typed appendix forms | Claude 4.6 | Gemini Pro | Qwen |
+| Complex dual-page forms (nominal rolls) | Claude 4.6 | Qwen (only strength) | Flash |
+| Message forms (dense typed, grey paper) | Claude 4.6 | Gemini Pro | Qwen (S→3 errors) |
+| Artillery / DF task tables | Claude 4.6 | Gemini Pro | Qwen (swaps entries) |
+| Hand-drawn maps / traces | Claude 4.6 | Gemini Pro | Qwen (image loop) |
+| Handwritten narrative | olmOCR | Claude 4.6 | Flash (critical failure) |
+| Printed formal documents | Claude 4.6 | Gemini Pro | olmOCR |
+| Regimental journal / illustrated pages | Claude 4.6 | Gemini Pro | olmOCR (loops) |
+| Signing pages / simple forms | All adequate | — | — |
+
+---
+
+## 5. Known Failure Modes (All Models)
+
+### 5.1 Repetition / Infinite Loop
+**Affected models:** olmOCR (rare), Qwen3 (frequent — ~40% of complex pages)  
+**Trigger:** Ambiguous multi-column text, complex layouts, or anything that breaks the model's parsing flow  
+**Symptom:** Model gets stuck on a phrase and repeats it for the rest of the token budget  
+**Detection:** Output length disproportionately long; repeated n-grams in output  
+**Fix:** Post-processing loop detection (count repeated 10-gram occurrences); flag for human review
+
+### 5.2 Hallucination from Training Priors
+**Affected models:** olmOCR confirmed; presumed risk on all small/medium models  
+**Example:** "General Montgomery" substituted for "General Crerar" — statistically more probable completion pulled from training weights  
+**Danger:** Output looks completely plausible; undetectable without ground truth or image comparison  
+**Mitigation:** Use larger models with stronger instruction-following; provide Stacey/reference context to enable contradiction detection
+
+### 5.3 Token Truncation
+**Affected models:** Flash (systematic), Pro (on complex forms), Claude 4.6 (at default 4096 limit)  
+**Trigger:** Dense pages, long narrative entries, complex forms with many rows  
+**Fix:** Set `max_tokens: 8192` for all models; detect truncation by checking if output ends mid-sentence
+
+### 5.4 Base64 Image Injection
+**Affected models:** Gemini Pro  
+**Trigger:** Handwritten marks in Remarks column (circled initials, check marks)  
+**Symptom:** `<img src="data:image/png;base64,...">` garbage strings embedded in HTML table cells  
+**Fix:** Post-processing regex strip of any `<img>` tags containing base64 data
+
+### 5.5 `<think>` Block Leakage
+**Affected models:** Gemini Flash, Gemini Pro  
+**Symptom:** Model reasoning chains appear in output (verbose internal monologue about how to interpret the page)  
+**Fix:** Post-processing strip of `<think>...</think>` blocks
+
+### 5.6 Systematic Character Substitution on Degraded Scans
+**Affected models:** Qwen3 (severe), Flash (moderate)  
+**Known substitution clusters:**
+- `S` → `3` (e.g., "3ASK R" for "SASK R", "GUARD3" for "GUARDS")
+- `Z` → `S` (e.g., "PS FAUST" for "PZ FAUST")
+- `I` → `J` (e.g., "J. See" for "I Sec")
+  **Fix:** Post-processing correction dictionary for known military vocabulary and unit names
+
+### 5.7 Handwriting — Fundamental Failure
+**Affected models:** Gemini Flash (critical), Qwen3 (severe)  
+**Safe models for handwriting:** olmOCR (best), Claude Sonnet 4.6 (good)  
+**Recommendation:** Route handwritten pages to olmOCR or Claude; do not use Flash or Qwen
+
+---
+
+## 6. Cost Analysis
+
+| Model | Input (per 1M tokens) | Output (per 1M tokens) | Avg cost/page | Full corpus (150k pages) |
+|-------|-----------------------|------------------------|---------------|--------------------------|
+| olmOCR (DeepInfra) | ~$0.20 | ~$0.20 | ~$0.00039 | **~$60** |
+| Gemini Flash (DeepInfra) | $0.10 | $0.40 | ~$0.003 | **~$450** |
+| Gemini Pro (DeepInfra) | $1.25 | $10.00 | ~$0.018 | **~$2,700** |
+| Claude Sonnet 4.6 (Anthropic) | $3.00 | $15.00 | ~$0.0195 | **~$2,900** |
+| Qwen3-VL-30B (DeepInfra) | Variable | Variable | N/A | **Disqualified** |
+
+**Budget ceiling:** $2,000–$5,000 (confirmed acceptable)  
+**olmOCR was the original baseline** — $60 for the full corpus, essentially unlimited for prototype work with $20 loaded  
+**Claude Sonnet 4.6** sits comfortably within budget at ~$2,900 for full corpus re-OCR
+
+### Image Rendering
+- PDF pages rendered at **150 DPI** via Poppler (`pdftoppm`)
+- PNG output compressed to **JPEG 85%** to stay under Anthropic's 5MB per-image limit
+- Quality tested at page 32 (handwriting) — 150 DPI JPEG 85% produced the Claude results above
+- No quality regression observed vs PNG at this compression level
+
+---
+
+## 7. Architecture Decisions
+
+### 7.1 Two-Pass Pipeline is Correct
+**Question raised:** Is it wasteful to OCR in step 1 then run an LLM again in step 3?  
+**Answer:** No. Step 1 is vision work (image → text). Step 3 is reasoning work (text → structured JSON). Different tasks, different models, different costs.
+- Step 1 sends PNG images — expensive per page, requires visual understanding
+- Step 3 reads clean OCR text — cheap input, no image involved
+- Keeping them separate means if the extraction schema changes, you re-run step 3 only against already-OCR'd MD files. You never re-OCR.
+
+### 7.2 Why Not Collapse Into One Step
+Collapsing step 1 and step 3 would mean paying vision token rates for every field extraction on every page. It would also make re-runs of the extraction schema (which will happen as the project matures) require re-OCRing everything. The two-pass is the economical choice.
+
+### 7.3 Gemini Pro vs olmOCR Hybrid
+Three options were evaluated:
+
+**Option A — Gemini Pro everywhere**  
+Run Gemini Pro on all pages. Post-process to strip `<think>` blocks and base64 injections. Best accuracy, moderate cost.
+
+**Option B — Hybrid routing**
+- Classifier identifies page type first
+- Gemini Pro (or Claude) for: diary narrative pages, message forms, typed appendices, maps/traces
+- olmOCR as fallback for pages where Gemini Pro is known to inject base64 (handwritten marks in Remarks column)
+
+**Option C — olmOCR + LLM correction pass**  
+Run olmOCR first (fast, cheap). Flag low-confidence pages. LLM correction pass using Gemini Pro or Claude with Stacey/reference context injected.
+- Input to correction is text-only (no image) — costs ~$1.25/1M tokens, not vision rates
+- Stacey's Victory Campaign (~330,000 tokens) can be injected in full given Gemini Pro's 1M context window
+- Correction model can catch loops, garbled text, broken sentences — but cannot catch plausible hallucinations (e.g., Montgomery for Crerar) without seeing the image
+
+**Current decision:** Claude Sonnet 4.6 directly is the cleanest option given it outperforms the hybrid on all page types tested.
+
+### 7.4 pgvector over Chroma DB
+Deliberate architectural choice. Chroma is a pure vector store — semantic similarity only. pgvector inside Postgres enables a single SQL query combining:
+- Semantic similarity (vector search)
+- Date range filters
+- Unit ID filters
+- PostGIS spatial queries (`ST_DWithin` — "find events within 5km of this grid reference")
+
+This spatial query capability is the architectural moat. Chroma cannot do this. Adding PostGIS to a Chroma pipeline would require a separate system and application-layer joins.
+
+### 7.5 Document Classification
+Two-pass classification approach:
+
+**Pass 1 — Free, no API (OCR quality scoring):**  
+Score each page's OCR output for alpha-ratio and recognisable text patterns. Classifies ~80% of pages without any API call.
+
+**Pass 2 — Vision LLM on ambiguous pages (~20%):**  
+Llama 3.2 Vision or Qwen2.5-VL on DeepInfra. One API call per ambiguous page, tiny output (classification label + confidence JSON).  
+Estimated cost: $2–5 for full 150,000-page corpus.
+
+---
+
+## 8. Key Principles
+
+- **Sources problem vs prompt problem:** Low-quality OCR output from war diary narrative pages is partly a sources problem — the diary itself is officer-centric. The solution is broader document type coverage (patrol reports, sitreps, message logs, Part 1 Orders), not better prompting.
+- **olmOCR hallucinations cannot be fixed by prompting.** A 7B OCR-fine-tuned model does not reliably follow "transcribe verbatim" instructions when it encounters statistically probable alternative completions.
+- **Handwriting routing is essential.** Flash and Qwen are not fit for purpose on cursive handwritten pages. olmOCR and Claude 4.6 must handle these.
+- **Bad grids silently poison PostGIS.** Extraction accuracy in step 3 matters more than cost — a wrong grid reference goes into the database looking correct.
+- **The viewer MD and the JSON outputs serve different consumers.** Viewer MD = display artifact for Jonesy QA and Project '44 frontend. Step 3 JSONs = database inputs. Both are necessary and neither replaces the other.
+- **Transkribus is not viable at scale.** ~€10,000–12,000 for 150,000 pages vs ~$60 for olmOCR or ~$2,900 for Claude. The accuracy premium does not justify the cost delta.
+
+---
+
+## 9. Open Questions & Next Tests
+
+| Question | Priority | Notes |
+|----------|----------|-------|
+| Claude 4.6 vs Llama 70B for step 3 extraction | High | Step 3 currently uses Llama for narrative/position extraction. Claude likely more accurate on ambiguous table rows and grid references. Run same diary pages through both and compare `_positions.json` output. |
+| `anthropic/claude-4-sonnet` via DeepInfra for vision | Medium | Untested — Claude 4 (not 4.6) is available on DeepInfra. May support image input unlike Claude 3.7 Sonnet. Would enable single-API approach if it matches Claude 4.6 quality. |
+| LLM correction pass on olmOCR output | Medium | Gemini Pro or Claude as post-processor on olmOCR text, with Stacey context injected. Can catch loops and garbled text but not plausible hallucinations. Worth quantifying the accuracy delta vs direct Claude OCR. |
+| 100 DPI vs 150 DPI JPEG 85% | Low | Test page 32 (handwriting) at 100 DPI PNG vs current 150 DPI JPEG 85%. Determine if quality regression is measurable. 100 DPI PNG would remove Pillow dependency and eliminate compression artifacts. |
+| Transkribus small-batch test | Low | Test 50 handwritten appendix pages to quantify accuracy vs olmOCR. Not viable at corpus scale but useful data point for handwriting quality ceiling. |

notes/wardiary_copilot_lessons-learned.md

@@ -0,0 +1,335 @@
+# Project '44 — OCR Model Lessons Learned
+*War Diary OCR Pipeline | Calgary Highlanders Sep 44 Control Test*
+*Compiled May 2026*
+
+---
+
+## 1. Test Corpus
+
+All model evaluations used the same 32-page control PDF derived from the Calgary Highlanders War Diary, September 1944. The 32 pages were selected to cover the full range of document types present in a typical war diary file:
+
+| Document Type | Pages in Test | Notes |
+|---|---|---|
+| Typed narrative diary entries (HTML table) | 7 | Core data pages — Place / Date / Hour / Summary / Remarks |
+| Message forms (Army Form C2136) | ~8 | Dense typed text, some on grey or degraded paper |
+| Artillery / DF task tables | 2 | Complex tabular data with grid references |
+| Field Return of Officers (dual-page form) | 1 | Complex nominal roll — 42 named officers |
+| Field Return of Other Ranks | 1 | Complex dual-form, dense |
+| Movement orders | 2 | Mixed typed + handwritten annotation |
+| Hand-drawn maps / traces / sketch maps | 4 | APIS trace, disposition sketch, report lines, DMA order |
+| Handwritten narrative appendix | 1 | Cursive, "Enroute Loon Plage to Antwerp" |
+| Regimental journal (The Glen) | 3 | Cover, CO's address (degraded), newsletter pages |
+| Administrative pages | 3 | Cover, instructions, appendix index, signing page |
+
+---
+
+## 2. Models Tested
+
+| Model | API / Provider | Status | Run Method |
+|---|---|---|---|
+| olmOCR 2-7B (`allenai/olmOCR-2-7B-1025`) | DeepInfra | ✅ Completed | `step1_ocr_wardiaries_olmocr.py` |
+| Gemini 2.0 Flash (`google/gemini-2.0-flash`) | DeepInfra | ✅ Completed | `ocr_model_comparison.py` |
+| Gemini 2.5 Flash (`google/gemini-2.5-flash`) | DeepInfra | ✅ Completed | `ocr_model_comparison.py` |
+| Gemini 2.5 Pro (`google/gemini-2.5-pro`) | DeepInfra | ✅ Completed | `ocr_model_comparison_anthropic.py` |
+| Qwen3-VL-30B (`Qwen/Qwen3-VL-30B-A3B-Instruct`) | DeepInfra | ✅ Completed (disqualified) | `ocr_model_comparison.py` |
+| Claude 3.7 Sonnet (`anthropic/claude-3-7-sonnet-latest`) | DeepInfra | ❌ 404 — model not available on DeepInfra | `ocr_model_comparison.py` |
+| Claude Sonnet 4.6 (`claude-sonnet-4-6`) | Anthropic API direct | ✅ Completed | `step1_ocr_wardiaries_claude-4-6.py` |
+
+### Key Lesson: Claude via DeepInfra Does Not Work
+
+`claude-3-7-sonnet-latest` was routed through DeepInfra's OpenAI-compatible endpoint. Every page returned `404 Not Found`. DeepInfra does not host Anthropic models. All Claude testing must use the Anthropic API directly (`api.anthropic.com/v1/messages`) with `x-api-key` header and `anthropic-version: 2023-06-01`.
+
+---
+
+## 3. Overall Model Rankings
+
+### Typed Diary Content (narrative pages, message forms, movement orders)
+
+| Rank | Model | Notes |
+|---|---|---|
+| 1 | **Gemini 2.5 Pro** | Best overall. No paraphrase contamination, no factual hallucination. Wins ~25/32 pages by category. |
+| 2 | **Claude Sonnet 4.6** | Strong on structured diary tables. Outputs clean HTML tables consistently. Not yet formally scored against control test on all 32 pages. |
+| 3 | **olmOCR 2-7B** | Reliable baseline. Consistent output structure. See failure modes below. |
+| 4 | **Gemini 2.5 Flash / 2.0 Flash** | Identical outputs. Viable for typed pages; fails on handwriting. Not a Pro replacement. |
+| 5 | **Qwen3-VL-30B** | **Disqualified.** Catastrophic failure modes. Do not use. |
+
+---
+
+## 4. Model-by-Model Findings
+
+---
+
+### 4.1 olmOCR 2-7B
+
+**Overall verdict:** Reliable baseline. Never enters infinite loops, never silently fails on typed pages. Failure modes are predictable and lower-severity than Gemini Flash or Qwen. Currently used as the production step 1 model pending full Claude 4.6 evaluation.
+
+**Strengths:**
+- Consistent structured output (HTML tables where appropriate, plain text elsewhere)
+- No base64 embedding or `<think>` block leakage
+- Good performance on clean typed diary narrative pages
+- Handles most message forms adequately
+- Timeout is the only failure mode on complex forms (deterministic — not silence)
+
+**Failure modes:**
+
+| Failure | Page(s) | Severity | Example |
+|---|---|---|---|
+| Paraphrase contamination | 2, 5 | High | "of the force" → "of the army as a whole"; verbatim accuracy compromised on boilerplate |
+| Factual hallucination | 4 | **Critical** | "General Montgomery" instead of "General Crerar" — wrong person at the Dieppe march past |
+| Meaning reversal | 6 | **Critical** | "roads were reported clear" → "roads were reported closed" — flips historical record |
+| Artillery garbling | 7 (p.30) | High | "some 25 pdrs fell short" → "some 20 plrs fell asleep" — loses calibre and military meaning |
+| Timeout on complex forms | 9 | High | Field Return of Other Ranks times out at 180s. Complex multi-part forms exceed limit. |
+| Handwriting degradation | 32 | High | "Loon Plage" → "Caen Place"; "MacLauchlan" → "G.H. Launchlan"; distances wrong |
+| Reversed initials | 8 (p.51) | Low | "R. L. Ellis" → "A. R. L. Ellis" |
+| Nominal roll errors | 10 | Medium | CO name garbled; table structure breaks down mid-form |
+
+**Post-processing notes:**
+- olmOCR output is the cleanest raw text of all tested models — no stripping required
+- Timeout is the only deterministic failure; retry with smaller page batches
+
+---
+
+### 4.2 Gemini 2.0 Flash and Gemini 2.5 Flash
+
+**Overall verdict:** Both models produced **identical outputs** on every page — no meaningful difference between them for this document type. Viable for clean typed appendix pages as a supplementary model. Not a replacement for olmOCR. Not fit for handwritten content.
+
+**Strengths:**
+- Good to excellent on typed message forms, movement orders, typed narrative pages
+- Reasonable table structure preservation
+- Generally faster than Pro
+
+**Failure modes:**
+
+| Failure | Page(s) | Severity | Example |
+|---|---|---|---|
+| `<think>` block leakage | All | Medium | Reasoning chain present in output — must be stripped post-processing |
+| Truncation on dense multi-entry pages | 6 (doc p.7) | **High** | 6 Sep entry complete; entire 7 Sep entry (Loon Plage) dropped |
+| Truncation on long narrative pages | 29 (CO's address) | Medium | Final 1.5 paragraphs lost |
+| Truncation on sparse pages | 51 (signing) | Medium | Signature block at page bottom missed |
+| Handwriting total failure | 32 | **Critical** | "Leen Cire des. thae Laucklan", "Euront pour Koon plage" — largely unintelligible |
+| S→3 substitution | 23, 24+ | Medium | "SASK R" → "3ASK R", "SOUTH" → "30UTH", "GUARDS" → "GUARD3" |
+| Z→S substitution | 26 | Medium | "PZ FAUST" → "PS FAUST" |
+| I→J confusion | 17 | Medium | "I Sec" (Intelligence Section) → "J. See" |
+| Redaction marker garbling | 21 | Low | "XXXXXXX" → "Kaxxkuka" |
+| Complex dual-page form failure | 10 | **Critical** | Abandons after header; 44 named officers completely lost |
+| Diagram/trace partial reads | 14 | Medium | Unit names split across lines; edge text garbled with no uncertainty flag |
+
+**Recommendation:** Viable as a supplementary model for document classification and typed-content extraction only. Route handwritten pages and complex forms away from Flash.
+
+---
+
+### 4.3 Gemini 2.5 Pro
+
+**Overall verdict:** Clear winner on typed content and best-performing general model tested to date. Significantly better than olmOCR on narrative diary pages, message forms, maps, and handwritten content. Requires mandatory post-processing to strip `<think>` blocks and embedded base64 data.
+
+**Strengths:**
+- Wins typed narrative diary entries — no paraphrase contamination, no hallucination
+- Wins on maps and hand-drawn traces (only model that successfully extracts text from these)
+- Best performance on handwritten narrative appendix (page 32)
+- Correctly reads degraded/compressed typography better than any other model
+- Gets original typos and spelling errors correctly (verbatim faithfulness)
+- Gets superscript annotations, handwritten margin notes, signatures
+- Gets French place names accurately ("Loon Plage" → correct on all attempts)
+
+**Strongest pages vs olmOCR:**
+- Page 4: Correct "General Crerar" where olmOCR hallucinated "General Montgomery"
+- Page 5: Correct place names and diarist quotes where olmOCR paraphrased and garbled
+- Page 6: Correct "roads were reported clear" where olmOCR reversed to "closed"
+- Page 7: "25 pdrs fell short" correct where olmOCR produced "20 plrs fell asleep"
+- Page 32 (handwritten): Correct "Loon Plage 1368", "MacLauchlan", "Borsbeek 782933"
+
+**Failure modes:**
+
+| Failure | Page(s) | Severity | Example |
+|---|---|---|---|
+| `<think>` block leakage | All | Medium | Longer and more verbose than Flash; must be stripped |
+| Embedded base64 image data | 6, 7 | **High** | Hundreds of lines of `data:image/png;base64` embedded in Remarks column; file size explodes |
+| HTML verbosity (`<br>` proliferation) | 4, 7 | Low | Dozens of empty `<br>` tags in table cells |
+| Truncation on complex forms | 10, 16 | High | Field Return of Officers and Movement Order cut off mid-page (token limit) |
+| Page 29 degradation | 29 | Medium | Dense two-column CO's address: "Docompris", "noathing" — quality below usual standard |
+
+**Critical post-processing required:**
+```python
+import re
+
+def clean_gemini_output(text):
+    # Strip think blocks
+    text = re.sub(r'<think>.*?</think>', '', text, flags=re.DOTALL)
+    # Strip embedded base64 image data
+    text = re.sub(r'data:image/[^;]+;base64,[A-Za-z0-9+/=\n]+', '[IMAGE]', text)
+    # Strip excessive <br> padding in table cells
+    text = re.sub(r'(\s*<br>\s*){5,}', '\n', text)
+    return text.strip()
+```
+
+---
+
+### 4.4 Qwen3-VL-30B
+
+**Overall verdict: Disqualified. Do not use.**
+
+Multiple distinct catastrophic failure modes make Qwen3-VL-30B unsuitable for any production role in this pipeline. Even its single strength (nominal roll extraction on page 10) is not sufficient to justify inclusion.
+
+**Failure modes:**
+
+| Failure | Pages | Severity | Example |
+|---|---|---|---|
+| Infinite loop / token flood | 1, 17, 22, 24, 25, 27+ | **Critical** | "Bde Bde Bde Bde..." repeated thousands of times; or `<tr><td>2</td></tr>` ×hundreds |
+| Silent image placeholder failure | 14, 15, 19, 27, 28, 29, 31 | **Critical** | `<div class="image"><img/></div>` — zero text extracted. Affects ~40% of pages including clean printed pages |
+| S→3 systematic substitution | 23+ | High | "GUARDS" → "GUARD3", "SCOT" → "3COT", "3 SASK R", "FU3 MR" |
+| "LOON-PLACE" misread | 23, 32 | High | "LOON-PLAGE" → "LOON-PLACE" — wrong on the key Calgary Highlanders location throughout |
+| Numeric errors on intel content | 26 | High | "559 GR 331 inf div" → "859 GR 531 inf div" — corrupts unit identification |
+
+**Qwen's one notable strength:**
+- Page 10 (Field Return of Officers nominal roll) — Qwen correctly extracted all 42 officer names with ranks, corps, and appointments. This is the most structurally complex table in the test and Qwen outperformed every other model on this specific page. However this does not offset the ~40% silent failure rate.
+
+---
+
+### 4.5 Claude Sonnet 4.6
+
+**Status:** Successfully run via Anthropic API direct. Full 32-page comparison file exists (`step1_Calgary-Highlanders_War-Diary_Sep44_claude-4-6.md`). Formal page-by-page scoring against control test not yet completed.
+
+**What we know from the full diary run:**
+- Produces clean HTML tables for diary pages — same format as olmOCR
+- Output structure consistent and correct: `## Page N` markers, HTML tables where applicable
+- Image compression pipeline established: PDF → PNG (pdftoppm, 150 DPI) → JPEG (85% quality) → base64 → Anthropic API. JPEG compression required because PNG at 150 DPI is ~6-7 MB (over Anthropic's 5 MB image limit); JPEG 85% brings it to ~0.8-1.2 MB with no meaningful OCR detail loss.
+- Token usage logged per page: ~$3.00/1M input, $15.00/1M output
+
+**Technical parameters confirmed working:**
+- Endpoint: `https://api.anthropic.com/v1/messages`
+- Model ID: `claude-sonnet-4-6`
+- Headers: `x-api-key`, `anthropic-version: 2023-06-01`
+- Response path: `data["content"][0]["text"]`
+- `max_tokens: 8192` (double the extraction step — vision work needs more headroom)
+
+**Next step:** Formal page-by-page comparison of `step1_Calgary-Highlanders_War-Diary_Sep44_claude-4-6.md` against the Gemini Pro and olmOCR control test outputs.
+
+---
+
+### 4.6 Claude 3.7 Sonnet (via DeepInfra) — Complete Failure
+
+All 32 pages returned `[OCR FAILED — HTTP error: 404 Client Error: Not Found for url: https://api.deepinfra.com/v1/openai/chat/completions]`.
+
+**Root cause:** Anthropic models are not hosted on DeepInfra. The model ID `anthropic/claude-3-7-sonnet-latest` was passed to DeepInfra's OpenAI-compatible endpoint, which returned 404 for all requests.
+
+**Resolution:** Rewrote `step1_ocr_wardiaries_claude-4-6.py` to call the Anthropic API directly. Do not attempt to use any Anthropic model via DeepInfra.
+
+---
+
+## 5. Cross-Model Rankings by Page Type
+
+| Page Type | Best Model | Runner-up | Notes |
+|---|---|---|---|
+| Typed narrative diary entries | Gemini 2.5 Pro | Claude 4.6 (TBD) | olmOCR paraphrase/hallucination issues |
+| Typed appendix forms | Gemini 2.5 Pro | olmOCR | Qwen S→3 errors on grey pages |
+| Complex officer / rank nominal rolls | Qwen3 | Gemini 2.5 Pro | Qwen's only standout; truncation in Pro |
+| Message forms (dense typed) | Gemini 2.5 Pro | Qwen3 | olmOCR not fully sampled |
+| Artillery / DF task tables | Gemini 2.5 Pro | Qwen3 | Qwen swaps some entries |
+| Hand-drawn maps / traces | Gemini 2.5 Pro | — | Qwen fails, olmOCR fails |
+| Handwritten narrative | Gemini 2.5 Pro | olmOCR | Flash unintelligible; Qwen partially attempts |
+| Printed formal documents | Gemini 2.5 Pro | — | olmOCR and Qwen both fail page 27 (Montgomery) |
+| Illustrated / journal pages | Gemini 2.5 Pro | — | Qwen and olmOCR fail The Glen cover |
+| Signing pages / simple forms | Gemini 2.5 Pro / Qwen | — | All models adequate |
+
+---
+
+## 6. Character Substitution Patterns Found
+
+These substitution errors are scan/font artefacts and appear consistently across multiple models on degraded or grey-background pages:
+
+| Substitution | Affected Models | Pages | Correction Rule |
+|---|---|---|---|
+| `S` → `3` | Gemini Flash, Qwen | 23, 24+ | Post-processing rule against known military vocabulary (SASK R, SCOTS, GUARDS, SOUTH, FUS MR) |
+| `Z` → `S` | Gemini Flash | 26 | "PZ FAUST" → "PS FAUST" |
+| `I` → `J` | Gemini Flash | 17 | "I Sec" → "J. See" |
+| `F` → `P` | Qwen | 11 | "5 Cdn Fd Regt" → "5 Cdn Pd Regt" |
+| `O` → `0` | Qwen | 8 | "CA(O)" → "CA(0)" |
+
+**Recommendation:** Build a post-processing correction dictionary for known military unit names, abbreviations, and vocabulary. High-confidence substitutions can be corrected automatically; low-confidence ones should be flagged for human review.
+
+---
+
+## 7. Key Decisions Made
+
+### OCR Architecture
+- **claude-4-6 via Anthropic API** is the current production step 1 model for diary narrative pages
+- **olmOCR via DeepInfra** is the fallback / comparison baseline
+- Two-pass architecture (vision OCR → text extraction) is deliberate and correct — collapsing into one step would pay vision token rates for all field extraction
+
+### Image Processing
+- **150 DPI** is sufficient for this document type at typical war diary scan quality
+- **JPEG 85% compression** is required before sending to Anthropic API (5 MB limit); no meaningful OCR detail loss at this quality level
+- PNG at 150 DPI produces ~6-7 MB images — always exceeds Anthropic's limit
+
+### Output Format
+- **HTML tables** for diary pages (Place / Date / Hour / Summary / Remarks structure)
+- **Plain text** for all other page types
+- **`## Page N`** section headers in output MD — required by step 2 (viewer) and step 3 (extraction)
+
+### Pipeline Routing (proposed)
+- **Misc documents (~90% of pages):** Step 1 → Step 3 immediately. No human verification needed.
+- **Narrative diary pages (~10% of pages):** Step 1 → Step 2 viewer → human verification → Step 3. Only verified text feeds the descendant report.
+
+### Models Definitively Ruled Out
+- **Qwen3-VL-30B:** Do not use. ~40% silent failure rate, infinite loops, systematic character errors.
+- **Gemini Flash (either version):** Not for handwriting, not for complex forms.
+- **Any Anthropic model via DeepInfra:** Returns 404. Use Anthropic API directly.
+
+---
+
+## 8. OCR Correction Experiment (Gemini Pro + Stacey Context)
+
+**Test:** Attempted automated OCR correction by passing olmOCR output to Gemini 2.5 Pro along with contextual reference material (C.P. Stacey's *The Victory Campaign*).
+
+**Pages tested:** 4, 5, 6, 7, 30, 32
+
+**Result:** 0 corrections suggested across all 6 pages.
+
+**Analysis:** One of two interpretations:
+1. The olmOCR output on these pages was accurate enough that Gemini Pro found nothing to correct — a positive signal for olmOCR quality on the pages selected.
+2. The correction prompt was insufficiently targeted to surface the known errors (paraphrasing, meaning reversals, hallucinations) that were identified in the manual comparison.
+
+**Conclusion:** Automated correction via a second-pass LLM is not yet proven. Manual human review (via the OCR viewer) remains the quality gate for narrative pages.
+
+---
+
+## 9. Cost Comparison
+
+Based on DeepInfra pricing and Anthropic API pricing as at May 2026, and typical token counts for this document type (~1,500 input tokens per page for text, plus ~1,200 tokens for image via vision models):
+
+| Model | Provider | Input $/1M | Output $/1M | Relative Cost | Notes |
+|---|---|---|---|---|---|
+| olmOCR 2-7B | DeepInfra | ~$0.07 | ~$0.07 | 1× (baseline) | Cheapest. Consistent. |
+| Gemini 2.0 Flash | DeepInfra | ~$0.15 | ~$0.60 | ~4× | Same output as 2.5 Flash |
+| Gemini 2.5 Flash | DeepInfra | ~$0.15 | ~$0.60 | ~4× | No gain over 2.0 Flash |
+| Qwen3-VL-30B | DeepInfra | ~$0.20 | ~$0.88 | ~6× | Disqualified |
+| Gemini 2.5 Pro | DeepInfra | ~$1.25 | ~$10.00 | ~50× | Best accuracy; think blocks inflate token count |
+| Claude Sonnet 4.6 | Anthropic | $3.00 | $15.00 | ~100× | Best structured output; no post-processing needed |
+
+**For 150,000-page scale:**
+- olmOCR at scale: very low cost; accuracy limitations are the constraint
+- Gemini Pro at full corpus: materially expensive; hybrid approach preferred
+- Claude 4.6 at full corpus: highest cost but may be justified for narrative diary pages (only ~10% of pages)
+
+**Recommended approach at scale:** olmOCR for misc documents (90%); Claude 4.6 for narrative diary pages (10%). This concentrates the expensive model where accuracy matters most for the commercial product.
+
+---
+
+## 10. Known Open Questions
+
+1. **Formal Claude 4.6 scoring:** `step1_Calgary-Highlanders_War-Diary_Sep44_claude-4-6.md` exists but has not been formally scored against the control test comparison. This should be the next evaluation step.
+
+2. **Step 3 model choice:** Step 3 (extraction) currently uses `meta-llama/Meta-Llama-3.1-70B-Instruct` (olmOCR variant) and `claude-sonnet-4-6` (claude variant). Llama 70B has not been formally evaluated for grid reference extraction accuracy. Bad grids silently poison PostGIS downstream. Formal comparison of Llama vs Claude on `_positions.json` output quality is pending.
+
+3. **Handwriting routing:** No model has been confirmed for production-quality handwriting transcription. Gemini 2.5 Pro is the best tested but has truncation issues on complex pages. A dedicated handwriting model (e.g., Google Document AI, Azure Form Recognizer) has not been evaluated.
+
+4. **Complex form failures:** Field Return of Officers (nominal roll) causes truncation in Gemini Pro and timeouts in olmOCR. Qwen handled it but is otherwise disqualified. No production-viable solution confirmed for this page type.
+
+5. **`<think>` block suppression:** Gemini Pro's think blocks can be stripped post-processing, but they also inflate token counts and costs. Testing a `thinkingBudget: 0` parameter (if available via DeepInfra) is pending.
+
+---
+
+*Document covers: olmOCR 2-7B, Gemini 2.0 Flash, Gemini 2.5 Flash, Gemini 2.5 Pro, Qwen3-VL-30B, Claude 3.7 Sonnet (failed), Claude Sonnet 4.6*
+*Test corpus: Calgary Highlanders War Diary Sep 44, 32-page control test*
+*Pipeline scripts: `step1_ocr_wardiaries_claude-4-6.py`, `step1_ocr_wardiaries_olmocr.py`, `ocr_model_comparison.py`, `ocr_model_comparison_anthropic.py`*
+

notes/wardiary_pipeline_notes.md

@@ -0,0 +1,96 @@
+# Project '44 — Pipeline Session Notes
+*May 2026 | Working log — pipeline architecture & decisions*
+
+---
+
+## 1. Pipeline Overview
+
+| Step | Name | Input | Output | Model | Notes |
+|------|------|-------|--------|-------|-------|
+| 1 | OCR | PDF pages (PNG) | Raw MD with HTML tables | Claude 4.6 or olmOCR | Vision work — reads scanned images. Claude 4.6 significantly better than olmOCR on structured docs. |
+| 2 | Viewer MD | Step 1 MD | Clean viewer MD (no HTML) | Script (no LLM) | Display artifact only — for Jonesy QA and Project '44 frontend. Not a data input. |
+| 3 | Extraction | Step 1 MD (text only) | _narratives.json + _positions.json | Llama 70B (TBD: Claude?) | Text reasoning only — no vision needed. Re-runnable cheaply if schema changes. |
+
+---
+
+## 2. Why Two Passes (Step 1 + Step 3) Is Not Wasteful
+
+Step 1 is vision work (image → text). Step 3 is reasoning work (text → structured JSON). Fundamentally different tasks.
+
+- Step 1 sends PNG images to a vision-capable model — expensive per page, requires visual understanding of scanned documents.
+- Step 3 reads clean OCR text and extracts structured data (dates, grids, place names, categories). No image involved. Costs a fraction of step 1.
+- Keeping them separate means if the extraction schema changes (it will), you re-run step 3 only against already-OCR'd MD files. You never re-OCR.
+- Collapsing into one step would mean paying vision token rates for every field extraction on every page — more expensive and harder to prompt correctly.
+
+**Decision: Two-pass is the correct architecture. The separation is a cost advantage, not a cost problem.**
+
+---
+
+## 3. What Step 3 Actually Produces (and Why It's Needed)
+
+The viewer MD looks good — but step 3 is still necessary. The two outputs serve completely different consumers.
+
+### _narratives.json
+- Dated entries as discrete records: date, place, weather, full summary text.
+- Feeds the descendant report generation pipeline.
+- Without it, Claude has no way to query "what happened on 14 Sep 44" without reading the entire diary.
+- The viewer MD is prose — it cannot be queried or filtered by date.
+
+### _positions.json
+- Geo-extractable records with raw grid references, sheet refs, position categories (END_OF_DAY, SUBUNIT, ENEMY, PATROL, etc.), confidence levels, and context snippets.
+- Feeds the mapping layer — Postgres with pgvector + PostGIS.
+- Without it, there are no coordinates to plot on the map.
+
+**Viewer MD = display artifact. Step 3 JSONs = database inputs. Both necessary.**
+
+---
+
+## 4. Page Composition of a War Diary — The 300/30 Insight
+
+A typical war diary PDF of ~300 pages breaks down roughly as:
+
+| Document Type | Approx. Pages | Notes |
+|---------------|---------------|-------|
+| Narrative diary pages | ~30 (10%) | The HTML table pages with Place / Date / Hour / Summary. Need human verification before narrative extraction. |
+| Misc documents | ~270 (90%) | Message logs, intel reports, newsletters, Part 1 Orders, nominal rolls, appendices, sitreps, patrol reports, traces. |
+
+---
+
+## 5. Proposed Pipeline Routing
+
+Narrative pages and misc documents have different quality gates — they should not share the same queue.
+
+### Misc documents (~270 pages)
+- Step 1 (OCR) → straight to step 3 extraction immediately.
+- No human verification needed before extraction.
+- Feed intel reports, patrol data, message metadata into the RAG pipeline.
+
+### Narrative diary pages (~30 pages)
+- Step 1 (OCR) → step 2 viewer MD → Jonesy verifies → step 3 narrative extraction.
+- Only after human sign-off does the page go to step 3.
+- Ensures `_narratives.json` (which feeds descendant reports) is built on verified text.
+
+**Practical benefit: 90% of extraction work flows through immediately. The bottleneck only applies to the 30 pages that matter most for the commercial product.**
+
+---
+
+## 6. Open Question: Llama vs Claude for Step 3
+
+Step 3 currently uses `meta-llama/Meta-Llama-3.1-70B-Instruct`. May produce more errors on:
+
+- Ambiguous HTML table rows with missing or split dates
+- Grid reference parsing and 4-digit → 6-digit expansion
+- Correct position category assignment (END_OF_DAY vs SUBUNIT vs FRIENDLY)
+
+**To test:** Run step 3 on the same Calgary Highlanders pages with (a) Llama 70B and (b) Claude Haiku or Sonnet. Compare `_positions.json` output quality, especially grid confidence and category accuracy. Bad grids silently poison PostGIS, so accuracy matters more than cost here.
+
+---
+
+## 7. Key Principles Confirmed This Session
+
+- **OCR quality:** olmOCR is decent; Claude 4.6 is significantly better for structured historical documents.
+- **Two passes:** The two-pass architecture (vision OCR → text extraction) is deliberate and cost-effective.
+- **Step 3 purpose:** Step 3 is not redundant — viewer MD and JSON outputs serve completely different consumers.
+- **Page ratio:** ~10% narrative pages, ~90% misc documents. Most of the pipeline runs without waiting for human review.
+- **Quality gate:** Narrative pages need human verification before step 3. Misc pages do not.
+- **Data quality:** pgvector + PostGIS in a single SQL query is a core architectural advantage. Bad grids are hard to catch downstream — model accuracy in step 3 matters.