OCR-Viewer (#1)

Co-authored-by: nathan <nathan.kehler@gmail.com> Reviewed-on: #1
2026-05-19 22:03:28 +00:00
parent 727f4fcc57
commit 330b703539
104 changed files with 144649 additions and 2 deletions
--- a/notes/pilot-pipeline-tasklist.md
+++ b/notes/pilot-pipeline-tasklist.md
@@ -0,0 +1,10 @@
+1. Pilot scope: 5 CIB + 4 battalions, Sept–Oct 44 (~10–12 PDFs)
+2. Write 30–50 evaluation questions and answer them by hand from the source documents — this is your ground truth
+3. omOCR the pilot corpus
+4. Human-verify the OCR — full review on this set, since it's also benchmark data
+5. Design the inventory schema (page-range based, with document-level granularity where pages are uniform)
+6. Build the inventory using LLM-assisted prompts, refined against your hand-built inventories of CH/5CIB/BW Sept 44
+7. Design and test chunking strategy — chunk size, metadata attachment, special handling for tables/orders
+8. RAG-chunk and embed
+9. Run evaluation questions against the system, score, iterate
+10. Once accuracy targets are hit on the pilot, scale to the full 1,200-doc corpus with automated OCR + inventory + chunking, with human spot-checks rather than full verification
--- a/notes/report-building-evaluations/v1-evaluation-accuracy.md
+++ b/notes/report-building-evaluations/v1-evaluation-accuracy.md
@@ -0,0 +1,10 @@
+Line 15 - Talks about Basingstoke, which I assume is where the Calgary highlanders were based in England before they went to France. This is a detail that I would expect to be in the report, and it is not. The report should be grounded in the diary, and this is a detail that is in the diary but not in the report. This is a miss, and it should be included in the report to give a sense of where the soldiers were before they went to France.
+line 15 - it says the co who know his officers by their first names. Is this in the war diaries or an assumption? 
+line 17 - great detail about the first world war actions of the 10th battalion
+line 19 - seems accurate to the war diary. It talks about GR Crockett recommended for the victoria cross,but I know crockett did not get the VC. Be careful to word it so that it is clear that he was recommended but did not receive it.
+line 27 - unless we are showing these position on the map, giving grid references in the report is not helpful to the family. It is more important to give a sense of where they were in relation to known places (e.g. "the town of X" or "the river Y") rather than giving grid references that the family will not understand. If we are showing these positions on the map, then it is fine to include the grid references, but if not, then it is better to omit them from the report.
+line 48 - spot checked. seems accurate to the war diary
+Line 50 - spot checked. seems accurate to the war diary.
+Line 52 - spot checked. seems accurate to the war diary.
+Line 55 - spot checked. seems accurate to the war diary.
+Line 61 - spot checked. seems accurate to the war diary.
--- a/notes/report-building-evaluations/v1-evaluation-customer.md
+++ b/notes/report-building-evaluations/v1-evaluation-customer.md
@@ -0,0 +1,19 @@
+Did I learn what my grandfather lived through, in a way I'll remember?
+
+I think the average person will get a good sense of where their grandfather was, and roughly wha they are doing. I do think that the because the war diary is very officer centric, and based around the command post and tactical headquarters, it is hard to get a sense of what the average private would have experienced. The report does a good job of trying to include the human texture of the experience, but it is still very much focused on the operational events and the perspective of the officers. I think that is a limitation of the source material, and it is something we will have to be mindful of as we produce these reports. We may want to consider supplementing the war diary with other sources that give a more ground level perspective, such as letters, diaries, or oral histories from soldiers who were there. That would help us to give a more well rounded picture of what the average soldier experienced during those battles.
+
+Are there moments here I'll quote to my family at Christmas?
+
+I think there are some sections people would want to share, but again very officer centric.
+
+Did the report respect my intelligence, or did it talk down to me / over my head?
+
+The report is written in a way that is accessible to a general audience, but it does not talk down to the reader. It assumes a certain level of knowledge about the war and the battles, but it also explains things in a way that is easy to understand. I think it strikes a good balance between being informative and being engaging, without being condescending or overly simplistic.
+
+What do I now want to ask, that the report didn't answer?
+
+More on what the average soldier experienced, and what the fighting was like from their perspective. I would also want to know more about the specific actions that Bill was involved in, if we can find that out. I would also want to know more about the human texture of the experience, such as what the rations were like, what the billets were like, what the weather was like, and what the civilian encounters were like. Those are the things that really bring the experience to life for me, and I think they would be important for other people as well.
+
+Would I pay for this? If yes, roughly what?
+
+I would pay for this, but how much I dont know. I think $20 at a minimum. If there was more depth to what it was like for an average soldier it would be worth more. As well, if we could include photos and maps, and see where Bill was day by day, and where the most important events happened, that would add a lot of value. 
--- a/notes/report-building-evaluations/v1-evaluation-operational.md
+++ b/notes/report-building-evaluations/v1-evaluation-operational.md
@@ -0,0 +1,8 @@
+Is this actually a mappable event? (Some movements/billets aren't really points; some are.)
+Most of it is very mappable. Especially if we map the events to the units location ie we know where the unit was, so you are already near it, use the gsgs base maps to find the location, and then use the grid reference to find the exact location. Very very useful
+
+Does the location text contain enough information to geocode? (Grid refs are gold; "Hinkelenoord" alone is harder.)
+Grids are perfect. Locations are harder, but I think over time as we map a place with a name, it needs to be captured into like a gazetter or toponomy record so that we can find it again. I think the more we do this, the better we will get at geocoding these locations. We will also get better at recognizing when a location is not specific enough to geocode, and when we need to do some additional research to find out where it is.
+
+Would Jonesy accept it without changes, edit it, or reject it? Estimate the proportions.
+I think Jonesy would accept most of it without changes, but there would be some edits for clarity and to make sure that we are not including any information that is not supported by the war diary. I would estimate that about 80% of the events would be accepted without changes, about 15% would be edited for clarity or to remove unsupported information, and about 5% would be rejected because they are not supported by the war diary or because they are not mappable.
--- a/notes/report-building-evaluations/v1-evaluation-voice.md
+++ b/notes/report-building-evaluations/v1-evaluation-voice.md
@@ -0,0 +1,3 @@
+The voice and tone is good but I feel it does not provide enough gravity to the heavy fightinng near hoogerheide, and Waclheren. I would like to see more of the emotional weight of the fighting in that area, and the toll it took on the soldiers. The report should not shy away from the brutality of the fighting, and should give a sense of the fear and uncertainty that the soldiers must have felt during those battles. Though, if it is missing from the war diary, it is not a problem to omit it from the report. I just want to make sure that we are not sanitizing the experience for the family, and that we are giving them a sense of what their ancestor went through during those battles.
+
+We can expand on the voice when we use other texts in the future such as CP Staceys The Victory Campaign. 
--- a/notes/report-building-evaluations/v2-source-selection.md
+++ b/notes/report-building-evaluations/v2-source-selection.md
@@ -0,0 +1,110 @@
+# Supplementary Source Inventory — Pte. Bill Bloggins (placeholder)
+## Calgary Highlanders, 1–10 September 1944
+
+**Purpose:** Curated list of supplementary documents to accompany the Calgary Highlanders main war diary narrative for the v2 prototype descendant report.
+**Focus window:** 1–10 September 1944
+**Source PDFs covered:** Calgary Highlanders, 5 Cdn Inf Bde, RHC Black Watch (R de Mais inventory not provided)
+**Inventory coverage:** CH pp. 1–45 of 343 · 5 CIB pp. 1–56 of 193 · BW pp. 1–51 of 144
+**Document count:** 24 items
+**Status:** Ready for use; gaps flagged at end
+
+---
+
+## Selection criteria (priority order)
+
+1. Documents covering specific events the Calgary Highlanders were involved in during 1–10 Sep 1944
+2. Brigade-level signals, sitreps, or orders covering the same dates and actions
+3. Sister-battalion entries that mention or coordinate with the Calgary Highlanders during the focus window
+4. Patrol programmes, patrol reports, casualty returns, or other operational documents adding detail beyond the diary narrative
+
+---
+
+## Source 1 — Calgary Highlanders War Diary
+**File:** `Calgary-Highlanders_War-Diary_Sep44.pdf` (343 pp; inventoried 1–45)
+
+| Page(s) | Document type | Date covered | Note |
+|---|---|---|---|
+| 6 | Appendix index | Sep 44 | Master key to the file — locates Appx 12 (Memorial Service 3 Sep), Appx 13 (Dieppe march past), Appx 10 (Daily News sheets) referenced in narrative for focus window |
+| 21–26 | War diary narrative pp. 15–20 | 11–13 Sep 44 | **Just outside window** — direct continuation of Loon Plage fighting Bloggins entered 7–10 Sep; drop if hard 10 Sep cut required |
+
+---
+
+## Source 2 — 5 Cdn Inf Bde War Diary
+**File:** `5CIB_War-Diary_Sep44.pdf` (193 pp; inventoried 1–56)
+
+| Page(s) | Document type | Date covered | Note |
+|---|---|---|---|
+| 5–20 | Bde war diary narrative (Brig. Megill) | 1–30 Sep 44 | Brigade-level command perspective on the same operations; only 1–10 Sep entries in scope |
+| 22–26 | Part I Orders (Appx 1) | 15 Sep 44 | Marginal — dated outside window but routine orders often cover preceding fortnight (dress, censorship, discipline) |
+| 28–34 | AFW 3008 Field Returns of Officers (Appx 3) | wks ending 2 & 9 Sep 44 | Officer strength returns spanning focus window — establishes who was commanding what when Bloggins arrived |
+| 35–43 | AFW 3009 Field Returns of Other Ranks (Appx 4) | wks ending 2 & 9 Sep 44 | OR strength returns — directly relevant to Bloggins as reinforcement; bde-level intake picture in his first days |
+| 44 | Mov Order No 4 | 1 Sep 44 | Bde move order on Day 1 of window — order under which CH moved from Totes area |
+| 45 | Trace "T" — unit dispositions | 1 Sep 44 | Companion trace to Mov Order 4 — shows CH placement within bde at start of window |
+| 46 | Mov Order No 5 | 1 Sep 44 | Second 1 Sep move order — sequencing of bde group movement |
+| 47 | Trace "R" — route sketch | 1 Sep 44 | Route sketch for Mov Order 5 |
+| 48–49 | Mov Order No 6 | 5 Sep 44 | Mid-window move order covering advance toward Dunkerque outpost line |
+| 50 | Route Card, 2 Cdn Inf Div | 5 Sep 44 | Div-level route card companion to Mov Order 6 — divisional context for bde move |
+| 51–52 | Mov Order No 7 | 6 Sep 44 | Move order issued day before CH narrative records Bourbourgville advance — likely the order that put CH into contact |
+
+---
+
+## Source 3 — RHC Black Watch War Diary
+**File:** `RHC-Blackwatch_War-Diary_Sep44.pdf` (144 pp; inventoried 1–51)
+
+| Page(s) | Document type | Date covered | Note |
+|---|---|---|---|
+| 6–15 | Bn war diary narrative (Lt-Col Ritchie) | 1–30 Sep 44 | Sister-bn narrative; only 1–10 Sep entries in scope. BW alongside CH through Dieppe parade (3 Sep) and advance to Dunkerque approaches |
+| 18 | SHAEF "Enemy Defences" trace | 23 Aug 44 | Pre-window but describes enemy defences CH would encounter at Bourbourgville/Loon Plage from 7 Sep — operational intel context |
+| 19 | Dunkerque Plan of Port & Town | n/d | Terrain CH was advancing toward through focus window |
+| 20 | Dunkerque defence overprint map | Ed. 11 Sep 44 | Compiled from air photo info as at 1 Sep and other sources as at 10 Sep — situation across Bloggins's first ten days |
+| 21 | Defence overprint legend | n/d | Companion key to the overprint — needed to read p. 20 |
+| 24–25 | RHC Field Return of Officers | wk ending 2 Sep 44 | Sister-bn officer strength at start of window |
+| 26–27 | RHC Field Return of Officers | wk ending 9 Sep 44 | Sister-bn officer strength at close of window |
+| 33 | Field Message Form (Lt-Col Ritchie to coys) | 10 Sep 44, 0025 hrs | Only timed BW signal in window — H-hour change to 0755, RV at 853093 at 0700; bde-sector operational tempo on closing night |
+| 34–35 | RHC Field Return of Other Ranks | wk ending 3 Sep 44 | Sister-bn OR strength — comparison of reinforcement intake across CH/RHC/R de M |
+| 36–37 | RHC Field Return of Other Ranks | wk ending 10 Sep 44 | Same, end of window |
+| 50 | APIS 2 Cdn Inf Div trace, Part I | 1 Sep 44 | Divisional intel trace from first day of window — beach defences, MGs, minefields in bde sector |
+| 51 | APIS 2 Cdn Inf Div trace, Part II | ~1 Sep 44 | Continuation of p. 50 trace |
+
+---
+
+## Items considered and excluded
+
+| Source | Page(s) | Reason for exclusion |
+|---|---|---|
+| BW | 22–23, 28–29, 30–31 | Officer returns wks ending 16, 23, 30 Sep — outside window |
+| BW | 38–39, 40–41, 42–44 | OR returns wks ending 16, 23, 30 Sep — outside window |
+| BW | 46–49 | Undated coy messages and faded carbon — cannot place in window with confidence |
+| 5 CIB | 53–56 | Mov Orders 9 (17 Sep) and later — outside window |
+
+---
+
+## Gaps — material expected but not in catalogued ranges
+
+1. **R de Mais war diary inventory** — not provided. Largest gap. R de Mais was the third bn in 5 CIB, operating alongside CH throughout focus window. Without it, criterion 3 (sister-bn coordination) is half-answered.
+
+2. **Calgary Highlanders Appendices 3–15** — none catalogued. The CH Move Orders (Appx 4), Messages (Appx 5), Patrol Programme (Appx 6), Memorial Service 3 Sep (Appx 12), and March Past Dieppe (Appx 13) are all directly in scope and almost certainly the highest-value supplementary documents in the entire source set. **Inventorying CH pp. 46–343 should be the next priority.**
+
+3. **5 CIB Ops Log (Appx 13, sheets 109–161)** — un-catalogued in pp. 57–193. Bde signal log for the whole month; would be the single richest source of bde-level granularity for 1–10 Sep.
+
+4. **5 CIB Patrol Programmes and Patrol Reports (Appx 6, 7)** — un-catalogued. Patrol activity at Loon Plage from 9–10 Sep described in CH narrative; patrol reports would corroborate.
+
+5. **5 CIB Int Report on Dunkerque, 13 Sep** (Appx 10) — outside date but retrospective on focus-window area; un-catalogued.
+
+6. **BW Appx 6 (Warning Orders, movement tables)** and **Appx 8 (Sitreps)** — un-catalogued in pp. 52–144. Sitreps would map well onto focus window.
+
+7. **No casualty returns** in any catalogued range. CH narrative refers to casualties 7–10 Sep at Bourbourgville/Loon Plage; actual casualty returns (Part II Orders, AFW 3010-series, or unit casualty lists) aren't in catalogued portions of any file. Likely in CH Appx 2 (Part II Orders) or BW Appx 2 ("to follow").
+
+8. **No artillery task tables** in scope. CH Appx 3 (Arty Task Tables) and BW Appx 7 (RCA Conc tables, tasks) both un-catalogued and would directly cover fire support for actions Bloggins was in.
+
+---
+
+## Recommended next inventory pass
+
+To meaningfully increase supplementary-source density:
+
+- **CH pp. 46–100** — capture early appendices likely covering 1–10 Sep (Move Orders, Messages, Patrol Programme, Memorial Service, Dieppe March Past)
+- **5 CIB pp. 57–100** — find start of Ops Log (Appx 13) and patrol material (Appx 6, 7)
+- **R de Mais Sep 44 war diary** — full inventory pp. 1–~50 to recover sister-bn coordination evidence
+
+These three passes would roughly double the supplementary-source density of this list.
--- a/notes/scoring-schema.md
+++ b/notes/scoring-schema.md
@@ -0,0 +1,63 @@
+Tier 1 — Direct factual lookup (10 questions)
+Tests basic retrieval: does the right chunk come back for a clear factual query?
+
+What unit was the Calgary Highlanders' Dog Coy attacking on 15 September 1944, and what was the location?
+Who signed the 5 CIB war diary for September 1944?
+On what date did the Calgary Highlanders force the Albert Canal crossing?
+What was the grid reference of the lock gates the Calgary Highlanders crossed at on 22 September 1944?
+Who was singled out for distinction during the Albert Canal crossing, and what coy/platoon were they in?
+What was the date of the Dieppe march past attended by the Calgary Highlanders?
+What was the Black Watch's location at the start of September 1944 (per the cover sheet)?
+How many German prisoners did the Calgary Highlanders take on 14 September 1944 in the action toward Fme Geersen?
+What was the H-hour given in the Black Watch field message of 10 September 1944, 0025 hrs?
+What route was issued under 5 CIB Mov Order No 5 on 1 September 1944?
+
+
+Tier 2 — Cross-document synthesis (10 questions)
+Tests whether the system can combine information from multiple sources to answer a single question.
+
+Compare the Calgary Highlanders and Black Watch officer strength at the start and end of September 1944. What does the trend suggest?
+On 22 September 1944, the Calgary Highlanders were ordered to take over a crossing the Black Watch had failed to make. What did the Black Watch war diary say about that failed attempt?
+What does the 5 CIB war diary narrative say about the Calgary Highlanders' actions on 7–10 September 1944, and how does it compare to the Calgary Highlanders' own narrative for those dates?
+Which battalion in 5 CIB had the highest other-rank strength in the week ending 9 September 1944?
+Trace the movement of 5 CIB from Totes to the Dunkerque outpost line using the Move Orders (No 4, 5, 6, 7) and the brigade narrative.
+What enemy defences were the Calgary Highlanders likely to encounter at Bourbourgville based on the SHAEF and APIS traces in the Black Watch and brigade files?
+On 5 September 1944, what was 5 CIB's planned route at the divisional level (per the 2 Cdn Inf Div Route Card) and how does it compare to the brigade's own Mov Order No 6?
+Identify all the locations mentioned in any 5 CIB document for the date 1 September 1944. What movement is implied?
+Which sister battalion was closest to the Calgary Highlanders' position on 9 September 1944?
+What casualties did 5 CIB units sustain in the first ten days of September 1944, and where would that information be found?
+
+
+Tier 3 — Operational and tactical reasoning (10 questions)
+Tests whether the system can answer questions that require understanding military context, not just text retrieval.
+
+What was 5 CIB's tactical role between 1 and 10 September 1944 — exploitation, set-piece attack, defensive holding, or something else? Cite evidence.
+Where was the Calgary Highlanders' advance party on 16–18 September 1944, and what does that suggest about brigade movement plans?
+What was the significance of the Calgary Highlanders crossing at Fort 1 on the Albert Canal rather than at the original Black Watch crossing point?
+Based on the patrol activity recorded around Loon Plage 9–13 September 1944, what was the tactical situation?
+What artillery support was available to the Calgary Highlanders during the Bourbourgville–Loon Plage operations? Where would that be documented?
+The 5 CIB war diary for September mentions "Dunkerque outpost line." What does that mean operationally, and which 5 CIB documents add detail to that picture?
+Why did the Calgary Highlanders move from France to Belgium between 18–19 September 1944, and what was the operational objective?
+What evidence exists in the brigade and battalion war diaries for the strength and quality of the German garrison the Calgary Highlanders were facing in early September 1944?
+Was the Calgary Highlanders' advance from Bourbourgville to Loon Plage opposed, lightly opposed, or heavily opposed? Cite specific document evidence.
+What changes in unit identifications from PW interrogations (per the GO-1 ISUM) might have affected 5 CIB's understanding of the enemy in early September 1944?
+
+
+Tier 4 — Reinforcement / personnel queries (5 questions)
+Tests the "His War" use case directly — reconstructing what an individual soldier would have experienced.
+
+A reinforcement arriving at the Calgary Highlanders on or about 1 September 1944 would have been part of which company actions in the following ten days? Walk through what they would have experienced day by day.
+What named reinforcements arrived at the Calgary Highlanders during 16–17 September 1944?
+If a soldier joined the Calgary Highlanders as a reinforcement in late August 1944, what was the strength of the battalion when they arrived (per the field returns), and how did it change over September?
+What activities did the Calgary Highlanders conduct that were not direct combat operations during 1–10 September 1944 (parades, baths, religious services, etc.)?
+A Calgary Highlanders soldier killed or wounded on 8 September 1944 at Bourbourgville would have their casualty recorded in which document(s), and on what dates would those documents have been filed?
+
+
+Tier 5 — Stress tests and adversarial queries (5 questions)
+Tests failure modes — does the system hallucinate when the answer isn't in the corpus, or does it correctly say "I don't know"?
+
+What were the Régiment de Maisonneuve's casualties on 8 September 1944? (R de Mais inventory not provided in pilot — system should say it doesn't have that source, not invent an answer.)
+What did General Crerar say in his speech at the Dieppe march past? (Speech text is not in the inventory; the march past is mentioned but the speech content isn't. Correct answer: "not in available sources.")
+What was the weather on 7 September 1944 at Bourbourgville? (May or may not be in the war diaries; system should answer from source if present, otherwise say so — not pull from external knowledge.)
+Who commanded the German garrison at Dunkerque in September 1944? (Likely not in 5 CIB sources; tests whether system stays within corpus or hallucinates from training data.)
+What does the war diary say about Calgary Highlanders operations on 5 October 1944? (October is in scope for the pilot but not yet inventoried in our session — tests whether system handles dates within scope but in un-verified portions of the corpus.)
--- a/notes/wardiary_claude_lessons-learned.md
+++ b/notes/wardiary_claude_lessons-learned.md
@@ -0,0 +1,381 @@
+# War Diary OCR — Lessons Learned
+*Project '44 | Running log of model tests, findings, and decisions*
+*Control test: Calgary Highlanders War Diary Sep 44 (32-page control PDF)*
+
+---
+
+## Table of Contents
+
+1. [Models Tested](#1-models-tested)
+2. [Control Test Setup](#2-control-test-setup)
+3. [Model-by-Model Findings](#3-model-by-model-findings)
+    - [olmOCR (allenai/olmOCR-2-7B-1025)](#31-olmocr)
+    - [Gemini 2.5 Flash](#32-gemini-25-flash)
+    - [Gemini 2.5 Pro](#33-gemini-25-pro)
+    - [Qwen3-VL-30B](#34-qwen3-vl-30b)
+    - [Claude Sonnet 4.6](#35-claude-sonnet-46)
+4. [Cross-Model Comparison by Page Type](#4-cross-model-comparison-by-page-type)
+5. [Known Failure Modes (All Models)](#5-known-failure-modes-all-models)
+6. [Cost Analysis](#6-cost-analysis)
+7. [Architecture Decisions](#7-architecture-decisions)
+8. [Key Principles](#8-key-principles)
+9. [Open Questions & Next Tests](#9-open-questions--next-tests)
+
+---
+
+## 1. Models Tested
+
+| Model | Provider | API Endpoint | Vision | Notes |
+|-------|----------|-------------|--------|-------|
+| `allenai/olmOCR-2-7B-1025` | DeepInfra | OpenAI-compatible | ✅ | Fine-tuned specifically for document OCR |
+| `google/gemini-2.5-flash` / `google/gemini-2.0-flash` | DeepInfra | OpenAI-compatible | ✅ | Both versions produced identical output |
+| `google/gemini-2.5-pro` | DeepInfra | OpenAI-compatible | ✅ | Best general-purpose performer tested |
+| `Qwen/Qwen3-VL-30B-A3B-Instruct` | DeepInfra | OpenAI-compatible | ✅ | Disqualified — critical failure modes |
+| `claude-sonnet-4-6` | Anthropic API (direct) | `api.anthropic.com` | ✅ | Best overall performer; not available via DeepInfra |
+| `anthropic/claude-3-7-sonnet-latest` | DeepInfra | OpenAI-compatible | ❌ | Vision/image requests return 404 via DeepInfra |
+
+> **Note:** `claude-sonnet-4-6` is only available through the Anthropic API directly. DeepInfra's OpenAI-compatible endpoint does not support multimodal input for Claude models. Claude Sonnet 4 is on DeepInfra as `anthropic/claude-4-sonnet` but has not been tested.
+
+---
+
+## 2. Control Test Setup
+
+**Source document:** Calgary Highlanders War Diary, September 1944 — 32-page control PDF  
+**Pages tested:** 7–32 (skipping admin/cover pages)  
+**Image format:** PNG rendered at 150 DPI via Poppler, then compressed to JPEG 85% to stay under Anthropic's 5MB per-image limit  
+**API pattern:** All DeepInfra models use the same OpenAI-compatible endpoint with base64-encoded image_url in the messages array
+
+**Document types in the control test:**
+
+| Page(s) | Document Type | Notes |
+|---------|--------------|-------|
+| 7–8, 11–13, 16, 22–26 | Typed narrative diary entries (HTML table format) | Core diary — Place / Date / Hour / Summary / Remarks columns |
+| 9–10 | Field Return of Officers / Other Ranks | Complex dual-page nominal roll form, densest page in test |
+| 14–15, 17–18, 20–21 | Movement orders, typed appendices | Mixed typed content with occasional handwritten annotation |
+| 19 | Hand-drawn trace / map overlay | Sparse — report line names with grid markers |
+| 27 | Printed formal letter (Montgomery) | Clean typeset, should be easy |
+| 28–29 | C.O.'s address (typed narrative) | Long-form typed, degraded scan |
+| 30–31 | Regimental journal pages (Tumbleweed letter) | Typed narrative, page 30 was the loop-trigger for olmOCR |
+| 32 | Handwritten narrative appendix | Cursive handwriting — hardest page in the test |
+
+---
+
+## 3. Model-by-Model Findings
+
+### 3.1 olmOCR
+
+**Model:** `allenai/olmOCR-2-7B-1025`  
+**Cost (control test):** ~$0.000388/page  
+**Full corpus estimate (150,000 pages):** ~$60
+
+#### Strengths
+- Designed specifically for document OCR — performs best on clean typed text
+- Reliable on typed narrative diary pages (the main diary HTML table entries)
+- Good on handwritten content relative to general VLMs — best of all tested models on page 32 (handwritten appendix)
+- No base64 image dumps, no think blocks, no padding loops on most pages
+- Predictable failure modes — when it fails, it fails consistently enough to detect
+- Output is clean, directly usable HTML tables
+
+#### Failures & Known Issues
+
+| Issue | Page | Severity | Description |
+|-------|------|----------|-------------|
+| Hallucination from training priors | 4 | **Critical** | Substitutes "General Montgomery" for "General Crerar" — does not misread the letters, pulls the statistically more probable name from training weights. Completely plausible, undetectable without ground truth. |
+| Repetition loop | 30 | **Critical** | Enters infinite repetition loop starting at the Tumbleweed letter ("A follow came out...") and repeats for the rest of its token budget. Same failure mode as Qwen but triggered less frequently. |
+| Paraphrase contamination | Multiple | High | Rewrites sentences in its own words rather than transcribing verbatim. Not a hallucination but not a transcription either — corrupts the historical record. |
+| Handwriting misreads | 32 | Medium | "Evacout from Leuven Plage" vs correct "Enroute from Loon Plage". Key place name wrong, though close enough to recognise. |
+| "Lieut" → "Heard" | Multiple | Medium | Consistent misread of the abbreviation "Lieut" in cursive as "Heard". Pattern is consistent enough to post-process. |
+
+#### Verdict
+olmOCR was the right initial choice vs Adobe OCR — it is meaningfully better on structured historical documents. But its ceiling is now clear. The Montgomery/Crerar hallucination and the loop failure are architectural limitations of a 7B model, not tuning problems. No prompt change will fix them.
+
+---
+
+### 3.2 Gemini 2.5 Flash
+
+**Models:** `google/gemini-2.5-flash`, `google/gemini-2.0-flash`  
+**Both versions produced identical output on all 32 pages.**
+
+#### Strengths
+- Good to excellent on typed appendix pages — message forms, movement orders
+- Handles moderately complex forms
+- Fast
+
+#### Failures & Known Issues
+
+| Issue | Page(s) | Severity | Description |
+|-------|---------|----------|-------------|
+| Dense page truncation | 6 (doc p.7) | High | Truncates after the 6 Sep entry, drops the entire 7 Sep entry including Loon Plage operational passage. |
+| Sparse page truncation | 51 | Medium | Misses entire signature block at bottom of page — processes empty space, stops before reaching content. |
+| Long narrative truncation | 29 | Medium | Good accuracy but drops final 1.5 paragraphs. Truncation pattern is not purely length-dependent. |
+| Complete failure on complex forms | 10 | High | Abandons transcription of the Field Return of Officers after the header — no table data produced. |
+| Handwriting — fundamental failure | 32 | **Critical** | Largely unintelligible on cursive handwriting. "Leen Cire des. thae Laucklan", "Euront pour Koon plage". Not fit for purpose on handwritten pages. |
+| `<think>` block leakage | All | Medium | Reasoning chains appear in output throughout. Requires post-processing to strip. |
+| `I`/`J` confusion | 17 | Medium | "I Sec" (Intelligence Section) misread as "J. See". Pattern likely recurs across corpus. |
+| `S`→`3` substitution | 23, 24+ | Medium | Systematic: "3ASK R" (SASK R), "30UTH" (SOUTH), "GUARD3" (GUARDS). Amenable to post-processing correction for known unit names. |
+| `Z`→`S` substitution | 26 | Medium | "PS FAUST" → "PZ FAUST" (Panzerfaust). Corrupts weapons identification. |
+
+#### Handwriting Ranking (page 32)
+**olmOCR > Claude Sonnet 4.6 > Flash** — Flash is not fit for purpose on handwritten pages.
+
+#### Verdict
+Viable for typed appendix pages. Not viable as primary OCR engine. Potential role: supplementary typed-content extraction or document classification, not as olmOCR replacement.
+
+---
+
+### 3.3 Gemini 2.5 Pro
+
+**Model:** `google/gemini-2.5-pro`  
+**Cost (input/output):** $1.25 / $10.00 per 1M tokens  
+**Output volume:** ~4,144 lines for 32 pages — ~3× olmOCR, driven by think-block overhead and verbose HTML
+
+#### Strengths
+- Best general-purpose performer before Claude 4.6 was tested
+- Correct on "General Crerar" — no Montgomery substitution
+- No repetition loops (page 30 complete)
+- Gets the Montgomery letter (page 27) correct in full
+- Handles maps and traces — gets all report line names on page 19
+- Best handwriting performance of the non-Claude models
+- Strong on complex forms — message forms, DF task tables, movement orders
+
+#### Failures & Known Issues
+
+| Issue | Page(s) | Severity | Description |
+|-------|---------|----------|-------------|
+| Base64 image injection | 6, 7 (Remarks column) | High | Embeds `<img src="data:image/png;base64,...">` garbage strings in Remarks column when it encounters handwritten initials/circled marks. Post-processing strip required. |
+| Token exhaustion on complex forms | 9, 10 | Medium | Page 9 cuts off mid-table in the tradesmen section. Fix: increase `max_tokens` to 8192. Not a capability failure. |
+| `<think>` block leakage | All | Medium | Verbose reasoning chains throughout output. Requires post-processing strip. |
+
+#### Verdict
+Significantly better than olmOCR on typed content. The base64 injection and token exhaustion issues are tractable post-processing problems, not fundamental model failures. Recommended as a replacement for olmOCR on typed content if cost is acceptable.
+
+---
+
+### 3.4 Qwen3-VL-30B
+
+**Model:** `Qwen/Qwen3-VL-30B-A3B-Instruct`  
+**Status: DISQUALIFIED — do not use in production**
+
+#### Failures & Known Issues
+
+| Issue | Page(s) | Severity | Description |
+|-------|---------|----------|-------------|
+| Infinite repetition loop | Multiple | **Critical** | Triggered on ~40% of complex pages. Repeats the same phrase endlessly for the rest of the token budget. |
+| Silent failure / empty output | Multiple | **Critical** | ~40% silent-fail rate — produces empty or near-empty output on complex pages without any error signal. |
+| Infinite image loop | 17, 19 | **Critical** | On map/trace pages: outputs hundreds of `<div class="image"><img/></div>` elements. |
+| Systematic `S`→`3` substitution | 23, 25+ | High | "3ASK R", "30UTH", "GUARD3". More severe than Flash. |
+| Numeric digit errors | 26 | High | "404 BLANDES" vs "484", "559 GR 331" vs "531 inf div". Corrupts unit identification in intelligence reports. |
+| Handwriting | 32 | High | Partially attempts but degrades badly on cursive. Below olmOCR. |
+| Padding loop | 22, 25 | Medium | Outputs hundreds of empty `<tr><td></td></tr>` rows after content ends. |
+
+**One genuine strength:** Nominal roll forms (page 10 — Field Return of Officers). Qwen completed the 44-officer roll where Flash completely failed. This single strength is not sufficient to justify inclusion given the catastrophic failures elsewhere.
+
+#### Verdict
+**Do not use.** Infinite loops, ~40% silent-fail rate, and systematic character substitutions make Qwen3-VL-30B unsuitable for any production role.
+
+---
+
+### 3.5 Claude Sonnet 4.6
+
+**Model:** `claude-sonnet-4-6`  
+**API:** Anthropic direct (`api.anthropic.com`) — not available via DeepInfra  
+**Image format:** 150 DPI PNG → JPEG 85% compression (to stay under Anthropic's 5MB image limit)  
+**max_tokens:** 8192 (required — pages 9 and 10 exceeded 4096)  
+**Cost:** $3.00 / $15.00 per 1M tokens (input/output)  
+**Control test cost (pages 7–32):** $0.49 total, avg $0.0195/page  
+**Full corpus estimate (150,000 pages):** $2,300–$2,900
+
+#### Strengths
+- **Best overall performer** — only model to pass all key validation tests cleanly
+- No repetition loops — page 30 (Tumbleweed letter) completes correctly
+- No hallucination substitution — "General Crerar" correct on page 4
+- No base64 image injection — clean HTML tables throughout
+- Completes page 10 Field Return of Officers (44-officer nominal roll) in full
+- Gets "Loon Plage" correct on page 32 (handwritten appendix)
+- Gets Montgomery letter (page 27) complete and accurate
+- Gets all report line names on trace map (page 19)
+- Clean output format — no think blocks, no padding artifacts, no inline CSS
+
+#### Failures & Known Issues
+
+| Issue | Page(s) | Severity | Description |
+|-------|---------|----------|-------------|
+| Token truncation on dense forms | 9, 10 | Medium | Truncated at 4096 default `max_tokens`. Fixed by setting 8192. Not a capability failure. |
+| Minor name misreads (handwriting) | 32 | Low | "D.F. MacLauchlan" vs "D.G.", "Hutchell" vs "Mitchell", "Wiegils" vs "Megill". Operational content intact. |
+| Superscript misreads | 7 | Low | "wome 25 pdrs fell short" → "some"; "morep at 2317" → missing superscript "hrs". |
+
+#### Key Validation Checks (Pass/Fail)
+
+| Check | Source of Failure on Other Models | Result |
+|-------|----------------------------------|--------|
+| Page 4 — "General Crerar" | olmOCR hallucination | ✅ Pass |
+| Page 10 — 44-officer nominal roll complete | Flash abandons after header | ✅ Pass |
+| Page 19 — trace map report lines | Qwen infinite image loop | ✅ Pass |
+| Page 27 — Montgomery letter complete | olmOCR fails | ✅ Pass |
+| Page 30 — no repetition loop | olmOCR and Qwen loop | ✅ Pass |
+| Page 32 — "Loon Plage" in header | olmOCR "Leuven", Flash "Koon" | ✅ Pass |
+| No base64 injection in Remarks | Gemini Pro failure | ✅ Pass |
+| No `<think>` block leakage | Flash and Pro failure | ✅ Pass |
+
+#### Verdict
+**Best model tested. Current recommendation for production OCR.** The cost ($2,300–$2,900 for full corpus) is within budget ceiling. The only structural issue is that it requires direct Anthropic API access — it cannot be run via DeepInfra's OpenAI-compatible endpoint for vision tasks.
+
+---
+
+## 4. Cross-Model Comparison by Page Type
+
+| Page Type | Winner | Runner-up | Avoid |
+|-----------|--------|-----------|-------|
+| Typed narrative diary entries | Claude 4.6 | Gemini Pro | Qwen |
+| Typed appendix forms | Claude 4.6 | Gemini Pro | Qwen |
+| Complex dual-page forms (nominal rolls) | Claude 4.6 | Qwen (only strength) | Flash |
+| Message forms (dense typed, grey paper) | Claude 4.6 | Gemini Pro | Qwen (S→3 errors) |
+| Artillery / DF task tables | Claude 4.6 | Gemini Pro | Qwen (swaps entries) |
+| Hand-drawn maps / traces | Claude 4.6 | Gemini Pro | Qwen (image loop) |
+| Handwritten narrative | olmOCR | Claude 4.6 | Flash (critical failure) |
+| Printed formal documents | Claude 4.6 | Gemini Pro | olmOCR |
+| Regimental journal / illustrated pages | Claude 4.6 | Gemini Pro | olmOCR (loops) |
+| Signing pages / simple forms | All adequate | — | — |
+
+---
+
+## 5. Known Failure Modes (All Models)
+
+### 5.1 Repetition / Infinite Loop
+**Affected models:** olmOCR (rare), Qwen3 (frequent — ~40% of complex pages)  
+**Trigger:** Ambiguous multi-column text, complex layouts, or anything that breaks the model's parsing flow  
+**Symptom:** Model gets stuck on a phrase and repeats it for the rest of the token budget  
+**Detection:** Output length disproportionately long; repeated n-grams in output  
+**Fix:** Post-processing loop detection (count repeated 10-gram occurrences); flag for human review
+
+### 5.2 Hallucination from Training Priors
+**Affected models:** olmOCR confirmed; presumed risk on all small/medium models  
+**Example:** "General Montgomery" substituted for "General Crerar" — statistically more probable completion pulled from training weights  
+**Danger:** Output looks completely plausible; undetectable without ground truth or image comparison  
+**Mitigation:** Use larger models with stronger instruction-following; provide Stacey/reference context to enable contradiction detection
+
+### 5.3 Token Truncation
+**Affected models:** Flash (systematic), Pro (on complex forms), Claude 4.6 (at default 4096 limit)  
+**Trigger:** Dense pages, long narrative entries, complex forms with many rows  
+**Fix:** Set `max_tokens: 8192` for all models; detect truncation by checking if output ends mid-sentence
+
+### 5.4 Base64 Image Injection
+**Affected models:** Gemini Pro  
+**Trigger:** Handwritten marks in Remarks column (circled initials, check marks)  
+**Symptom:** `<img src="data:image/png;base64,...">` garbage strings embedded in HTML table cells  
+**Fix:** Post-processing regex strip of any `<img>` tags containing base64 data
+
+### 5.5 `<think>` Block Leakage
+**Affected models:** Gemini Flash, Gemini Pro  
+**Symptom:** Model reasoning chains appear in output (verbose internal monologue about how to interpret the page)  
+**Fix:** Post-processing strip of `<think>...</think>` blocks
+
+### 5.6 Systematic Character Substitution on Degraded Scans
+**Affected models:** Qwen3 (severe), Flash (moderate)  
+**Known substitution clusters:**
+- `S` → `3` (e.g., "3ASK R" for "SASK R", "GUARD3" for "GUARDS")
+- `Z` → `S` (e.g., "PS FAUST" for "PZ FAUST")
+- `I` → `J` (e.g., "J. See" for "I Sec")
+  **Fix:** Post-processing correction dictionary for known military vocabulary and unit names
+
+### 5.7 Handwriting — Fundamental Failure
+**Affected models:** Gemini Flash (critical), Qwen3 (severe)  
+**Safe models for handwriting:** olmOCR (best), Claude Sonnet 4.6 (good)  
+**Recommendation:** Route handwritten pages to olmOCR or Claude; do not use Flash or Qwen
+
+---
+
+## 6. Cost Analysis
+
+| Model | Input (per 1M tokens) | Output (per 1M tokens) | Avg cost/page | Full corpus (150k pages) |
+|-------|-----------------------|------------------------|---------------|--------------------------|
+| olmOCR (DeepInfra) | ~$0.20 | ~$0.20 | ~$0.00039 | **~$60** |
+| Gemini Flash (DeepInfra) | $0.10 | $0.40 | ~$0.003 | **~$450** |
+| Gemini Pro (DeepInfra) | $1.25 | $10.00 | ~$0.018 | **~$2,700** |
+| Claude Sonnet 4.6 (Anthropic) | $3.00 | $15.00 | ~$0.0195 | **~$2,900** |
+| Qwen3-VL-30B (DeepInfra) | Variable | Variable | N/A | **Disqualified** |
+
+**Budget ceiling:** $2,000–$5,000 (confirmed acceptable)  
+**olmOCR was the original baseline** — $60 for the full corpus, essentially unlimited for prototype work with $20 loaded  
+**Claude Sonnet 4.6** sits comfortably within budget at ~$2,900 for full corpus re-OCR
+
+### Image Rendering
+- PDF pages rendered at **150 DPI** via Poppler (`pdftoppm`)
+- PNG output compressed to **JPEG 85%** to stay under Anthropic's 5MB per-image limit
+- Quality tested at page 32 (handwriting) — 150 DPI JPEG 85% produced the Claude results above
+- No quality regression observed vs PNG at this compression level
+
+---
+
+## 7. Architecture Decisions
+
+### 7.1 Two-Pass Pipeline is Correct
+**Question raised:** Is it wasteful to OCR in step 1 then run an LLM again in step 3?  
+**Answer:** No. Step 1 is vision work (image → text). Step 3 is reasoning work (text → structured JSON). Different tasks, different models, different costs.
+- Step 1 sends PNG images — expensive per page, requires visual understanding
+- Step 3 reads clean OCR text — cheap input, no image involved
+- Keeping them separate means if the extraction schema changes, you re-run step 3 only against already-OCR'd MD files. You never re-OCR.
+
+### 7.2 Why Not Collapse Into One Step
+Collapsing step 1 and step 3 would mean paying vision token rates for every field extraction on every page. It would also make re-runs of the extraction schema (which will happen as the project matures) require re-OCRing everything. The two-pass is the economical choice.
+
+### 7.3 Gemini Pro vs olmOCR Hybrid
+Three options were evaluated:
+
+**Option A — Gemini Pro everywhere**  
+Run Gemini Pro on all pages. Post-process to strip `<think>` blocks and base64 injections. Best accuracy, moderate cost.
+
+**Option B — Hybrid routing**
+- Classifier identifies page type first
+- Gemini Pro (or Claude) for: diary narrative pages, message forms, typed appendices, maps/traces
+- olmOCR as fallback for pages where Gemini Pro is known to inject base64 (handwritten marks in Remarks column)
+
+**Option C — olmOCR + LLM correction pass**  
+Run olmOCR first (fast, cheap). Flag low-confidence pages. LLM correction pass using Gemini Pro or Claude with Stacey/reference context injected.
+- Input to correction is text-only (no image) — costs ~$1.25/1M tokens, not vision rates
+- Stacey's Victory Campaign (~330,000 tokens) can be injected in full given Gemini Pro's 1M context window
+- Correction model can catch loops, garbled text, broken sentences — but cannot catch plausible hallucinations (e.g., Montgomery for Crerar) without seeing the image
+
+**Current decision:** Claude Sonnet 4.6 directly is the cleanest option given it outperforms the hybrid on all page types tested.
+
+### 7.4 pgvector over Chroma DB
+Deliberate architectural choice. Chroma is a pure vector store — semantic similarity only. pgvector inside Postgres enables a single SQL query combining:
+- Semantic similarity (vector search)
+- Date range filters
+- Unit ID filters
+- PostGIS spatial queries (`ST_DWithin` — "find events within 5km of this grid reference")
+
+This spatial query capability is the architectural moat. Chroma cannot do this. Adding PostGIS to a Chroma pipeline would require a separate system and application-layer joins.
+
+### 7.5 Document Classification
+Two-pass classification approach:
+
+**Pass 1 — Free, no API (OCR quality scoring):**  
+Score each page's OCR output for alpha-ratio and recognisable text patterns. Classifies ~80% of pages without any API call.
+
+**Pass 2 — Vision LLM on ambiguous pages (~20%):**  
+Llama 3.2 Vision or Qwen2.5-VL on DeepInfra. One API call per ambiguous page, tiny output (classification label + confidence JSON).  
+Estimated cost: $2–5 for full 150,000-page corpus.
+
+---
+
+## 8. Key Principles
+
+- **Sources problem vs prompt problem:** Low-quality OCR output from war diary narrative pages is partly a sources problem — the diary itself is officer-centric. The solution is broader document type coverage (patrol reports, sitreps, message logs, Part 1 Orders), not better prompting.
+- **olmOCR hallucinations cannot be fixed by prompting.** A 7B OCR-fine-tuned model does not reliably follow "transcribe verbatim" instructions when it encounters statistically probable alternative completions.
+- **Handwriting routing is essential.** Flash and Qwen are not fit for purpose on cursive handwritten pages. olmOCR and Claude 4.6 must handle these.
+- **Bad grids silently poison PostGIS.** Extraction accuracy in step 3 matters more than cost — a wrong grid reference goes into the database looking correct.
+- **The viewer MD and the JSON outputs serve different consumers.** Viewer MD = display artifact for Jonesy QA and Project '44 frontend. Step 3 JSONs = database inputs. Both are necessary and neither replaces the other.
+- **Transkribus is not viable at scale.** ~€10,000–12,000 for 150,000 pages vs ~$60 for olmOCR or ~$2,900 for Claude. The accuracy premium does not justify the cost delta.
+
+---
+
+## 9. Open Questions & Next Tests
+
+| Question | Priority | Notes |
+|----------|----------|-------|
+| Claude 4.6 vs Llama 70B for step 3 extraction | High | Step 3 currently uses Llama for narrative/position extraction. Claude likely more accurate on ambiguous table rows and grid references. Run same diary pages through both and compare `_positions.json` output. |
+| `anthropic/claude-4-sonnet` via DeepInfra for vision | Medium | Untested — Claude 4 (not 4.6) is available on DeepInfra. May support image input unlike Claude 3.7 Sonnet. Would enable single-API approach if it matches Claude 4.6 quality. |
+| LLM correction pass on olmOCR output | Medium | Gemini Pro or Claude as post-processor on olmOCR text, with Stacey context injected. Can catch loops and garbled text but not plausible hallucinations. Worth quantifying the accuracy delta vs direct Claude OCR. |
+| 100 DPI vs 150 DPI JPEG 85% | Low | Test page 32 (handwriting) at 100 DPI PNG vs current 150 DPI JPEG 85%. Determine if quality regression is measurable. 100 DPI PNG would remove Pillow dependency and eliminate compression artifacts. |
+| Transkribus small-batch test | Low | Test 50 handwritten appendix pages to quantify accuracy vs olmOCR. Not viable at corpus scale but useful data point for handwriting quality ceiling. |
--- a/notes/wardiary_copilot_lessons-learned.md
+++ b/notes/wardiary_copilot_lessons-learned.md
@@ -0,0 +1,335 @@
+# Project '44 — OCR Model Lessons Learned
+*War Diary OCR Pipeline | Calgary Highlanders Sep 44 Control Test*
+*Compiled May 2026*
+
+---
+
+## 1. Test Corpus
+
+All model evaluations used the same 32-page control PDF derived from the Calgary Highlanders War Diary, September 1944. The 32 pages were selected to cover the full range of document types present in a typical war diary file:
+
+| Document Type | Pages in Test | Notes |
+|---|---|---|
+| Typed narrative diary entries (HTML table) | 7 | Core data pages — Place / Date / Hour / Summary / Remarks |
+| Message forms (Army Form C2136) | ~8 | Dense typed text, some on grey or degraded paper |
+| Artillery / DF task tables | 2 | Complex tabular data with grid references |
+| Field Return of Officers (dual-page form) | 1 | Complex nominal roll — 42 named officers |
+| Field Return of Other Ranks | 1 | Complex dual-form, dense |
+| Movement orders | 2 | Mixed typed + handwritten annotation |
+| Hand-drawn maps / traces / sketch maps | 4 | APIS trace, disposition sketch, report lines, DMA order |
+| Handwritten narrative appendix | 1 | Cursive, "Enroute Loon Plage to Antwerp" |
+| Regimental journal (The Glen) | 3 | Cover, CO's address (degraded), newsletter pages |
+| Administrative pages | 3 | Cover, instructions, appendix index, signing page |
+
+---
+
+## 2. Models Tested
+
+| Model | API / Provider | Status | Run Method |
+|---|---|---|---|
+| olmOCR 2-7B (`allenai/olmOCR-2-7B-1025`) | DeepInfra | ✅ Completed | `step1_ocr_wardiaries_olmocr.py` |
+| Gemini 2.0 Flash (`google/gemini-2.0-flash`) | DeepInfra | ✅ Completed | `ocr_model_comparison.py` |
+| Gemini 2.5 Flash (`google/gemini-2.5-flash`) | DeepInfra | ✅ Completed | `ocr_model_comparison.py` |
+| Gemini 2.5 Pro (`google/gemini-2.5-pro`) | DeepInfra | ✅ Completed | `ocr_model_comparison_anthropic.py` |
+| Qwen3-VL-30B (`Qwen/Qwen3-VL-30B-A3B-Instruct`) | DeepInfra | ✅ Completed (disqualified) | `ocr_model_comparison.py` |
+| Claude 3.7 Sonnet (`anthropic/claude-3-7-sonnet-latest`) | DeepInfra | ❌ 404 — model not available on DeepInfra | `ocr_model_comparison.py` |
+| Claude Sonnet 4.6 (`claude-sonnet-4-6`) | Anthropic API direct | ✅ Completed | `step1_ocr_wardiaries_claude-4-6.py` |
+
+### Key Lesson: Claude via DeepInfra Does Not Work
+
+`claude-3-7-sonnet-latest` was routed through DeepInfra's OpenAI-compatible endpoint. Every page returned `404 Not Found`. DeepInfra does not host Anthropic models. All Claude testing must use the Anthropic API directly (`api.anthropic.com/v1/messages`) with `x-api-key` header and `anthropic-version: 2023-06-01`.
+
+---
+
+## 3. Overall Model Rankings
+
+### Typed Diary Content (narrative pages, message forms, movement orders)
+
+| Rank | Model | Notes |
+|---|---|---|
+| 1 | **Gemini 2.5 Pro** | Best overall. No paraphrase contamination, no factual hallucination. Wins ~25/32 pages by category. |
+| 2 | **Claude Sonnet 4.6** | Strong on structured diary tables. Outputs clean HTML tables consistently. Not yet formally scored against control test on all 32 pages. |
+| 3 | **olmOCR 2-7B** | Reliable baseline. Consistent output structure. See failure modes below. |
+| 4 | **Gemini 2.5 Flash / 2.0 Flash** | Identical outputs. Viable for typed pages; fails on handwriting. Not a Pro replacement. |
+| 5 | **Qwen3-VL-30B** | **Disqualified.** Catastrophic failure modes. Do not use. |
+
+---
+
+## 4. Model-by-Model Findings
+
+---
+
+### 4.1 olmOCR 2-7B
+
+**Overall verdict:** Reliable baseline. Never enters infinite loops, never silently fails on typed pages. Failure modes are predictable and lower-severity than Gemini Flash or Qwen. Currently used as the production step 1 model pending full Claude 4.6 evaluation.
+
+**Strengths:**
+- Consistent structured output (HTML tables where appropriate, plain text elsewhere)
+- No base64 embedding or `<think>` block leakage
+- Good performance on clean typed diary narrative pages
+- Handles most message forms adequately
+- Timeout is the only failure mode on complex forms (deterministic — not silence)
+
+**Failure modes:**
+
+| Failure | Page(s) | Severity | Example |
+|---|---|---|---|
+| Paraphrase contamination | 2, 5 | High | "of the force" → "of the army as a whole"; verbatim accuracy compromised on boilerplate |
+| Factual hallucination | 4 | **Critical** | "General Montgomery" instead of "General Crerar" — wrong person at the Dieppe march past |
+| Meaning reversal | 6 | **Critical** | "roads were reported clear" → "roads were reported closed" — flips historical record |
+| Artillery garbling | 7 (p.30) | High | "some 25 pdrs fell short" → "some 20 plrs fell asleep" — loses calibre and military meaning |
+| Timeout on complex forms | 9 | High | Field Return of Other Ranks times out at 180s. Complex multi-part forms exceed limit. |
+| Handwriting degradation | 32 | High | "Loon Plage" → "Caen Place"; "MacLauchlan" → "G.H. Launchlan"; distances wrong |
+| Reversed initials | 8 (p.51) | Low | "R. L. Ellis" → "A. R. L. Ellis" |
+| Nominal roll errors | 10 | Medium | CO name garbled; table structure breaks down mid-form |
+
+**Post-processing notes:**
+- olmOCR output is the cleanest raw text of all tested models — no stripping required
+- Timeout is the only deterministic failure; retry with smaller page batches
+
+---
+
+### 4.2 Gemini 2.0 Flash and Gemini 2.5 Flash
+
+**Overall verdict:** Both models produced **identical outputs** on every page — no meaningful difference between them for this document type. Viable for clean typed appendix pages as a supplementary model. Not a replacement for olmOCR. Not fit for handwritten content.
+
+**Strengths:**
+- Good to excellent on typed message forms, movement orders, typed narrative pages
+- Reasonable table structure preservation
+- Generally faster than Pro
+
+**Failure modes:**
+
+| Failure | Page(s) | Severity | Example |
+|---|---|---|---|
+| `<think>` block leakage | All | Medium | Reasoning chain present in output — must be stripped post-processing |
+| Truncation on dense multi-entry pages | 6 (doc p.7) | **High** | 6 Sep entry complete; entire 7 Sep entry (Loon Plage) dropped |
+| Truncation on long narrative pages | 29 (CO's address) | Medium | Final 1.5 paragraphs lost |
+| Truncation on sparse pages | 51 (signing) | Medium | Signature block at page bottom missed |
+| Handwriting total failure | 32 | **Critical** | "Leen Cire des. thae Laucklan", "Euront pour Koon plage" — largely unintelligible |
+| S→3 substitution | 23, 24+ | Medium | "SASK R" → "3ASK R", "SOUTH" → "30UTH", "GUARDS" → "GUARD3" |
+| Z→S substitution | 26 | Medium | "PZ FAUST" → "PS FAUST" |
+| I→J confusion | 17 | Medium | "I Sec" (Intelligence Section) → "J. See" |
+| Redaction marker garbling | 21 | Low | "XXXXXXX" → "Kaxxkuka" |
+| Complex dual-page form failure | 10 | **Critical** | Abandons after header; 44 named officers completely lost |
+| Diagram/trace partial reads | 14 | Medium | Unit names split across lines; edge text garbled with no uncertainty flag |
+
+**Recommendation:** Viable as a supplementary model for document classification and typed-content extraction only. Route handwritten pages and complex forms away from Flash.
+
+---
+
+### 4.3 Gemini 2.5 Pro
+
+**Overall verdict:** Clear winner on typed content and best-performing general model tested to date. Significantly better than olmOCR on narrative diary pages, message forms, maps, and handwritten content. Requires mandatory post-processing to strip `<think>` blocks and embedded base64 data.
+
+**Strengths:**
+- Wins typed narrative diary entries — no paraphrase contamination, no hallucination
+- Wins on maps and hand-drawn traces (only model that successfully extracts text from these)
+- Best performance on handwritten narrative appendix (page 32)
+- Correctly reads degraded/compressed typography better than any other model
+- Gets original typos and spelling errors correctly (verbatim faithfulness)
+- Gets superscript annotations, handwritten margin notes, signatures
+- Gets French place names accurately ("Loon Plage" → correct on all attempts)
+
+**Strongest pages vs olmOCR:**
+- Page 4: Correct "General Crerar" where olmOCR hallucinated "General Montgomery"
+- Page 5: Correct place names and diarist quotes where olmOCR paraphrased and garbled
+- Page 6: Correct "roads were reported clear" where olmOCR reversed to "closed"
+- Page 7: "25 pdrs fell short" correct where olmOCR produced "20 plrs fell asleep"
+- Page 32 (handwritten): Correct "Loon Plage 1368", "MacLauchlan", "Borsbeek 782933"
+
+**Failure modes:**
+
+| Failure | Page(s) | Severity | Example |
+|---|---|---|---|
+| `<think>` block leakage | All | Medium | Longer and more verbose than Flash; must be stripped |
+| Embedded base64 image data | 6, 7 | **High** | Hundreds of lines of `data:image/png;base64` embedded in Remarks column; file size explodes |
+| HTML verbosity (`<br>` proliferation) | 4, 7 | Low | Dozens of empty `<br>` tags in table cells |
+| Truncation on complex forms | 10, 16 | High | Field Return of Officers and Movement Order cut off mid-page (token limit) |
+| Page 29 degradation | 29 | Medium | Dense two-column CO's address: "Docompris", "noathing" — quality below usual standard |
+
+**Critical post-processing required:**
+```python
+import re
+
+def clean_gemini_output(text):
+    # Strip think blocks
+    text = re.sub(r'<think>.*?</think>', '', text, flags=re.DOTALL)
+    # Strip embedded base64 image data
+    text = re.sub(r'data:image/[^;]+;base64,[A-Za-z0-9+/=\n]+', '[IMAGE]', text)
+    # Strip excessive <br> padding in table cells
+    text = re.sub(r'(\s*<br>\s*){5,}', '\n', text)
+    return text.strip()
+```
+
+---
+
+### 4.4 Qwen3-VL-30B
+
+**Overall verdict: Disqualified. Do not use.**
+
+Multiple distinct catastrophic failure modes make Qwen3-VL-30B unsuitable for any production role in this pipeline. Even its single strength (nominal roll extraction on page 10) is not sufficient to justify inclusion.
+
+**Failure modes:**
+
+| Failure | Pages | Severity | Example |
+|---|---|---|---|
+| Infinite loop / token flood | 1, 17, 22, 24, 25, 27+ | **Critical** | "Bde Bde Bde Bde..." repeated thousands of times; or `<tr><td>2</td></tr>` ×hundreds |
+| Silent image placeholder failure | 14, 15, 19, 27, 28, 29, 31 | **Critical** | `<div class="image"><img/></div>` — zero text extracted. Affects ~40% of pages including clean printed pages |
+| S→3 systematic substitution | 23+ | High | "GUARDS" → "GUARD3", "SCOT" → "3COT", "3 SASK R", "FU3 MR" |
+| "LOON-PLACE" misread | 23, 32 | High | "LOON-PLAGE" → "LOON-PLACE" — wrong on the key Calgary Highlanders location throughout |
+| Numeric errors on intel content | 26 | High | "559 GR 331 inf div" → "859 GR 531 inf div" — corrupts unit identification |
+
+**Qwen's one notable strength:**
+- Page 10 (Field Return of Officers nominal roll) — Qwen correctly extracted all 42 officer names with ranks, corps, and appointments. This is the most structurally complex table in the test and Qwen outperformed every other model on this specific page. However this does not offset the ~40% silent failure rate.
+
+---
+
+### 4.5 Claude Sonnet 4.6
+
+**Status:** Successfully run via Anthropic API direct. Full 32-page comparison file exists (`step1_Calgary-Highlanders_War-Diary_Sep44_claude-4-6.md`). Formal page-by-page scoring against control test not yet completed.
+
+**What we know from the full diary run:**
+- Produces clean HTML tables for diary pages — same format as olmOCR
+- Output structure consistent and correct: `## Page N` markers, HTML tables where applicable
+- Image compression pipeline established: PDF → PNG (pdftoppm, 150 DPI) → JPEG (85% quality) → base64 → Anthropic API. JPEG compression required because PNG at 150 DPI is ~6-7 MB (over Anthropic's 5 MB image limit); JPEG 85% brings it to ~0.8-1.2 MB with no meaningful OCR detail loss.
+- Token usage logged per page: ~$3.00/1M input, $15.00/1M output
+
+**Technical parameters confirmed working:**
+- Endpoint: `https://api.anthropic.com/v1/messages`
+- Model ID: `claude-sonnet-4-6`
+- Headers: `x-api-key`, `anthropic-version: 2023-06-01`
+- Response path: `data["content"][0]["text"]`
+- `max_tokens: 8192` (double the extraction step — vision work needs more headroom)
+
+**Next step:** Formal page-by-page comparison of `step1_Calgary-Highlanders_War-Diary_Sep44_claude-4-6.md` against the Gemini Pro and olmOCR control test outputs.
+
+---
+
+### 4.6 Claude 3.7 Sonnet (via DeepInfra) — Complete Failure
+
+All 32 pages returned `[OCR FAILED — HTTP error: 404 Client Error: Not Found for url: https://api.deepinfra.com/v1/openai/chat/completions]`.
+
+**Root cause:** Anthropic models are not hosted on DeepInfra. The model ID `anthropic/claude-3-7-sonnet-latest` was passed to DeepInfra's OpenAI-compatible endpoint, which returned 404 for all requests.
+
+**Resolution:** Rewrote `step1_ocr_wardiaries_claude-4-6.py` to call the Anthropic API directly. Do not attempt to use any Anthropic model via DeepInfra.
+
+---
+
+## 5. Cross-Model Rankings by Page Type
+
+| Page Type | Best Model | Runner-up | Notes |
+|---|---|---|---|
+| Typed narrative diary entries | Gemini 2.5 Pro | Claude 4.6 (TBD) | olmOCR paraphrase/hallucination issues |
+| Typed appendix forms | Gemini 2.5 Pro | olmOCR | Qwen S→3 errors on grey pages |
+| Complex officer / rank nominal rolls | Qwen3 | Gemini 2.5 Pro | Qwen's only standout; truncation in Pro |
+| Message forms (dense typed) | Gemini 2.5 Pro | Qwen3 | olmOCR not fully sampled |
+| Artillery / DF task tables | Gemini 2.5 Pro | Qwen3 | Qwen swaps some entries |
+| Hand-drawn maps / traces | Gemini 2.5 Pro | — | Qwen fails, olmOCR fails |
+| Handwritten narrative | Gemini 2.5 Pro | olmOCR | Flash unintelligible; Qwen partially attempts |
+| Printed formal documents | Gemini 2.5 Pro | — | olmOCR and Qwen both fail page 27 (Montgomery) |
+| Illustrated / journal pages | Gemini 2.5 Pro | — | Qwen and olmOCR fail The Glen cover |
+| Signing pages / simple forms | Gemini 2.5 Pro / Qwen | — | All models adequate |
+
+---
+
+## 6. Character Substitution Patterns Found
+
+These substitution errors are scan/font artefacts and appear consistently across multiple models on degraded or grey-background pages:
+
+| Substitution | Affected Models | Pages | Correction Rule |
+|---|---|---|---|
+| `S` → `3` | Gemini Flash, Qwen | 23, 24+ | Post-processing rule against known military vocabulary (SASK R, SCOTS, GUARDS, SOUTH, FUS MR) |
+| `Z` → `S` | Gemini Flash | 26 | "PZ FAUST" → "PS FAUST" |
+| `I` → `J` | Gemini Flash | 17 | "I Sec" → "J. See" |
+| `F` → `P` | Qwen | 11 | "5 Cdn Fd Regt" → "5 Cdn Pd Regt" |
+| `O` → `0` | Qwen | 8 | "CA(O)" → "CA(0)" |
+
+**Recommendation:** Build a post-processing correction dictionary for known military unit names, abbreviations, and vocabulary. High-confidence substitutions can be corrected automatically; low-confidence ones should be flagged for human review.
+
+---
+
+## 7. Key Decisions Made
+
+### OCR Architecture
+- **claude-4-6 via Anthropic API** is the current production step 1 model for diary narrative pages
+- **olmOCR via DeepInfra** is the fallback / comparison baseline
+- Two-pass architecture (vision OCR → text extraction) is deliberate and correct — collapsing into one step would pay vision token rates for all field extraction
+
+### Image Processing
+- **150 DPI** is sufficient for this document type at typical war diary scan quality
+- **JPEG 85% compression** is required before sending to Anthropic API (5 MB limit); no meaningful OCR detail loss at this quality level
+- PNG at 150 DPI produces ~6-7 MB images — always exceeds Anthropic's limit
+
+### Output Format
+- **HTML tables** for diary pages (Place / Date / Hour / Summary / Remarks structure)
+- **Plain text** for all other page types
+- **`## Page N`** section headers in output MD — required by step 2 (viewer) and step 3 (extraction)
+
+### Pipeline Routing (proposed)
+- **Misc documents (~90% of pages):** Step 1 → Step 3 immediately. No human verification needed.
+- **Narrative diary pages (~10% of pages):** Step 1 → Step 2 viewer → human verification → Step 3. Only verified text feeds the descendant report.
+
+### Models Definitively Ruled Out
+- **Qwen3-VL-30B:** Do not use. ~40% silent failure rate, infinite loops, systematic character errors.
+- **Gemini Flash (either version):** Not for handwriting, not for complex forms.
+- **Any Anthropic model via DeepInfra:** Returns 404. Use Anthropic API directly.
+
+---
+
+## 8. OCR Correction Experiment (Gemini Pro + Stacey Context)
+
+**Test:** Attempted automated OCR correction by passing olmOCR output to Gemini 2.5 Pro along with contextual reference material (C.P. Stacey's *The Victory Campaign*).
+
+**Pages tested:** 4, 5, 6, 7, 30, 32
+
+**Result:** 0 corrections suggested across all 6 pages.
+
+**Analysis:** One of two interpretations:
+1. The olmOCR output on these pages was accurate enough that Gemini Pro found nothing to correct — a positive signal for olmOCR quality on the pages selected.
+2. The correction prompt was insufficiently targeted to surface the known errors (paraphrasing, meaning reversals, hallucinations) that were identified in the manual comparison.
+
+**Conclusion:** Automated correction via a second-pass LLM is not yet proven. Manual human review (via the OCR viewer) remains the quality gate for narrative pages.
+
+---
+
+## 9. Cost Comparison
+
+Based on DeepInfra pricing and Anthropic API pricing as at May 2026, and typical token counts for this document type (~1,500 input tokens per page for text, plus ~1,200 tokens for image via vision models):
+
+| Model | Provider | Input $/1M | Output $/1M | Relative Cost | Notes |
+|---|---|---|---|---|---|
+| olmOCR 2-7B | DeepInfra | ~$0.07 | ~$0.07 | 1× (baseline) | Cheapest. Consistent. |
+| Gemini 2.0 Flash | DeepInfra | ~$0.15 | ~$0.60 | ~4× | Same output as 2.5 Flash |
+| Gemini 2.5 Flash | DeepInfra | ~$0.15 | ~$0.60 | ~4× | No gain over 2.0 Flash |
+| Qwen3-VL-30B | DeepInfra | ~$0.20 | ~$0.88 | ~6× | Disqualified |
+| Gemini 2.5 Pro | DeepInfra | ~$1.25 | ~$10.00 | ~50× | Best accuracy; think blocks inflate token count |
+| Claude Sonnet 4.6 | Anthropic | $3.00 | $15.00 | ~100× | Best structured output; no post-processing needed |
+
+**For 150,000-page scale:**
+- olmOCR at scale: very low cost; accuracy limitations are the constraint
+- Gemini Pro at full corpus: materially expensive; hybrid approach preferred
+- Claude 4.6 at full corpus: highest cost but may be justified for narrative diary pages (only ~10% of pages)
+
+**Recommended approach at scale:** olmOCR for misc documents (90%); Claude 4.6 for narrative diary pages (10%). This concentrates the expensive model where accuracy matters most for the commercial product.
+
+---
+
+## 10. Known Open Questions
+
+1. **Formal Claude 4.6 scoring:** `step1_Calgary-Highlanders_War-Diary_Sep44_claude-4-6.md` exists but has not been formally scored against the control test comparison. This should be the next evaluation step.
+
+2. **Step 3 model choice:** Step 3 (extraction) currently uses `meta-llama/Meta-Llama-3.1-70B-Instruct` (olmOCR variant) and `claude-sonnet-4-6` (claude variant). Llama 70B has not been formally evaluated for grid reference extraction accuracy. Bad grids silently poison PostGIS downstream. Formal comparison of Llama vs Claude on `_positions.json` output quality is pending.
+
+3. **Handwriting routing:** No model has been confirmed for production-quality handwriting transcription. Gemini 2.5 Pro is the best tested but has truncation issues on complex pages. A dedicated handwriting model (e.g., Google Document AI, Azure Form Recognizer) has not been evaluated.
+
+4. **Complex form failures:** Field Return of Officers (nominal roll) causes truncation in Gemini Pro and timeouts in olmOCR. Qwen handled it but is otherwise disqualified. No production-viable solution confirmed for this page type.
+
+5. **`<think>` block suppression:** Gemini Pro's think blocks can be stripped post-processing, but they also inflate token counts and costs. Testing a `thinkingBudget: 0` parameter (if available via DeepInfra) is pending.
+
+---
+
+*Document covers: olmOCR 2-7B, Gemini 2.0 Flash, Gemini 2.5 Flash, Gemini 2.5 Pro, Qwen3-VL-30B, Claude 3.7 Sonnet (failed), Claude Sonnet 4.6*
+*Test corpus: Calgary Highlanders War Diary Sep 44, 32-page control test*
+*Pipeline scripts: `step1_ocr_wardiaries_claude-4-6.py`, `step1_ocr_wardiaries_olmocr.py`, `ocr_model_comparison.py`, `ocr_model_comparison_anthropic.py`*
+
--- a/notes/wardiary_pipeline_notes.md
+++ b/notes/wardiary_pipeline_notes.md
@@ -0,0 +1,96 @@
+# Project '44 — Pipeline Session Notes
+*May 2026 | Working log — pipeline architecture & decisions*
+
+---
+
+## 1. Pipeline Overview
+
+| Step | Name | Input | Output | Model | Notes |
+|------|------|-------|--------|-------|-------|
+| 1 | OCR | PDF pages (PNG) | Raw MD with HTML tables | Claude 4.6 or olmOCR | Vision work — reads scanned images. Claude 4.6 significantly better than olmOCR on structured docs. |
+| 2 | Viewer MD | Step 1 MD | Clean viewer MD (no HTML) | Script (no LLM) | Display artifact only — for Jonesy QA and Project '44 frontend. Not a data input. |
+| 3 | Extraction | Step 1 MD (text only) | _narratives.json + _positions.json | Llama 70B (TBD: Claude?) | Text reasoning only — no vision needed. Re-runnable cheaply if schema changes. |
+
+---
+
+## 2. Why Two Passes (Step 1 + Step 3) Is Not Wasteful
+
+Step 1 is vision work (image → text). Step 3 is reasoning work (text → structured JSON). Fundamentally different tasks.
+
+- Step 1 sends PNG images to a vision-capable model — expensive per page, requires visual understanding of scanned documents.
+- Step 3 reads clean OCR text and extracts structured data (dates, grids, place names, categories). No image involved. Costs a fraction of step 1.
+- Keeping them separate means if the extraction schema changes (it will), you re-run step 3 only against already-OCR'd MD files. You never re-OCR.
+- Collapsing into one step would mean paying vision token rates for every field extraction on every page — more expensive and harder to prompt correctly.
+
+**Decision: Two-pass is the correct architecture. The separation is a cost advantage, not a cost problem.**
+
+---
+
+## 3. What Step 3 Actually Produces (and Why It's Needed)
+
+The viewer MD looks good — but step 3 is still necessary. The two outputs serve completely different consumers.
+
+### _narratives.json
+- Dated entries as discrete records: date, place, weather, full summary text.
+- Feeds the descendant report generation pipeline.
+- Without it, Claude has no way to query "what happened on 14 Sep 44" without reading the entire diary.
+- The viewer MD is prose — it cannot be queried or filtered by date.
+
+### _positions.json
+- Geo-extractable records with raw grid references, sheet refs, position categories (END_OF_DAY, SUBUNIT, ENEMY, PATROL, etc.), confidence levels, and context snippets.
+- Feeds the mapping layer — Postgres with pgvector + PostGIS.
+- Without it, there are no coordinates to plot on the map.
+
+**Viewer MD = display artifact. Step 3 JSONs = database inputs. Both necessary.**
+
+---
+
+## 4. Page Composition of a War Diary — The 300/30 Insight
+
+A typical war diary PDF of ~300 pages breaks down roughly as:
+
+| Document Type | Approx. Pages | Notes |
+|---------------|---------------|-------|
+| Narrative diary pages | ~30 (10%) | The HTML table pages with Place / Date / Hour / Summary. Need human verification before narrative extraction. |
+| Misc documents | ~270 (90%) | Message logs, intel reports, newsletters, Part 1 Orders, nominal rolls, appendices, sitreps, patrol reports, traces. |
+
+---
+
+## 5. Proposed Pipeline Routing
+
+Narrative pages and misc documents have different quality gates — they should not share the same queue.
+
+### Misc documents (~270 pages)
+- Step 1 (OCR) → straight to step 3 extraction immediately.
+- No human verification needed before extraction.
+- Feed intel reports, patrol data, message metadata into the RAG pipeline.
+
+### Narrative diary pages (~30 pages)
+- Step 1 (OCR) → step 2 viewer MD → Jonesy verifies → step 3 narrative extraction.
+- Only after human sign-off does the page go to step 3.
+- Ensures `_narratives.json` (which feeds descendant reports) is built on verified text.
+
+**Practical benefit: 90% of extraction work flows through immediately. The bottleneck only applies to the 30 pages that matter most for the commercial product.**
+
+---
+
+## 6. Open Question: Llama vs Claude for Step 3
+
+Step 3 currently uses `meta-llama/Meta-Llama-3.1-70B-Instruct`. May produce more errors on:
+
+- Ambiguous HTML table rows with missing or split dates
+- Grid reference parsing and 4-digit → 6-digit expansion
+- Correct position category assignment (END_OF_DAY vs SUBUNIT vs FRIENDLY)
+
+**To test:** Run step 3 on the same Calgary Highlanders pages with (a) Llama 70B and (b) Claude Haiku or Sonnet. Compare `_positions.json` output quality, especially grid confidence and category accuracy. Bad grids silently poison PostGIS, so accuracy matters more than cost here.
+
+---
+
+## 7. Key Principles Confirmed This Session
+
+- **OCR quality:** olmOCR is decent; Claude 4.6 is significantly better for structured historical documents.
+- **Two passes:** The two-pass architecture (vision OCR → text extraction) is deliberate and cost-effective.
+- **Step 3 purpose:** Step 3 is not redundant — viewer MD and JSON outputs serve completely different consumers.
+- **Page ratio:** ~10% narrative pages, ~90% misc documents. Most of the pipeline runs without waiting for human review.
+- **Quality gate:** Narrative pages need human verification before step 3. Misc pages do not.
+- **Data quality:** pgvector + PostGIS in a single SQL query is a core architectural advantage. Bad grids are hard to catch downstream — model accuracy in step 3 matters.