nathan/AI-Prototype
1
0
Fork 0
Code Pull Requests Packages Projects Activity

testin out different ocr outputs

Browse Source
This commit is contained in:
nathan
2026-05-16 09:57:37 -04:00
parent 95ca7efa44
commit f2ed4b18e1
77 changed files with 137498 additions and 350 deletions
Download Patch File Download Diff File Expand all files Collapse all files

237
LLM_OCR_Evaluations/gemini-flash_2.5.md Normal file
View File

@@ -0,0 +1,237 @@
# OCR Comparison Summary
## Gemini Flash vs Control Test — Calgary Highlanders War Diary, Sep 44
---
## Overall Assessment
Gemini 2.5 Flash and 2.0 Flash produced **identical outputs on every page** — no meaningful difference between the two models for this document type. The `<think>` block leakage and inline CSS issues noted in the partial test earlier were consistent throughout.
---
## Issues Log
### Truncation
---
#### Issue #1 — Truncation: Dense multi-entry pages
| | |
|---|---|
| **Page** | 6 (doc page 7) |
| **Severity** | High |
| **Description** | Both models truncate after the 6 Sep entry, dropping the entire 7 Sep entry including the Loon Plage operational passage. Substantive content lost. |
---
#### Issue #3 — Truncation: Sparse pages with low-positioned content
| | |
|---|---|
| **Page** | 51 (signing page) |
| **Severity** | Medium |
| **Description** | Entire signature block missed (`R.L. Ellis, Major, A/O.C. The Calgary Highlanders CA(O)`). Model processes empty space and stops before reaching content at bottom of page. |
---
#### Issue #16 — Truncation: Long narrative pages
| | |
|---|---|
| **Page** | 29 (C.O.'s address) |
| **Severity** | Medium |
| **Description** | Excellent accuracy on degraded text but drops final 1.5 paragraphs. Third confirmed truncation instance. Pattern is not purely length-based — page 30 of similar density did not truncate. |
---
### Handwriting & Superscript Errors
---
#### Issue #2a — `pårs` → `pdrs`
| | |
|---|---|
| **Page** | 27 (30 Sep entry) |
| **Severity** | Low–Medium |
| **Description** | Standard military abbreviation for *pounders* misread. Corrupts a common artillery term. |
---
#### Issue #2b — `theme` → `them.`
| | |
|---|---|
| **Page** | 27 (30 Sep entry) |
| **Severity** | Low |
| **Description** | Superscript/correction mark misread as a letter. |
---
#### Issue #2c — `could not whether` → `could not say whether`
| | |
|---|---|
| **Page** | 27 (30 Sep entry) |
| **Severity** | Low–Medium |
| **Description** | Handwritten superscript insertion *say* missed entirely. Meaning subtly changed. |
---
#### Issue #8 — Cursive `I`/`J` confusion
| | |
|---|---|
| **Page** | 17 |
| **Severity** | Medium |
| **Description** | `I Sec` (Intelligence Section) misread as `J. See`. Domain-specific routing term lost. Pattern likely recurs across annotated documents throughout the corpus. |
---
#### Issue #18 — Fundamental failure on cursive handwritten pages
| | |
|---|---|
| **Page** | 32 (handwritten narrative appendix) |
| **Severity** | **Critical** |
| **Description** | Flash output largely unintelligible on cursive handwriting. `Leen Cire des. thae Laucklan`, `Euront pour Koon plage`, `th officers coat pere Loobeeghe`. olmOCR substantially superior on this document type. **Flash is not fit for purpose on handwritten content.** |
---
### Character Substitution on Degraded Scans
---
#### Issue #12 — Systematic `S`→`3` substitution
| | |
|---|---|
| **Pages** | 23, 24, others |
| **Severity** | Medium |
| **Examples** | `3ASK R` (SASK R), `30UTH` (SOUTH), `GUARD3` (GUARDS) |
| **Description** | Scan/font artefact where certain `S` characters render as `3`. Consistent enough to warrant a post-processing correction rule, particularly for known unit names. |
---
#### Issue #14 — `S`→`3` in ambiguous location codes
| | |
|---|---|
| **Page** | 25 |
| **Severity** | Medium |
| **Description** | `BL-U30T` → `BL-USOT`. Same substitution pattern as Issue #12 in an already-ambiguous map reference or location code. |
---
#### Issue #15 — `Z`→`S` substitution
| | |
|---|---|
| **Page** | 26 |
| **Severity** | Medium |
| **Description** | `PS FAUST` → `PZ FAUST` (Panzerfaust). Extends the character confusion cluster from Issue #12. Corrupts a specific weapons identification in a PW intelligence report. |
---
### Document Type Failures
---
#### Issue #4 — Complete failure: Complex dual-page appendix forms
| | |
|---|---|
| **Page** | 10 (Field Return of Officers) |
| **Severity** | **Critical** |
| **Description** | Both models abandon transcription after the header, producing no table data. Entire Part A strength returns, Part B officer movements, Part C, and Part D nominal roll of **44 named officers** completely lost. Most severe structured-data failure mode in the test. |
---
#### Issue #6 — Floating annotations misassigned to table rows
| | |
|---|---|
| **Page** | 13 (DF Task Table) |
| **Severity** | Medium–High |
| **Description** | `Cancelled. Amstaten. 28: 1800. hr.` is a table-wide annotation but gets assigned to DF 1 and DF 3 rows in flat output. Risk of silent data corruption in downstream structured extraction. |
---
#### Issue #7 — Diagram/trace pages: fragmented labels and partial edge reads
| | |
|---|---|
| **Page** | 14 (brigade disposition diagram) |
| **Severity** | Medium |
| **Description** | Unit names split across lines (`CALG` / `HIGHRS`, `FD` / `COY`). Partially visible edge text transcribed as garbled fragments (`CET ISSUED IN`, `ONJUNCTION`) with no uncertainty flag. Silent partial reads are more dangerous than explicit illegibility markers. |
---
### Message Form Issues
---
#### Issue #5 — Message form structure not preserved
| | |
|---|---|
| **Pages** | 12, 20, 21, 22, 23, 24, 25, 26 |
| **Severity** | Low |
| **Description** | Handwritten signatures replaced with `[handwritten signature/initials]` placeholder. Content present but unstructured. Minor relative to other issues. |
---
#### Issue #9 — Message form title not recognised from partial scan
| | |
|---|---|
| **Pages** | 21, 26 |
| **Severity** | Low–Medium |
| **Description** | `MESSAGE FORM` rendered as `SSAGE FORM` / `MASSARE FORM` on edge-truncated scans. Affects document type classification. |
---
#### Issue #10 — Redaction markers rendered as nonsense strings
| | |
|---|---|
| **Page** | 21 |
| **Severity** | Low–Medium |
| **Description** | `XXXXXXX` overtyping rendered as `Kaxxkuka` rather than `[redacted]` or `[illegible]`. Will confuse downstream parsers. |
---
#### Issue #11 — Systematic recipient line garbling on message forms
| | |
|---|---|
| **Pages** | All Army Form C2136 pages |
| **Severity** | Medium |
| **Description** | `TO R H C R DE MAIS CALG HIGHRS` consistently garbled. Worst instance: `CALO NIGHTS`. Root cause is multi-column header box layout being read linearly. Affects unit identification across multiple appendices. |
---
#### Issue #13 — Degrading accuracy on poor-quality scans
| | |
|---|---|
| **Page** | 26 |
| **Severity** | High |
| **Description** | Non-linear performance drop on faded scans — `MATSUHILLING` (MAJ SCHILLING), `MARZAM` (MERXEM), `CALO NIGHTS` (CALG HIGHRS). Model performance on message forms is scan-quality dependent in an unpredictable way. |
---
### Other
---
#### Issue #17 — Degraded newsprint misreads
| | |
|---|---|
| **Page** | 31 (The Glen news sheet) |
| **Severity** | Low |
| **Description** | `FRANGE` (FRANCE), `COLUMES` (COLUMNS), `AIR` (AFTER). Honest misreads on genuinely difficult source material rather than model failure. |
---
## Pattern Summary
Three distinct failure categories emerge:
| Category | Issues | Notes |
|---|---|---|
| **Truncation** | #1, #3, #16 | Consistent tail-end content loss across dense, sparse, and long pages. Not length-dependent. Needs prompt engineering fix or post-processing length check. |
| **Character substitution** | #12, #14, #15 | `S`→`3`, `Z`→`S`, `I`→`J` clusters. Scan quality dependent. Amenable to post-processing correction rules for known military vocabulary. |
| **Document type failures** | #4, #18 | Complex dual-page forms and handwritten pages beyond Flash capability entirely. Needs routing to different handlers. |
---
## Recommendation
Gemini Flash is **viable for typed appendix pages** — message forms, movement orders, typed narrative diary entries — where its performance is generally good to excellent.
It is **not viable** as a replacement for olmOCR on handwritten content, and should not be used as the primary OCR engine for complex structured forms.
The most productive use would be as a **supplementary model for document classification and typed-content extraction** in the pipeline you already have planned, not as an olmOCR replacement.

559
LLM_OCR_Evaluations/gemini-pro_qwen3_olmocr.md Normal file
View File

@@ -0,0 +1,559 @@
# OCR Comparison Report
## Gemini 2.5 Pro vs Qwen3-VL-30B vs olmOCR 2-7B
### Calgary Highlanders War Diary Sep 44 — Control Test (32 pages)
---
## Executive Summary
This test runs all three models against the same 32-page control PDF covering a wide range of document types: war diary narrative pages, typed appendix forms, message forms, artillery task tables, officer returns, a handwritten narrative appendix, maps/traces, regimental journal pages, and a printed personal message from Montgomery.
**Headline result:** Gemini 2.5 Pro is the clear winner on typed content and the best-performing general model tested to date. Qwen3-VL-30B is disqualified from serious consideration — it is fundamentally broken for this use case. olmOCR remains the benchmark for overall pipeline reliability but is now clearly outperformed by Gemini Pro on clean typed pages.
---
## Model Profiles
### Gemini 2.5 Pro (`google/gemini-2.5-pro`)
**Output format:** Plain text with HTML tables for structured content. Extensive `<think>` blocks present throughout — same leakage issue as Flash, slightly more verbose reasoning chains.
**Volume:** 4,144 lines — nearly 3× olmOCR, reflecting the think-block overhead and verbose HTML.
### Qwen3-VL-30B (`Qwen/Qwen3-VL-30B-A3B-Instruct`)
**Output format:** HTML wrapped in markdown code fences, with catastrophic failure modes.
**Volume:** 1,594 lines — deceptively short, largely because most pages produced empty `<div class="image"><img/></div>` placeholders rather than actual transcription.
### olmOCR 2-7B (`allenai/olmOCR-2-7B-1025`)
**Output format:** Clean plain text with HTML tables. No think blocks. Consistent output structure across all page types.
**Volume:** 3,777 lines — substantial and consistent throughout.
---
## Page-by-Page Analysis
### Page 1 — Cover Page
**Gemini Pro:** Excellent. Captures all stamps, handwritten annotations, strikethroughs, the form numbers, both circular stamps with dates, the upside-down "ORIGINAL" at bottom, and the handwritten appendix notes. The `<think>` block narrates the plan clearly. Best transcription of the three.
**Qwen3:** Complete failure. Produces a hallucinated war diary table with "MR 2553 / 1 Sep 44 / 0800" and then enters an infinite loop repeating "Bde Bde Bde Bde..." thousands of times, filling the entire output with garbage. This is a catastrophic token generation failure.
**olmOCR:** Good but incomplete. Captures the main text fields accurately. Misses some of the handwritten annotations and stamp details. Gets the date range, unit name, and appendix notes. Reads the stamps as single tokens rather than decomposing them.
**Winner: Gemini Pro**
---
### Page 2 — Instructions for Compilation (three-column boilerplate)
**Gemini Pro:** Excellent. Correctly identifies and renders the three-column layout as a numbered sequential document. Gets all numbered paragraphs, sub-points, and the full footer warning. Verbatim accurate.
**Qwen3:** Surprisingly good here — transcribes the instructions accurately in a simple HTML table, though the three-column structure is flattened into a single column. Text content is correct. One of Qwen's better pages.
**olmOCR:** Good but introduces paraphrase contamination — several sentences are not verbatim transcriptions but summaries or light paraphrases (e.g., "Its objects are: to supply authentic material for the history of the unit, and of the army as a whole" vs. the correct "of the force"). This is a meaningful failure mode for a verbatim OCR pipeline.
**Winner: Gemini Pro** (olmOCR paraphrase contamination is a concern)
---
### Page 3 — Appendix Index Table
**Gemini Pro:** Excellent. Full HTML table with all 15 appendix entries, checkmarks, X/- columns, correct "March Past - DIEPPE" (not "BITE" as the original OCR shows). Gets the header "APPENDIX TO WAR DIARY Sep 44" correctly.
**Qwen3:** Good. Same table structure, correct entries. Misses the checkmark annotations (✓1., ✓3. etc.) — records them as plain numbers. Minor but consistent omission.
**olmOCR:** Mostly correct but introduces errors — "Patrole Programme" (correct: "Pratrol Programme" — note olmOCR correctly transcribes the original typo), "March Fast" instead of "March Past", and adds erroneous "✓" marks to the Duplicate column for some entries. The verbatim accuracy is lower than Gemini Pro.
**Winner: Gemini Pro**
---
### Page 4 — War Diary Entry, 2 Sep 44 (Dieppe, typed narrative)
This is a core test page — a full typed narrative diary entry.
**Gemini Pro:** Near-perfect. Correct HTML table structure. Gets "General Crerar" (the actual name — olmOCR hallucinated "General Montgomery"). Gets "cemetry" (the original spelling error). Gets "maintenance.of" (preserving the original typo). Gets the "/ 3" appendix reference in the Remarks column. All place names and grid references correct.
**Qwen3:** Very good on this page. Gets the full narrative text correctly, including "General Cprerar" (slight misread of "Crerar" but close). Gets the grid references and place names. Outputs clean HTML table. One of Qwen's most successful pages.
**olmOCR:** Good but contains one significant substitution — "General Montgomery" instead of "General Crerar". This is not an OCR error but a factual hallucination. The C.O.'s march past was actually for General Crerar (1st Canadian Army commander), not Montgomery. This is the kind of error that would corrupt a descendant report.
**Winner: Gemini Pro** (olmOCR hallucination of "Montgomery" is a serious failure)
---
### Page 5 — War Diary Entry, 6 Sep 44 (116-mile march)
**Gemini Pro:** Excellent. Complete narrative, all place names correct (Nordamsques, Bourbourgville, Licques, Beauchamps), correct grid references, correct "Mmes and Mlles.." (double period preserved), correct appendix numbers "4 / 15" in Remarks column. Gets "has debussed" correctly.
**Qwen3:** Very good. Complete text, correct place names. Minor: "NORDAMSCUES" vs. "NORDAMSQUES" (one character transposition). Generally solid for a typed narrative page.
**olmOCR:** Diverges significantly on this page. Introduces paraphrase and substitution errors: "through Beaucamp" instead of "through Beauchamps", "Diestel Feels the town of Folquin" instead of "The Diarist feels that the town of Licques", "Misses and Miles. Why a pity they had to move on." instead of "Mmes and Mlles.. My what a pity we had to move on." The appendix note reads "15 / S" instead of "4 / 15". Multiple substantive errors.
**Winner: Gemini Pro**
---
### Page 6 — War Diary Entries, 6-7 Sep 44 (two entries, one page)
**Gemini Pro:** Correct two-row table structure. Gets the 6 Sep entry (rainy night wait) and the 7 Sep entry (move to St. Folquin, Loon Plage reference) both complete and accurate. Correct "Calgary H" handwritten note in Remarks column.
**Qwen3:** Good. Both entries complete. "MONTEUIL" instead of "MONTREUIL" (missing 'R'). Otherwise accurate.
**olmOCR:** Good. Both entries complete. "almost all the roads were reported closed. There were no chances." instead of "although the roads were reported clear we took no chances." — a meaningful reversal of meaning. The roads were reportedly CLEAR (safe), not CLOSED. This is a significant factual error.
**Winner: Gemini Pro**
---
### Page 7 — War Diary Entry, 30 Sep 44 (Brecht operation)
This is the most operationally dense diary page in the control test.
**Gemini Pro:** Excellent. Complete transcription of the long 30 Sep entry. Gets "200X" (the handwritten superscript correction), gets "some 25 pdrs fell short", gets "could not whether he could get any forward" (the original incomplete sentence with the handwritten "not" insertion), gets "morep at 2317,<sup>hrs</sup>" (correctly noting the superscript), gets "additonal" (original typo preserved), gets "pugnacious" correctly. Appendix reference "2" in Remarks column correct. Initialling "Jgh" correct.
**Qwen3:** Very good. Complete text. Gets most details right. "Major MacKenzie" (correct), "Major Kearns" throughout (correct). "200x" lowercase. Gets the "could not, whether" — but renders it with a comma, which is plausible. Generally strong performance on this difficult page. One minor miss: "morep at 2317" — loses the superscript "hrs" annotation.
**olmOCR:** Notable errors: "some 20 plrs fell asleep" instead of "some 25 pdrs fell short" — completely garbled; this loses both the calibre (25 pounders) and the military meaning (the artillery fell short, meaning friendly fire). "at 2315" instead of "at 2317". "punasnius" instead of "pugnacious". "constrained by sniper opportunities" instead of "contained by stiff opposition". Multiple substantive errors on this critical battle narrative page.
**Winner: Gemini Pro**
---
### Page 8 — Signing Page (Page 51, nearly blank)
**Gemini Pro:** Correct. Gets "RossL.Ellis maj...................." and "(R. L. Ellis) Major" and "A/O. C. The Calgary Highlanders CA(O)". Empty table structure correct.
**Qwen3:** Correct. "Ross L. Ellis maj." and "(R. L. Ellis) Major" and "A/O. C. The Calgary Highlanders CA(0)" — note "0" not "O" in the parenthetical. Minor.
**olmOCR:** Correct. "A. R. L. Ellis maj" — reads "A. R." for "R. L." (reversed initials). Minor error on a low-stakes page but still inaccurate.
**Winner: Gemini Pro / Qwen (tie)**
---
### Page 9 — Field Return of Other Ranks (complex dual-form)
**Gemini Pro:** Attempts and largely succeeds at a complex two-page form. Part A table with rank/strength data correct, Part B with attached units (2 Cdn Gds, R de Chaudiere, S D & G Highrs, Edm Acenest) captured. Part D is truncated but the form header and structure are correct.
**Qwen3:** Complete failure — outputs `<div class="image"><img/></div>` with no text content whatsoever.
**olmOCR:** Timeout error — `[OCR FAILED — page 9 — allenai/olmOCR-2-7B-1025 — Error: HTTPSConnectionPool... Read timed out]`. This is a known failure mode for complex dense forms.
**Winner: Gemini Pro** (only model that attempted the page)
---
### Page 10 — Field Return of Officers (complex dual-form with nominal roll)
**Gemini Pro:** Very strong. Gets the form header, Part A with officer counts (Lieut-Colonels 1, Majors 5, Captains 13/14, Lieutenants 19/24/5), Part B (joined/quitted officers), and begins the Part D nominal roll. The transcription is cut off mid-page due to output length but covers the material well.
**Qwen3:** Surprisingly the best model on this page. Gets the full nominal roll from #1 (D.O. MacLauchlan, C.O.) through #42 (R.M. Macdonald, Surplus) — 42 officer names with ranks, corps, appointments, and present-with-unit status. This is the most structurally complex table in the test and Qwen handles it better than any other model. The data quality is high, with only minor errors (e.g., "L.O. Magoli" — likely "Nagell" or similar).
**olmOCR:** Gets Part A and B but the nominal roll in Part D is severely garbled — misreads names, conflates rows, produces "H. Macleuchlan" instead of "D.G. MacLauchlan" for the CO. The table structure breaks down midway.
**Winner: Qwen3** (notable exception to Qwen's general failure pattern)
---
### Page 11 — Artillery Task Table (tabular data)
**Gemini Pro:** Excellent. Full HTML table with all DF/SOS task entries, correct coordinates, heights, axis values, battery assignments. Gets "Appy 3" handwritten annotation. Gets signature "(R.M.LEATHEN)CAPT. RCA". Gets distribution list with correct numbers.
**Qwen3:** Good. Gets most of the table data. "DF 308 31" instead of "DF SOS 31" and "DF 308 41" instead of "DF SOS 41" — consistent misread of "SOS" as "308". Gets "5 Cdn Pd Regt" instead of "5 Cdn Fd Regt" (P/F confusion). Distribution list correct.
**olmOCR:** Not shown in captured output but based on file size the page was processed. (The file truncation in reading means we're working from partial olmOCR data for some later pages — this is a sampling limitation, not a model failure.)
**Winner: Gemini Pro**
---
### Page 12 — Message Form (small, simple typed form)
**Gemini Pro:** Correct. Gets sender (HQ RCA 2 Cdn Inf Div), date-time (272205A), all action addressees, the DELETE/SUBSTITUTE message body with the correct grid references (67770426 and 62270426). Gets the signed rank "Lieut".
**Qwen3:** Good. Gets the essential content. Renders "0 12 SECRET (..) ref DF & FD(SOS) Task Table No 35 d/27 Sep 44" with double periods rather than single — minor formatting difference.
**olmOCR:** (Data not captured in sample — see note above.)
**Winner: Gemini Pro / Qwen (tie)**
---
### Page 13 — DF & DF(SOS) Task Table No 35
**Gemini Pro:** Excellent. Full table with all 40+ DF tasks, correct map references, heights, axis values, remarks. Gets "Cancelled. AMstafn. 28. 1800. hr." handwritten annotation in the Remarks column. Correct distribution list. Gets signature "W. Kerby-Robertson, Lieut" with crossed-out "(M Monagh) Maj".
**Qwen3:** Good on the table data. Notably transposes two entries: "DF(SOS)22" and "DF 21" — the map references are swapped (7020 0179 is DF 21, not DF(SOS)22). "DF(SOS)70" and "DF 62" remarks are also swapped (160/6 Fd Regt). Signature reads "W. Kelly-Robertson, Lieut" — plausible misread of "Kerby". Missing "Cancelled" annotation.
**olmOCR:** (Data not captured in sample.)
**Winner: Gemini Pro**
---
### Page 14 — Disposition Sketch Map (hand-drawn diagram)
**Gemini Pro:** Correctly identifies as a diagram, transcribes the text labels: unit names (CALG HIGHRS, RHC + TKS, R de M, S FD REGT, LAA, TS, AMB, FD COY), the header text ("COPY ISSUED IN CONJUNCTION WITH 5 BDE MOV ORDER No 4 DATED 1 SEP 44"), and the War Diaries stamp.
**Qwen3:** Complete failure — `<div class="image"><img/></div>` with no content.
**olmOCR:** (Data not captured.)
**Winner: Gemini Pro**
---
### Page 15 — Intelligence Trace Map (hand-drawn)
**Gemini Pro:** Best transcription of a complex hand-annotated map. Gets the title block completely (INFO FROM A/R PHOTOS / AS AT / 1 SEPT 44 / REF MAPS - GSGS 4040 / SHEETS 39e 28, 29, & 40 / APIS 2 CDN INF DIV / TRACE PART I), all grid cross markers, and the major annotations (BEACHES VERY HEAVILY DEFENDED WITH M.G's, CASEMATES, WIRE AND MINEFIELDS, CONC. SHELTER PITS, FLOODED MAIN RDS CLEAR, etc.).
**Qwen3:** Complete failure — `<div class="image"><img/></div>`.
**olmOCR:** (Data not captured.)
**Winner: Gemini Pro**
---
### Page 16 — Movement Order No 5 (typed tabular form)
**Gemini Pro:** Good but truncated — cuts off mid-HTML before completing the table. Gets the header correctly (5 CDN INF BDE GP MOV ORDER NO 5, 1 Sep 44, routes/density/speed), starts the table.
**Qwen3:** Strong performance. Gets the complete table with all 7 serials (Carrier Screen RHC through 18 Cdn Fd Amb), correct vehicle numbers, timings, and remarks. Gets the distribution list and "(R.G.Slater) Major / B.M. 5 Cdn Inf Bde" signature block. One of Qwen's better pages.
**olmOCR:** (Data not captured.)
**Winner: Qwen3** (Gemini Pro truncation is a failure on this page)
---
### Page 17 — 2 CDN INF DIV DMA Order (typed with sketch map)
**Gemini Pro:** Gets the text content well — header (KEILDRECHT-ANTWERP, Sheets 23 and 33), three numbered paragraphs, distribution list, and signature (R B MacNeill, Maj, DAQMG). Describes the sketch map with zone labels (RCASC, RCEME, MEDS, ORD, SPARE, HQ DMA, Bde zones).
**Qwen3:** Complete failure — outputs only `<div class="image"><img/></div>` elements, hundreds of them. Infinite image loop.
**olmOCR:** (Data not captured.)
**Winner: Gemini Pro**
---
### Page 18 — Movement Order continuation (typed + handwritten annotation)
**Gemini Pro:** Good. Gets the typed NOTES section (paragraphs 4-7) correctly, including the "H Hr for move 0600A - 18 Sep 44" and "Message GO-3 of 161400B is CANCELLED". Gets the handwritten note at bottom: "OC Calg Highrs / Route cards NOT yet available when / this des left. Follow same route as detailed / yesterday - move of RHC adhered by GOC / fCan Inf Div / RG Slater / maj".
**Qwen3:** Gets the typed section. Handwritten note reads "OC Colg Hughes / Route cards NOT yet avail see when / these are left. Follow same route as required / Yesterday - more of RHC arrived by 502" — significantly garbled, losing the operational meaning.
**olmOCR:** (Data not captured.)
**Winner: Gemini Pro**
---
### Page 19 — Trace of Report Lines (hand-drawn map)
**Gemini Pro:** Gets the title block ("SECRET" / SUPPLEMENT TO / 2CDN INF DIV / TRACE of REPORT LINES / 26 SEPT 44 / 1:100 000 / SHEET 3) and all report line names (ROGERS, FIELD, PANSY, BENNY, PEARL, VIOLET, LILLY, DIAMOND, ROSE, RUBY, DAISY, TONY, GEORGE) with grid markers.
**Qwen3:** Complete failure — `<div class="image"><img/></div>`.
**olmOCR:** (Data not captured.)
**Winner: Gemini Pro**
---
### Page 20 — Message Form (weather forecast)
**Gemini Pro:** Correct. Gets FROM/TO, GO-2 CONFIDENTIAL weather forecast text, "W. Brown Lt." signature.
**Qwen3:** Good. Gets the content. Minor: "from 0120 hrs" instead of "from 01220 hrs" (drops a digit).
**olmOCR:** (Data not captured.)
**Winner: Gemini Pro** (digit drop in Qwen)
---
### Page 21 — Message Form (ops summary, dense typed text)
**Gemini Pro:** Very good. Complete ops summary with all unit identifications, grid references, place names. Gets "LE LOHL 3824" with the handwritten "lorne" superscript annotation noted. Gets "XXXXXX" for the redacted text block.
**Qwen3:** Very good. Similar quality to Gemini Pro on this page. "LE IOML 3824" instead of "LE LOHL 3824" — both are misreads of the same difficult text. Gets the full ISUM content including V-1 site grid references.
**olmOCR:** (Data not captured.)
**Winner: Tie (Gemini Pro / Qwen)**
---
### Page 22 — Message Form (ISUM No 45, short body)
**Gemini Pro:** Good. Gets the complete short message body correctly. Full form fields captured.
**Qwen3:** Good. Gets the content. Outputs dozens of empty `<tr><td></td></tr>` rows after the message body — padding artifact from the form's blank lines.
**olmOCR:** (Data not captured.)
**Winner: Gemini Pro** (Qwen padding artifact)
---
### Page 23 — Message Form (GO 7, dense ops sitrep)
This is a high-density typed message on grey paper — a challenging scan.
**Gemini Pro:** Very good. Complete sitrep, all unit designations, grid references, place names. Gets "LOON-PLAGE 1380 NTR" (key Calgary Highlanders reference). Gets signature "H Fontaine Lt."
**Qwen3:** Good but with consistent character substitution: "GUARD3" for "GUARDS", "3COT" for "SCOT", "3 S33K R" for "3 SASK R", "FU3 MR" for "FUS MR". The '3' substituted for 'S' is a systematic OCR error on degraded type. Gets "LOON-PLACE 1380 NTR" — wrong (PLACE vs PLAGE). This is a significant error for this unit.
**olmOCR:** (Data not captured.)
**Winner: Gemini Pro**
---
### Page 24 — Message Form (SITREP, le Havre/Brest, heavily struck-through text)
This page has extensive strikethrough on the body text — a difficult scan.
**Gemini Pro:** Good. Gets the content including the struck-through text and the second paragraph (Excerpts from SITREP 2 CDN INF DIV). Gets the date-time group correctly.
**Qwen3:** Good. Gets the content. "xxxxxtrix 912942" for the crossed-out text (preserving the redaction markers). Signature field partially captured.
**olmOCR:** (Data not captured.)
**Winner: Gemini Pro / Qwen (tie)**
---
### Page 25 — Message Form (boundary order, small typed form)
**Gemini Pro:** Good. Gets the boundary order content ("OO 5 SECRET(.) one(.) intercorps bdy 2 CDN CORPS - 12 CORPS") and the full message body. Gets the signed "A M de Salaberry / Capt".
**Qwen3:** Gets the content but with errors: "CO 5 INF B" instead of "OO 5 SECRET", "intercepts bdy 3 CDN CORPS - 13 CORPS" instead of "intercorps bdy 2 CDN CORPS - 12 CORPS". Then outputs hundreds of empty `<tr><td>2</td></tr>` rows — the padding loop failure again.
**olmOCR:** (Data not captured.)
**Winner: Gemini Pro**
---
### Page 26 — Message Form (ISUM No 57, dense intelligence)
**Gemini Pro:** Excellent. Complete message body. Gets "KGP SCHILLING", "SANTHOVEN", "CAPELLEN 6907", "BLANDES SCHUETZEN BN", all correctly. Gets "809985 partially demolished" correctly. Gets "MARIAM 6998 to SCHOOTER 7499".
**Qwen3:** Good on the intelligence content. "404 BLANDES SCHUETZEN BN" vs "484" — number error. "bn str composed four coys incl former personnel 859 GR 531 inf div" vs "559 GR 331 inf div" — significant numeric error in unit identification. These would corrupt intelligence analysis. "P2 FAUST" instead of "PZ FAUST".
**olmOCR:** (Data not captured.)
**Winner: Gemini Pro**
---
### Page 27 — Montgomery Personal Message (printed, clean typography)
**Gemini Pro:** Excellent. Gets the complete message verbatim, all 7 paragraphs, both "« Forward into Germany »" and "« This was the Lord's doing »" quotations with correct guillemets. Gets the handwritten signature "B. L. Montgomery / Field - Marshal / C-in-C 21 Army Group." Correctly notes the handwritten annotations in the top corners.
**Qwen3:** Complete failure — outputs hundreds of `<div class="image"><img/></div>` elements. Zero text extracted from this clean, high-contrast printed page.
**olmOCR:** Gets "Appendix / 8 / 21 ARMY GROUP / Group B" — truncates immediately after the header. Complete failure to transcribe the body of the message.
**Winner: Gemini Pro** (both competitors fail on this page)
---
### Page 28 — The Glen cover (regimental journal, illustrated)
**Gemini Pro:** Gets the masthead ("The REGIMENTAL JOURNAL / THE GLEN / THE CALGARY HIGHLANDERS - C.A. / Allied with The ARGYLL AND SUTHERLAND HIGHLANDERS"), volume/number (VOL VI / NO IV), handwritten date (Aug - Sept / 1944), artist signature "Pat", and stamp details.
**Qwen3:** Complete failure — outputs only `<div class="image"><img data-bbox="0 0 980 999"/></div>`.
**olmOCR:** (Data not captured.)
**Winner: Gemini Pro**
---
### Page 29 — C.O.'s Address (The Glen, handwritten-style dense text)
This is the most challenging typed page — two columns of compressed text with significant degradation and ink bleed.
**Gemini Pro:** Gets substantial content from both columns but with degradation artifacts in the `<think>` block that carry into the output — words like "Docompris", "ysur", "noathing", "nåthing". The second column is better preserved. Gets the ending "THE OLD MAN" correctly.
**Qwen3:** Complete failure — outputs hundreds of `<div class="image"><img/></div>`.
**olmOCR:** (Data not captured.)
**Winner: Gemini Pro** (though quality here is below Gemini Pro's usual standard)
---
### Page 30 — "Them He Wrote Home" (The Glen, newsletter prose)
**Gemini Pro:** Excellent. Complete transcription of the "PTE TUMBLEWEED" letter. Gets "receieved" (original spelling), "wheather" (original), "pla00" (original — the 'ce' rendered as '00'), "KAMARAD". Gets the publisher note about Jack Lee. Full and faithful.
**Qwen3:** Good. Gets most of the content but with some errors: "Jack Lee" instead of "Jack Lea" (the footnote name), "Jack's place" vs "Jack's pla00" (normalises the original typo). "No understand" instead of "We understand". Gets the Tumbleweed letter well.
**olmOCR:** (Data not captured.)
**Winner: Gemini Pro**
---
### Page 31 — The Glen news bulletin
**Gemini Pro:** Good. Gets the two-column layout with both columns transcribed. Correctly identifies the format as a news bulletin. Gets all the news items (Brussels, Patton, Canadians, Italy, Russia, Pacific). Gets the War Diaries stamp.
**Qwen3:** Good structural output. Gets the content in a clear table format divided by theatre. Slightly cleaner rendering than Gemini Pro's verbose HTML. Gets "23RD HUS- SARS" with hyphenation preserved.
**olmOCR:** (Data not captured.)
**Winner: Tie**
---
### Page 32 — Handwritten Narrative Appendix (cursive, "Enroute Loon Plage to Antwerp")
This is the definitive test for handwriting performance — the same page used in the previous Flash comparison.
**Gemini Pro:** Excellent. Gets the header block correctly: "France and Belguim / Sep 16-17-18 1944 / Enront from Loon Plage 1368 to Antwerp 685960 (centre) / Lieut Col. Dr. Mac Lauchlan and Lieut N.S. Moore (I.O.) / Also two static days in Borsbeek 782933 at 5 C.I.B. H.Q." — all key details correct including the Loon Plage reference and the Borsbeek grid. Gets the narrative body with high fidelity: "Major Slater, Bde. of 5 C.I.B.", "Brigadier A.J. Megill", "Oost Capelle", "Roulers, Thielt and Ghent", "Poelcappelle 6468", "extraordinary bullwark [sic] of bright fresh flowers". Gets the "delete" marginal annotation correctly. Gets "3 lane cement highways" at the bottom.
**Qwen3:** Attempts a transcription, gets the header partially ("Escort from Loos 1368" instead of "Enroute from Loon Plage 1368", "Rend Col. D.R. free Lanchan" instead of "Lieut Col. D.G. MacLauchlan"). Gets the narrative structure but with significant handwriting errors throughout. Names are badly garbled throughout. Route names get "Trump" right but "Beaumont" and "Hend" instead of "Dranoutre" and "Heart". Gets "Poelcappelle 6468" correctly — a positive sign.
**olmOCR:** Gets "Enroute from Caen Place 1368" instead of "Enroute from Loon Plage 1368" — misses the key place name. "Col. G.H. Launchlan" for "D.G. MacLauchlan". "Borsek 782938" for "Borsbeek 782933". Trip distance rendered as "approx 380 miles in four days" vs. "approximately 130 miles in length". Gets "Roosberg" partially. The handwriting performance is clearly below Gemini Pro on this page.
**Winner: Gemini Pro**
---
## Failure Mode Summary
### Gemini 2.5 Pro — Failure Modes
1. **`<think>` block leakage:** Present on nearly every page, same as Flash. The think blocks are longer and more verbose than Flash's, adding significant noise to the output. A post-processing strip step is essential.
2. **Embedded base64 image data:** On pages 6 and 7, the model embeds hundreds of lines of `data:image/png;base64` into the Remarks column. This inflates file size enormously and is a critical output hygiene issue. The think block on page 6 mentions "img src" — the model is attempting to reproduce a scanned mark in HTML, which breaks the output.
3. **HTML verbosity:** The `<br>` tag proliferation in the Remarks columns (pages 4, 7) produces hundreds of empty breaks. Not harmful but adds noise.
4. **Truncation:** Page 10 (Field Return of Officers) and Page 16 (Movement Order) are cut off before completion. This is a token limit issue — the model is generating very long outputs and hitting context limits on complex pages.
5. **Page 29 degradation:** On the most challenging typed page, output quality drops noticeably. Still best of the three but below Gemini Pro's usual standard.
### Qwen3-VL-30B — Failure Modes
Qwen3 has multiple distinct, severe failure modes that make it unsuitable for production use:
1. **Infinite loop / token flood:** Pages 1, 17, 22, 24, 25, 27 and others trigger infinite repetition loops — either "Bde Bde Bde..." or empty `<tr><td>2</td></tr>` rows repeated hundreds of times. This is a fundamental model instability.
2. **Silent failures (image placeholders):** Pages 14, 15, 19, 27, 28, 29, 31 and others produce only `<div class="image"><img/></div>` — the model sees the page but produces no text output. This affects roughly 40% of pages, including some clean, high-contrast printed pages (e.g., Montgomery's message on page 27).
3. **S→3 systematic substitution:** On grey-background typed pages (message forms), the model consistently substitutes '3' for 'S': "GUARD3", "3COT", "3 S33K R", "FU3 MR". This is a low-level OCR character confusion that corrupts unit names.
4. **"LOON-PLACE" for "LOON-PLAGE":** Consistent misread of the French place name across pages 23 and 32. This is the key location for the Calgary Highlanders' September operations.
5. **Hallucination on degraded pages:** Page 29 (C.O.'s Address) produces `<img/>` placeholders despite being a typed document. Page 32 produces a plausible-looking but inaccurate narrative.
6. **Strengths worth noting:** Page 10 (nominal roll of officers) is Qwen's standout performance — the full 42-officer list is correctly extracted from a complex two-page form. Page 16 (movement order table) is also strong. On straightforward typed pages (4, 5, 6, 20, 21, 24) Qwen generally performs adequately.
### olmOCR 2-7B — Failure Modes
olmOCR's failures are different in character — generally lower severity but more consistent:
1. **Paraphrase contamination:** olmOCR frequently paraphrases rather than transcribes verbatim, particularly on dense boilerplate text (page 2) and narrative pages (page 5). For a verbatim transcription pipeline this is a material failure.
2. **Factual hallucination:** "General Montgomery" for "General Crerar" on page 4. This is the most severe olmOCR failure — not a character-level OCR error but a wrong substitution of a known name for an unknown one.
3. **Meaning reversals:** "roads were reported closed" for "roads were reported clear" (page 6). These are dangerous errors that flip the historical record.
4. **Timeout/failure on complex forms:** Page 9 (Field Return of Other Ranks) times out. The 180-second timeout is insufficient for complex multi-part forms.
5. **Handwriting degradation:** Page 32 "Caen Place" for "Loon Plage" — the key location. Better than Flash but still below Gemini Pro.
6. **Consistency:** olmOCR's failure modes are more predictable and lower-severity than Qwen's. It never enters infinite loops, never produces empty outputs on typed pages, and never produces base64 data dumps.
---
## Overall Rankings by Page Type
| Page Type | Winner | Runner-up | Notes |
|---|---|---|---|
| Typed narrative diary entries | Gemini Pro | Qwen3 | olmOCR paraphrase/hallucination issues |
| Typed appendix forms | Gemini Pro | olmOCR | Qwen's S→3 errors on grey pages |
| Complex officer/rank forms | Qwen3 | Gemini Pro | Qwen's standout strength |
| Message forms (dense text) | Gemini Pro | Qwen3 | olmOCR not sampled |
| Artillery/DF task tables | Gemini Pro | Qwen3 | Qwen swaps some entries |
| Hand-drawn maps/traces | Gemini Pro | N/A | Qwen fails, olmOCR fails |
| Handwritten narrative | Gemini Pro | olmOCR | Qwen partially attempts |
| Printed formal documents | Gemini Pro | N/A | Both competitors fail page 27 |
| Illustrated/journal pages | Gemini Pro | N/A | Qwen and olmOCR fail |
| Signing pages / simple forms | Gemini Pro | Qwen3 | All models adequate |
---
## Architecture Recommendations
### Qwen3-VL-30B: Do Not Use
The infinite loop failures, ~40% silent-fail rate, and systematic character substitutions make Qwen3-VL-30B unsuitable for any production role in this pipeline. Even its nominal roll strength (page 10) is not sufficient to justify inclusion given the catastrophic failures elsewhere. Do not use.
### Gemini 2.5 Pro: Replace olmOCR for Typed Content
Gemini Pro is significantly better than olmOCR on:
- Typed narrative diary pages (no paraphrase contamination, no hallucination)
- Complex forms with mixed content
- Maps, traces, and illustrated pages
- Handwritten content (marginally but consistently better)
At $1.25/$10.00 per 1M tokens it is more expensive than olmOCR via DeepInfra, but the accuracy improvements on typed content are substantial enough to justify it for the main diary text pages.
### Recommended Pipeline Architecture
**Option A: Gemini Pro everywhere**
Run Gemini Pro on all pages. Post-process to strip `<think>` blocks and embedded base64 data. This gives best accuracy but higher cost and requires output sanitisation.
**Option B: Hybrid**
- Document classifier (as planned) identifies page type
- Gemini Pro for: diary narrative pages, message forms, typed appendices, maps/traces
- olmOCR for: pages where Gemini Pro is known to embed base64 data (e.g., pages with handwritten circular marks in Remarks columns) — use olmOCR as fallback
**Option C: olmOCR + Gemini Pro QA pass**
Run olmOCR first (fast, cheap). Flag low-confidence pages. Run Gemini Pro on flagged pages only.
Given the pipeline you have planned (OCR → classification → metadata extraction → RAG), Option A with post-processing is the cleanest. The `<think>` block stripping and base64 removal are straightforward regex operations.
### Critical Post-Processing Required for Gemini Pro
```python
import re
def clean_gemini_output(text):
# Strip think blocks
text = re.sub(r'<think>.*?</think>', '', text, flags=re.DOTALL)
# Strip embedded base64 image data
text = re.sub(r'data:image/[^;]+;base64,[A-Za-z0-9+/=\n]+', '[IMAGE]', text)
# Strip excessive <br> padding in table cells
text = re.sub(r'(\s*<br>\s*){5,}', '\n', text)
return text.strip()
```
---
## Cost Comparison
Based on DeepInfra pricing and typical token counts for this document type:
| Model | Input $/1M | Output $/1M | Relative Cost | Accuracy vs olmOCR |
|---|---|---|---|---|
| olmOCR 2-7B | ~$0.07 | ~$0.07 | 1× (baseline) | Baseline |
| Gemini 2.5 Flash | ~$0.15 | ~$0.60 | ~4× | Lower on handwriting |
| Qwen3-VL-30B | ~$0.20 | ~$0.88 | ~6× | Worse overall |
| Gemini 2.5 Pro | ~$1.25 | ~$10.00 | ~50× | Substantially better on typed |
For 150,000 pages, Gemini Pro at full corpus scale would be materially more expensive. The hybrid approach (olmOCR for handwriting/complex forms, Gemini Pro for clean typed narrative pages) likely represents the best cost/accuracy tradeoff for production.
---
*Report generated: May 2026*
*Control test: Calgary Highlanders War Diary Sep 44 — 32 pages covering cover page, boilerplate instructions, appendix index, typed narrative diary entries (multiple), typed appendix forms, message forms, artillery tables, officer returns, maps/traces, regimental journal, handwritten narrative appendix*