# Project '44 — Pipeline Session Notes
*May 2026 | Working log — pipeline architecture & decisions*

---

## 1. Pipeline Overview

| Step | Name | Input | Output | Model | Notes |
|------|------|-------|--------|-------|-------|
| 1 | OCR | PDF pages (PNG) | Raw MD with HTML tables | Claude 4.6 or olmOCR | Vision work — reads scanned images. Claude 4.6 significantly better than olmOCR on structured docs. |
| 2 | Viewer MD | Step 1 MD | Clean viewer MD (no HTML) | Script (no LLM) | Display artifact only — for Jonesy QA and Project '44 frontend. Not a data input. |
| 3 | Extraction | Step 1 MD (text only) | _narratives.json + _positions.json | Llama 70B (TBD: Claude?) | Text reasoning only — no vision needed. Re-runnable cheaply if schema changes. |

---

## 2. Why Two Passes (Step 1 + Step 3) Is Not Wasteful

Step 1 is vision work (image → text). Step 3 is reasoning work (text → structured JSON). Fundamentally different tasks.

- Step 1 sends PNG images to a vision-capable model — expensive per page, requires visual understanding of scanned documents.
- Step 3 reads clean OCR text and extracts structured data (dates, grids, place names, categories). No image involved. Costs a fraction of step 1.
- Keeping them separate means if the extraction schema changes (it will), you re-run step 3 only against already-OCR'd MD files. You never re-OCR.
- Collapsing into one step would mean paying vision token rates for every field extraction on every page — more expensive and harder to prompt correctly.

**Decision: Two-pass is the correct architecture. The separation is a cost advantage, not a cost problem.**

---

## 3. What Step 3 Actually Produces (and Why It's Needed)

The viewer MD looks good — but step 3 is still necessary. The two outputs serve completely different consumers.

### _narratives.json
- Dated entries as discrete records: date, place, weather, full summary text.
- Feeds the descendant report generation pipeline.
- Without it, Claude has no way to query "what happened on 14 Sep 44" without reading the entire diary.
- The viewer MD is prose — it cannot be queried or filtered by date.

### _positions.json
- Geo-extractable records with raw grid references, sheet refs, position categories (END_OF_DAY, SUBUNIT, ENEMY, PATROL, etc.), confidence levels, and context snippets.
- Feeds the mapping layer — Postgres with pgvector + PostGIS.
- Without it, there are no coordinates to plot on the map.

**Viewer MD = display artifact. Step 3 JSONs = database inputs. Both necessary.**

---

## 4. Page Composition of a War Diary — The 300/30 Insight

A typical war diary PDF of ~300 pages breaks down roughly as:

| Document Type | Approx. Pages | Notes |
|---------------|---------------|-------|
| Narrative diary pages | ~30 (10%) | The HTML table pages with Place / Date / Hour / Summary. Need human verification before narrative extraction. |
| Misc documents | ~270 (90%) | Message logs, intel reports, newsletters, Part 1 Orders, nominal rolls, appendices, sitreps, patrol reports, traces. |

---

## 5. Proposed Pipeline Routing

Narrative pages and misc documents have different quality gates — they should not share the same queue.

### Misc documents (~270 pages)
- Step 1 (OCR) → straight to step 3 extraction immediately.
- No human verification needed before extraction.
- Feed intel reports, patrol data, message metadata into the RAG pipeline.

### Narrative diary pages (~30 pages)
- Step 1 (OCR) → step 2 viewer MD → Jonesy verifies → step 3 narrative extraction.
- Only after human sign-off does the page go to step 3.
- Ensures `_narratives.json` (which feeds descendant reports) is built on verified text.

**Practical benefit: 90% of extraction work flows through immediately. The bottleneck only applies to the 30 pages that matter most for the commercial product.**

---

## 6. Open Question: Llama vs Claude for Step 3

Step 3 currently uses `meta-llama/Meta-Llama-3.1-70B-Instruct`. May produce more errors on:

- Ambiguous HTML table rows with missing or split dates
- Grid reference parsing and 4-digit → 6-digit expansion
- Correct position category assignment (END_OF_DAY vs SUBUNIT vs FRIENDLY)

**To test:** Run step 3 on the same Calgary Highlanders pages with (a) Llama 70B and (b) Claude Haiku or Sonnet. Compare `_positions.json` output quality, especially grid confidence and category accuracy. Bad grids silently poison PostGIS, so accuracy matters more than cost here.

---

## 7. Key Principles Confirmed This Session

- **OCR quality:** olmOCR is decent; Claude 4.6 is significantly better for structured historical documents.
- **Two passes:** The two-pass architecture (vision OCR → text extraction) is deliberate and cost-effective.
- **Step 3 purpose:** Step 3 is not redundant — viewer MD and JSON outputs serve completely different consumers.
- **Page ratio:** ~10% narrative pages, ~90% misc documents. Most of the pipeline runs without waiting for human review.
- **Quality gate:** Narrative pages need human verification before step 3. Misc pages do not.
- **Data quality:** pgvector + PostGIS in a single SQL query is a core architectural advantage. Bad grids are hard to catch downstream — model accuracy in step 3 matters.