WD_1 - Import War Diaries
=========================
User: Admin
Input: War diary (PDF or JPEG), separated by month
Result: Document stored server-side, available for OCR

1. Go to Frame 44 UI or plugin
2. Upload PDF or JPEG
3. Assemble JPEG uploads into single PDF, confirm page order
4. Assign to existing unit or create new unit record
5. Fill in provenance metadata: archive source, unit ID, date range, digitisation quality
6. Tag narrative page range in thumbnail gallery
7. Click submit
8. Document stored server-side, status set to "pending OCR"

---

WD_2 - OCR War Diaries
=======================
User: System (automated)
Input: Imported document, narrative page range metadata
Result: Raw OCR text stored per page, narrative and non-narrative pages split into separate processing tracks

1. System detects document with status "pending OCR"
2. Full document is processed through OCR pipeline
3. Raw OCR text stored per page in database
4. Narrative pages set to status "pending review", queued for OCR editor
5. Non-narrative pages set to status "pending classification", queued for document classification

---

WD_3 - Document Classification
===============================
User: System (automated), Volunteer (review)
Input: Non-narrative pages from OCR
Result: Every non-narrative page has a document type tag before entering the RAG pipeline

1. System detects non-narrative pages with status "pending classification"
2. System classifies each page by document type: message log, intel report, newsletter, Part 1 Orders, nominal roll, appendix, sitrep, patrol report, trace, etc.
3. Confidence score assigned per classification
4. High confidence classifications proceed automatically to RAG chunking
5. Low confidence classifications flagged for human review
6. Reviewer confirms or corrects classification
7. Verified classifications fed back into system to improve future accuracy
8. All classified pages proceed to RAG pipeline

---

WD_4 - OCR Editor
=================
User: Volunteer
Input: Narrative pages with status "pending review"
Result: Every narrative page verified by a human before downstream processing

1. Volunteer logs into OCR editor
2. Volunteer sees work queue of narrative pages with status "pending review"
3. Volunteer selects a war diary to work on
4. Editor displays original page image on left, raw OCR text on right
5. Volunteer reads and corrects OCR text
6. Changes tracked per user for version control
7. Volunteer marks page as "verified"
8. Verified pages queued for entity extraction and RAG chunking

---

WD_5 - Entity Extraction
=========================
User: System (automated), Volunteer (flagged items)
Input: Verified narrative pages, classified non-narrative pages
Result: All entities extracted and stored as structured records linked to source page and RAG chunk

1. System detects verified narrative pages and classified non-narrative pages
2. System extracts entities from all pages:
    - Personnel: names and ranks
    - Place names: locations mentioned in text
    - Grid references: WWII map grid references
    - Unit mentions: friendly, allied, and enemy units
    - Events: attacks, patrols, reliefs, casualties, objectives
3. Structured record created per entity: entity type, value, context snippet, source page, source RAG chunk, date, unit
4. Low confidence or ambiguous extractions flagged for volunteer review
5. Volunteer confirms or corrects flagged extractions
6. All entity records stored in database linked to source page and RAG chunk

---

WD_6 - Gazetteer
================
User: System (automated), Volunteer (flagged items)
Input: Extracted place name records, known unit positions with date, modern GIS datasets
Result: All place names resolved to modern coordinates and standardised spellings

1. System detects extracted place name records
2. For each place name, system spatially constrains candidate matches to within a defined radius of the unit's known position on that date
3. Candidate matches ranked by name similarity and spatial proximity
4. High confidence matches resolved automatically
5. Where multiple diary entries from different units independently resolve the same string to the same coordinate, confidence increases automatically
6. Low confidence matches flagged for volunteer review
7. Volunteer confirms or corrects flagged matches
8. Gazetteer record created per resolved place: variant spellings, modern coordinate, confidence score, corroborating source entries
9. Unresolved place names remain flagged for future review
10. Gazetteer feeds back into improving future place name extraction

---

WD_7 - RAG Pipeline
====================
User: System (automated)
Input: Verified narrative OCR text, classified non-narrative pages, entity extraction records
Result: Full corpus chunked, embedded, and stored in Postgres with pgvector and PostGIS, queryable by semantic similarity, metadata filter, and spatial query

1. System detects verified narrative pages and classified non-narrative pages
2. Pages chunked into segments respecting natural document boundaries
3. Each chunk embedded and stored in Postgres with pgvector
4. Chunk metadata attached: source document, page, unit, date, document type, confidence level, linked entity records
5. PostGIS spatial data attached to chunks where gazetteer-resolved coordinates are available
6. Chunks queryable by semantic similarity, metadata filter, spatial query, or any combination in a single query

---

WD_8 - Grid Reference Extraction and Verification
==================================================
User: System (automated), Volunteer (verification)
Input: Grid reference records from entity extraction, known unit positions with date
Result: All grid references resolved to real world coordinates, verified positions linked to source RAG chunk

1. System detects grid reference records from entity extraction
2. System resolves each grid reference to coordinates using WWII map projections and known 100,000m grid square for the unit and date
3. Candidate position created per grid reference: coordinate, source diary entry, date, unit, position type, confidence level, relevance category
4. Candidate positions stored in Postgres with PostGIS, linked to source RAG chunk
5. All candidate positions set to status "pending verification"
6. Volunteer reviews candidate positions against reference materials
7. Volunteer confirms or corrects each position
8. Verified positions set to confidence Level 1
9. Unverified high confidence positions set to Level 2
10. Low confidence positions remain Level 3 and flagged for further review
11. Position type tagged: end of day, patrol, enemy, objective, etc.
12. Relevance category tagged: own unit activity or reference to another unit

---

WD_9 - Document Versioning and Corpus Expansion
================================================
User: Admin
Input: Higher quality scan of existing document, or new document belonging to existing unit
Result: Corpus updated without disrupting existing data

Scenario 1 — Higher Quality Scan:
1. Admin uploads higher quality scan via standard import
2. Admin identifies existing document record in database
3. Source image updated in database
4. All downstream records remain intact and unchanged

Scenario 2 — New Document Found:
1. Admin uploads new document via standard import pipeline
2. Admin assigns document to existing unit record
3. Document processed through full pipeline from WD_1 onwards
4. All outputs appended to existing unit records in database

---

WD_10 - Secondary Sources
==========================
User: Admin
Input: Published secondary source as clean digital text or scanned copy
Result: Secondary source ingested into RAG pipeline, attributed as secondary literature, entities extracted

1. Admin uploads secondary source via import pipeline
2. Admin assigns source to theatre and formation coverage (e.g. II Canadian Corps, Northwest Europe)
3. If clean digital text: proceeds directly to entity extraction and RAG chunking
4. If scanned copy: processed through OCR pipeline without human verification gate, then proceeds to entity extraction and RAG chunking
5. Entity extraction runs on all secondary sources: personnel, place names, unit mentions, events
6. Extracted place names and grid references passed to gazetteer and grid reference verification
7. Secondary source chunks stored in Postgres, attributed as secondary literature in all records
8. Secondary source queryable alongside primary sources in all downstream products