testing out OCR viewer of OCR'd text

notes/ocr-viewer-prompt.md

@@ -0,0 +1,46 @@
+# P44 OCR Viewer — Copilot Build Prompt
+
+## Context
+A desktop OCR review tool for verifying war diary OCR output page by page.
+Single HTML file, no build step, runs by double-clicking.
+
+## If Copilot times out — build in this order:
+
+### Step 1: Shell + layout
+> Build the shell of the P44 OCR Viewer. Two panels side by side using CSS Grid.
+> Dark background (#1a1a2e), gold accent (#c8a84b). Left panel: image display with
+> scroll. Right panel: scrollable text area. Top toolbar with three file inputs:
+> (1) page image, (2) OCR text file, (3) word-position JSON sidecar. No interactivity
+> yet — just layout and file loading that populates each panel.
+
+### Step 2: Word overlay on image
+> Add a word bounding box overlay layer to the left panel. The JSON sidecar format is:
+> [{ "word": "string", "id": "w001", "bbox": { "x": 0, "y": 0, "w": 50, "h": 20 } }]
+> Render transparent absolutely-positioned divs over the image for each word,
+> using the bbox coordinates scaled to the rendered image dimensions.
+> Each div gets a data-word-id attribute matching the JSON id.
+
+### Step 3: Word spans in text panel
+> Parse the OCR text in the right panel and wrap each whitespace-delimited token
+> in a <span> with a data-word-id attribute. IDs are assigned sequentially (w001,
+> w002...) matching the order in the JSON sidecar.
+
+### Step 4: Bidirectional hover highlight
+> On hover over any word span (right) or bounding box div (left), highlight the
+> matching element on the other side with a yellow background/border. Use the
+> data-word-id to find the match. Clear highlight on mouseout.
+
+### Step 5: Load example button
+> Add a "Load example" button to the toolbar that pre-populates all three panels
+> with bundled inline sample data (a small image placeholder, 2-3 lines of sample
+> OCR text, and a matching JSON sidecar with ~10 words). Tool should be fully
+> demonstrable without real files.
+
+## Full one-shot prompt (use if Copilot has capacity)
+Build a desktop OCR review tool called P44 OCR Viewer for reviewing war diary OCR output. The interface has two panels side by side:
+Left panel — the source document image. Load a JPEG or PNG page scan. Render it at full panel height, scrollable. Each word in the image should be represented by a transparent, hoverable bounding box overlay (coordinates come from a JSON sidecar file, format: [{ "word": "string", "id": "w001", "bbox": { "x": 0, "y": 0, "w": 50, "h": 20 } }]).
+Right panel — the OCR text output. Load the corresponding plain text or Markdown file. Each word in the text should be individually wrapped in a <span> with a matching data-word-id attribute that maps to the same ID in the JSON sidecar.
+Hover behaviour: Hovering over a word on either side highlights the corresponding word on the other side — a yellow highlight on the image bounding box overlay, and a yellow background on the text span. Bidirectional. Highlight clears on mouse-out.
+File loading: A toolbar at the top has three file inputs: (1) Page image, (2) OCR text file, (3) Word-position JSON sidecar. Files are loaded locally, no server required. A "Load example" button pre-populates with bundled sample data so the tool works without real data.
+Stack: Single HTML file, vanilla JS, no build step required. Designed to run by double-clicking — no localhost server, no npm. All layout via CSS Grid. Colour scheme: dark background (#1a1a2e), muted gold accent (#c8a84b) consistent with a WWII archival aesthetic.
+The word-alignment JSON sidecar format is designed to be produced by olmOCR (Allen Institute), which outputs bounding box data alongside transcribed text. The viewer is the QA interface a historian uses to verify OCR accuracy page by page.

notes/pilot-pipeline-tasklist.md

@@ -0,0 +1,10 @@
+1. Pilot scope: 5 CIB + 4 battalions, Sept–Oct 44 (~10–12 PDFs)
+2. Write 30–50 evaluation questions and answer them by hand from the source documents — this is your ground truth
+3. omOCR the pilot corpus
+4. Human-verify the OCR — full review on this set, since it's also benchmark data
+5. Design the inventory schema (page-range based, with document-level granularity where pages are uniform)
+6. Build the inventory using LLM-assisted prompts, refined against your hand-built inventories of CH/5CIB/BW Sept 44
+7. Design and test chunking strategy — chunk size, metadata attachment, special handling for tables/orders
+8. RAG-chunk and embed
+9. Run evaluation questions against the system, score, iterate
+10. Once accuracy targets are hit on the pilot, scale to the full 1,200-doc corpus with automated OCR + inventory + chunking, with human spot-checks rather than full verification

notes/scoring-schema.md

@@ -0,0 +1,63 @@
+Tier 1 — Direct factual lookup (10 questions)
+Tests basic retrieval: does the right chunk come back for a clear factual query?
+
+What unit was the Calgary Highlanders' Dog Coy attacking on 15 September 1944, and what was the location?
+Who signed the 5 CIB war diary for September 1944?
+On what date did the Calgary Highlanders force the Albert Canal crossing?
+What was the grid reference of the lock gates the Calgary Highlanders crossed at on 22 September 1944?
+Who was singled out for distinction during the Albert Canal crossing, and what coy/platoon were they in?
+What was the date of the Dieppe march past attended by the Calgary Highlanders?
+What was the Black Watch's location at the start of September 1944 (per the cover sheet)?
+How many German prisoners did the Calgary Highlanders take on 14 September 1944 in the action toward Fme Geersen?
+What was the H-hour given in the Black Watch field message of 10 September 1944, 0025 hrs?
+What route was issued under 5 CIB Mov Order No 5 on 1 September 1944?
+
+
+Tier 2 — Cross-document synthesis (10 questions)
+Tests whether the system can combine information from multiple sources to answer a single question.
+
+Compare the Calgary Highlanders and Black Watch officer strength at the start and end of September 1944. What does the trend suggest?
+On 22 September 1944, the Calgary Highlanders were ordered to take over a crossing the Black Watch had failed to make. What did the Black Watch war diary say about that failed attempt?
+What does the 5 CIB war diary narrative say about the Calgary Highlanders' actions on 7–10 September 1944, and how does it compare to the Calgary Highlanders' own narrative for those dates?
+Which battalion in 5 CIB had the highest other-rank strength in the week ending 9 September 1944?
+Trace the movement of 5 CIB from Totes to the Dunkerque outpost line using the Move Orders (No 4, 5, 6, 7) and the brigade narrative.
+What enemy defences were the Calgary Highlanders likely to encounter at Bourbourgville based on the SHAEF and APIS traces in the Black Watch and brigade files?
+On 5 September 1944, what was 5 CIB's planned route at the divisional level (per the 2 Cdn Inf Div Route Card) and how does it compare to the brigade's own Mov Order No 6?
+Identify all the locations mentioned in any 5 CIB document for the date 1 September 1944. What movement is implied?
+Which sister battalion was closest to the Calgary Highlanders' position on 9 September 1944?
+What casualties did 5 CIB units sustain in the first ten days of September 1944, and where would that information be found?
+
+
+Tier 3 — Operational and tactical reasoning (10 questions)
+Tests whether the system can answer questions that require understanding military context, not just text retrieval.
+
+What was 5 CIB's tactical role between 1 and 10 September 1944 — exploitation, set-piece attack, defensive holding, or something else? Cite evidence.
+Where was the Calgary Highlanders' advance party on 16–18 September 1944, and what does that suggest about brigade movement plans?
+What was the significance of the Calgary Highlanders crossing at Fort 1 on the Albert Canal rather than at the original Black Watch crossing point?
+Based on the patrol activity recorded around Loon Plage 9–13 September 1944, what was the tactical situation?
+What artillery support was available to the Calgary Highlanders during the Bourbourgville–Loon Plage operations? Where would that be documented?
+The 5 CIB war diary for September mentions "Dunkerque outpost line." What does that mean operationally, and which 5 CIB documents add detail to that picture?
+Why did the Calgary Highlanders move from France to Belgium between 18–19 September 1944, and what was the operational objective?
+What evidence exists in the brigade and battalion war diaries for the strength and quality of the German garrison the Calgary Highlanders were facing in early September 1944?
+Was the Calgary Highlanders' advance from Bourbourgville to Loon Plage opposed, lightly opposed, or heavily opposed? Cite specific document evidence.
+What changes in unit identifications from PW interrogations (per the GO-1 ISUM) might have affected 5 CIB's understanding of the enemy in early September 1944?
+
+
+Tier 4 — Reinforcement / personnel queries (5 questions)
+Tests the "His War" use case directly — reconstructing what an individual soldier would have experienced.
+
+A reinforcement arriving at the Calgary Highlanders on or about 1 September 1944 would have been part of which company actions in the following ten days? Walk through what they would have experienced day by day.
+What named reinforcements arrived at the Calgary Highlanders during 16–17 September 1944?
+If a soldier joined the Calgary Highlanders as a reinforcement in late August 1944, what was the strength of the battalion when they arrived (per the field returns), and how did it change over September?
+What activities did the Calgary Highlanders conduct that were not direct combat operations during 1–10 September 1944 (parades, baths, religious services, etc.)?
+A Calgary Highlanders soldier killed or wounded on 8 September 1944 at Bourbourgville would have their casualty recorded in which document(s), and on what dates would those documents have been filed?
+
+
+Tier 5 — Stress tests and adversarial queries (5 questions)
+Tests failure modes — does the system hallucinate when the answer isn't in the corpus, or does it correctly say "I don't know"?
+
+What were the Régiment de Maisonneuve's casualties on 8 September 1944? (R de Mais inventory not provided in pilot — system should say it doesn't have that source, not invent an answer.)
+What did General Crerar say in his speech at the Dieppe march past? (Speech text is not in the inventory; the march past is mentioned but the speech content isn't. Correct answer: "not in available sources.")
+What was the weather on 7 September 1944 at Bourbourgville? (May or may not be in the war diaries; system should answer from source if present, otherwise say so — not pull from external knowledge.)
+Who commanded the German garrison at Dunkerque in September 1944? (Likely not in 5 CIB sources; tests whether system stays within corpus or hallucinates from training data.)
+What does the war diary say about Calgary Highlanders operations on 5 October 1944? (October is in scope for the pilot but not yet inventoried in our session — tests whether system handles dates within scope but in un-verified portions of the corpus.)

notes/v1-evaluation-accuracy.md

@@ -0,0 +1,10 @@
+Line 15 - Talks about Basingstoke, which I assume is where the Calgary highlanders were based in England before they went to France. This is a detail that I would expect to be in the report, and it is not. The report should be grounded in the diary, and this is a detail that is in the diary but not in the report. This is a miss, and it should be included in the report to give a sense of where the soldiers were before they went to France.
+line 15 - it says the co who know his officers by their first names. Is this in the war diaries or an assumption? 
+line 17 - great detail about the first world war actions of the 10th battalion
+line 19 - seems accurate to the war diary. It talks about GR Crockett recommended for the victoria cross,but I know crockett did not get the VC. Be careful to word it so that it is clear that he was recommended but did not receive it.
+line 27 - unless we are showing these position on the map, giving grid references in the report is not helpful to the family. It is more important to give a sense of where they were in relation to known places (e.g. "the town of X" or "the river Y") rather than giving grid references that the family will not understand. If we are showing these positions on the map, then it is fine to include the grid references, but if not, then it is better to omit them from the report.
+line 48 - spot checked. seems accurate to the war diary
+Line 50 - spot checked. seems accurate to the war diary.
+Line 52 - spot checked. seems accurate to the war diary.
+Line 55 - spot checked. seems accurate to the war diary.
+Line 61 - spot checked. seems accurate to the war diary.

notes/v1-evaluation-customer.md

@@ -0,0 +1,19 @@
+Did I learn what my grandfather lived through, in a way I'll remember?
+
+I think the average person will get a good sense of where their grandfather was, and roughly wha they are doing. I do think that the because the war diary is very officer centric, and based around the command post and tactical headquarters, it is hard to get a sense of what the average private would have experienced. The report does a good job of trying to include the human texture of the experience, but it is still very much focused on the operational events and the perspective of the officers. I think that is a limitation of the source material, and it is something we will have to be mindful of as we produce these reports. We may want to consider supplementing the war diary with other sources that give a more ground level perspective, such as letters, diaries, or oral histories from soldiers who were there. That would help us to give a more well rounded picture of what the average soldier experienced during those battles.
+
+Are there moments here I'll quote to my family at Christmas?
+
+I think there are some sections people would want to share, but again very officer centric.
+
+Did the report respect my intelligence, or did it talk down to me / over my head?
+
+The report is written in a way that is accessible to a general audience, but it does not talk down to the reader. It assumes a certain level of knowledge about the war and the battles, but it also explains things in a way that is easy to understand. I think it strikes a good balance between being informative and being engaging, without being condescending or overly simplistic.
+
+What do I now want to ask, that the report didn't answer?
+
+More on what the average soldier experienced, and what the fighting was like from their perspective. I would also want to know more about the specific actions that Bill was involved in, if we can find that out. I would also want to know more about the human texture of the experience, such as what the rations were like, what the billets were like, what the weather was like, and what the civilian encounters were like. Those are the things that really bring the experience to life for me, and I think they would be important for other people as well.
+
+Would I pay for this? If yes, roughly what?
+
+I would pay for this, but how much I dont know. I think $20 at a minimum. If there was more depth to what it was like for an average soldier it would be worth more. As well, if we could include photos and maps, and see where Bill was day by day, and where the most important events happened, that would add a lot of value. 

notes/v1-evaluation-operational.md

@@ -0,0 +1,8 @@
+Is this actually a mappable event? (Some movements/billets aren't really points; some are.)
+Most of it is very mappable. Especially if we map the events to the units location ie we know where the unit was, so you are already near it, use the gsgs base maps to find the location, and then use the grid reference to find the exact location. Very very useful
+
+Does the location text contain enough information to geocode? (Grid refs are gold; "Hinkelenoord" alone is harder.)
+Grids are perfect. Locations are harder, but I think over time as we map a place with a name, it needs to be captured into like a gazetter or toponomy record so that we can find it again. I think the more we do this, the better we will get at geocoding these locations. We will also get better at recognizing when a location is not specific enough to geocode, and when we need to do some additional research to find out where it is.
+
+Would Jonesy accept it without changes, edit it, or reject it? Estimate the proportions.
+I think Jonesy would accept most of it without changes, but there would be some edits for clarity and to make sure that we are not including any information that is not supported by the war diary. I would estimate that about 80% of the events would be accepted without changes, about 15% would be edited for clarity or to remove unsupported information, and about 5% would be rejected because they are not supported by the war diary or because they are not mappable.

notes/v1-evaluation-voice.md

@@ -0,0 +1,3 @@
+The voice and tone is good but I feel it does not provide enough gravity to the heavy fightinng near hoogerheide, and Waclheren. I would like to see more of the emotional weight of the fighting in that area, and the toll it took on the soldiers. The report should not shy away from the brutality of the fighting, and should give a sense of the fear and uncertainty that the soldiers must have felt during those battles. Though, if it is missing from the war diary, it is not a problem to omit it from the report. I just want to make sure that we are not sanitizing the experience for the family, and that we are giving them a sense of what their ancestor went through during those battles.
+
+We can expand on the voice when we use other texts in the future such as CP Staceys The Victory Campaign. 

notes/v2-source-selection.md

@@ -0,0 +1,110 @@
+# Supplementary Source Inventory — Pte. Bill Bloggins (placeholder)
+## Calgary Highlanders, 1–10 September 1944
+
+**Purpose:** Curated list of supplementary documents to accompany the Calgary Highlanders main war diary narrative for the v2 prototype descendant report.
+**Focus window:** 1–10 September 1944
+**Source PDFs covered:** Calgary Highlanders, 5 Cdn Inf Bde, RHC Black Watch (R de Mais inventory not provided)
+**Inventory coverage:** CH pp. 1–45 of 343 · 5 CIB pp. 1–56 of 193 · BW pp. 1–51 of 144
+**Document count:** 24 items
+**Status:** Ready for use; gaps flagged at end
+
+---
+
+## Selection criteria (priority order)
+
+1. Documents covering specific events the Calgary Highlanders were involved in during 1–10 Sep 1944
+2. Brigade-level signals, sitreps, or orders covering the same dates and actions
+3. Sister-battalion entries that mention or coordinate with the Calgary Highlanders during the focus window
+4. Patrol programmes, patrol reports, casualty returns, or other operational documents adding detail beyond the diary narrative
+
+---
+
+## Source 1 — Calgary Highlanders War Diary
+**File:** `Calgary-Highlanders_War-Diary_Sep44.pdf` (343 pp; inventoried 1–45)
+
+| Page(s) | Document type | Date covered | Note |
+|---|---|---|---|
+| 6 | Appendix index | Sep 44 | Master key to the file — locates Appx 12 (Memorial Service 3 Sep), Appx 13 (Dieppe march past), Appx 10 (Daily News sheets) referenced in narrative for focus window |
+| 21–26 | War diary narrative pp. 15–20 | 11–13 Sep 44 | **Just outside window** — direct continuation of Loon Plage fighting Bloggins entered 7–10 Sep; drop if hard 10 Sep cut required |
+
+---
+
+## Source 2 — 5 Cdn Inf Bde War Diary
+**File:** `5CIB_War-Diary_Sep44.pdf` (193 pp; inventoried 1–56)
+
+| Page(s) | Document type | Date covered | Note |
+|---|---|---|---|
+| 5–20 | Bde war diary narrative (Brig. Megill) | 1–30 Sep 44 | Brigade-level command perspective on the same operations; only 1–10 Sep entries in scope |
+| 22–26 | Part I Orders (Appx 1) | 15 Sep 44 | Marginal — dated outside window but routine orders often cover preceding fortnight (dress, censorship, discipline) |
+| 28–34 | AFW 3008 Field Returns of Officers (Appx 3) | wks ending 2 & 9 Sep 44 | Officer strength returns spanning focus window — establishes who was commanding what when Bloggins arrived |
+| 35–43 | AFW 3009 Field Returns of Other Ranks (Appx 4) | wks ending 2 & 9 Sep 44 | OR strength returns — directly relevant to Bloggins as reinforcement; bde-level intake picture in his first days |
+| 44 | Mov Order No 4 | 1 Sep 44 | Bde move order on Day 1 of window — order under which CH moved from Totes area |
+| 45 | Trace "T" — unit dispositions | 1 Sep 44 | Companion trace to Mov Order 4 — shows CH placement within bde at start of window |
+| 46 | Mov Order No 5 | 1 Sep 44 | Second 1 Sep move order — sequencing of bde group movement |
+| 47 | Trace "R" — route sketch | 1 Sep 44 | Route sketch for Mov Order 5 |
+| 48–49 | Mov Order No 6 | 5 Sep 44 | Mid-window move order covering advance toward Dunkerque outpost line |
+| 50 | Route Card, 2 Cdn Inf Div | 5 Sep 44 | Div-level route card companion to Mov Order 6 — divisional context for bde move |
+| 51–52 | Mov Order No 7 | 6 Sep 44 | Move order issued day before CH narrative records Bourbourgville advance — likely the order that put CH into contact |
+
+---
+
+## Source 3 — RHC Black Watch War Diary
+**File:** `RHC-Blackwatch_War-Diary_Sep44.pdf` (144 pp; inventoried 1–51)
+
+| Page(s) | Document type | Date covered | Note |
+|---|---|---|---|
+| 6–15 | Bn war diary narrative (Lt-Col Ritchie) | 1–30 Sep 44 | Sister-bn narrative; only 1–10 Sep entries in scope. BW alongside CH through Dieppe parade (3 Sep) and advance to Dunkerque approaches |
+| 18 | SHAEF "Enemy Defences" trace | 23 Aug 44 | Pre-window but describes enemy defences CH would encounter at Bourbourgville/Loon Plage from 7 Sep — operational intel context |
+| 19 | Dunkerque Plan of Port & Town | n/d | Terrain CH was advancing toward through focus window |
+| 20 | Dunkerque defence overprint map | Ed. 11 Sep 44 | Compiled from air photo info as at 1 Sep and other sources as at 10 Sep — situation across Bloggins's first ten days |
+| 21 | Defence overprint legend | n/d | Companion key to the overprint — needed to read p. 20 |
+| 24–25 | RHC Field Return of Officers | wk ending 2 Sep 44 | Sister-bn officer strength at start of window |
+| 26–27 | RHC Field Return of Officers | wk ending 9 Sep 44 | Sister-bn officer strength at close of window |
+| 33 | Field Message Form (Lt-Col Ritchie to coys) | 10 Sep 44, 0025 hrs | Only timed BW signal in window — H-hour change to 0755, RV at 853093 at 0700; bde-sector operational tempo on closing night |
+| 34–35 | RHC Field Return of Other Ranks | wk ending 3 Sep 44 | Sister-bn OR strength — comparison of reinforcement intake across CH/RHC/R de M |
+| 36–37 | RHC Field Return of Other Ranks | wk ending 10 Sep 44 | Same, end of window |
+| 50 | APIS 2 Cdn Inf Div trace, Part I | 1 Sep 44 | Divisional intel trace from first day of window — beach defences, MGs, minefields in bde sector |
+| 51 | APIS 2 Cdn Inf Div trace, Part II | ~1 Sep 44 | Continuation of p. 50 trace |
+
+---
+
+## Items considered and excluded
+
+| Source | Page(s) | Reason for exclusion |
+|---|---|---|
+| BW | 22–23, 28–29, 30–31 | Officer returns wks ending 16, 23, 30 Sep — outside window |
+| BW | 38–39, 40–41, 42–44 | OR returns wks ending 16, 23, 30 Sep — outside window |
+| BW | 46–49 | Undated coy messages and faded carbon — cannot place in window with confidence |
+| 5 CIB | 53–56 | Mov Orders 9 (17 Sep) and later — outside window |
+
+---
+
+## Gaps — material expected but not in catalogued ranges
+
+1. **R de Mais war diary inventory** — not provided. Largest gap. R de Mais was the third bn in 5 CIB, operating alongside CH throughout focus window. Without it, criterion 3 (sister-bn coordination) is half-answered.
+
+2. **Calgary Highlanders Appendices 3–15** — none catalogued. The CH Move Orders (Appx 4), Messages (Appx 5), Patrol Programme (Appx 6), Memorial Service 3 Sep (Appx 12), and March Past Dieppe (Appx 13) are all directly in scope and almost certainly the highest-value supplementary documents in the entire source set. **Inventorying CH pp. 46–343 should be the next priority.**
+
+3. **5 CIB Ops Log (Appx 13, sheets 109–161)** — un-catalogued in pp. 57–193. Bde signal log for the whole month; would be the single richest source of bde-level granularity for 1–10 Sep.
+
+4. **5 CIB Patrol Programmes and Patrol Reports (Appx 6, 7)** — un-catalogued. Patrol activity at Loon Plage from 9–10 Sep described in CH narrative; patrol reports would corroborate.
+
+5. **5 CIB Int Report on Dunkerque, 13 Sep** (Appx 10) — outside date but retrospective on focus-window area; un-catalogued.
+
+6. **BW Appx 6 (Warning Orders, movement tables)** and **Appx 8 (Sitreps)** — un-catalogued in pp. 52–144. Sitreps would map well onto focus window.
+
+7. **No casualty returns** in any catalogued range. CH narrative refers to casualties 7–10 Sep at Bourbourgville/Loon Plage; actual casualty returns (Part II Orders, AFW 3010-series, or unit casualty lists) aren't in catalogued portions of any file. Likely in CH Appx 2 (Part II Orders) or BW Appx 2 ("to follow").
+
+8. **No artillery task tables** in scope. CH Appx 3 (Arty Task Tables) and BW Appx 7 (RCA Conc tables, tasks) both un-catalogued and would directly cover fire support for actions Bloggins was in.
+
+---
+
+## Recommended next inventory pass
+
+To meaningfully increase supplementary-source density:
+
+- **CH pp. 46–100** — capture early appendices likely covering 1–10 Sep (Move Orders, Messages, Patrol Programme, Memorial Service, Dieppe March Past)
+- **5 CIB pp. 57–100** — find start of Ops Log (Appx 13) and patrol material (Appx 6, 7)
+- **R de Mais Sep 44 war diary** — full inventory pp. 1–~50 to recover sister-bn coordination evidence
+
+These three passes would roughly double the supplementary-source density of this list.