early version of OCR viewer now integrated

prompts/ocr-viewer-audit.md

@@ -0,0 +1,441 @@
+# P44 OCR Viewer — Full Rebuild Audit
+*Combined: Copilot codebase audit + Nathan/Claude session history audit*
+*Date: May 2026 — Pre-rebuild reference document*
+
+---
+
+## Purpose of this document
+
+This document captures everything known about the current `p44-ocr-viewer` codebase — what works, what is broken, what is dead, and critically **why** things broke — so the rebuild starts from a complete picture rather than repeating the same mistakes.
+
+---
+
+## Part 1 — Copilot Codebase Audit
+
+### 1. Feature Inventory
+
+**PDF Rendering and Page Navigation**
+Functions: `loadPDFBuffer()`, `renderCurrentPage()`, `buildPDFWordOverlay()`, `makeWordBox()`, `goToPage()`, `_pendingPage` queue.
+Status: Working. DPR scaling, viewport fitting, zoom multiplier, and pending-page queue are all correct. `goToPage()` guards out-of-range values, debounces re-renders, and saves edits before navigating.
+
+---
+
+**Ctrl+Scroll Zoom on the Left Panel**
+Functions: `wheel` event on `imagePanelBody`, `updateZoomLabel()`, `btnZoomReset` click handler.
+Status: **Broken.** The handler has no `e.ctrlKey` check. It calls `e.preventDefault()` unconditionally on every wheel event when a PDF is loaded. In PDF mode, the user cannot scroll the left panel with the mouse wheel at all — every wheel movement zooms instead of scrolls. Sync-scroll from the left panel via wheel is also broken in PDF mode as a result.
+
+---
+
+**Synchronized Scrolling**
+Functions: `onImgScroll()`, `onTxtScroll()`, `_syncScrolling` / `_syncActive` flags, `btnSyncScroll` click handler.
+Status: Partially working. The proportional scroll logic and rAF anti-loop are correct. Toggle button works. However because the left-panel wheel handler kills normal scroll-via-wheel in PDF mode, users can only trigger left→right sync via the scrollbar. Right→left sync works normally.
+
+---
+
+**OCR Text Loading and Per-Page Rendering**
+Functions: `loadOCRText()`, `parseOCRByPage()`, `showOCRPage()`, `escH()`.
+Status: Working. `parseOCRByPage()` splits on `## Page N` headings into `state.ocrPages`. `showOCRPage()` reads a single page, checks for corrections, renders as `<pre>`. Startup race (OCR arrives before PDF) is handled correctly.
+
+---
+
+**Confidence Scoring Display (Per-Page)**
+Functions: `buildConfidenceIndicatorHTML()`, `updateNavConfidenceColour()`, `confTier()`, `confTierLabel()`, `loadConfidenceOnStartup()`.
+Status: Working. Loads from `REAL_CONFIDENCE_PATH` on startup. Per-page indicator bar with score, tier colour, uncertain word pills, and notes is injected above OCR text. Nav input/buttons change border colour to match tier.
+
+---
+
+**Confidence Summary Panel**
+Functions: `buildConfidenceSummaryHTML()`, `btn-conf-summary` click handler, `goToPage(0)` special case.
+Status: Working. Shows High/Med/Low counts, average score, distribution bar, and 10 lowest-scoring pages as clickable pills.
+
+---
+
+**Flag System (Pass 1 Flags)**
+Functions: `setFlag()`, `updateFlagButtons()`, `updateFlagsSummary()`, `saveFlags()`, `loadFlagsOnStartup()`, `clearPass1Queue()`.
+Status: Working. Pass 1 flags stored in `state.flags`. POSTed to `/save-flags` with localStorage fallback. Loaded from `/flags.json` on startup with localStorage fallback.
+Note: Pass 2 flags stored only in `pass2Flags` module variable and saved only to `localStorage('p44_pass2_flags')` — never to the server. Asymmetry with Pass 1.
+
+---
+
+**Inline Text Editing — Pass 1**
+Functions: `enterEditMode()`, `exitEditMode()`, `btnEdit` click handler, `updateCorrectionIndicator()`, `postCorrection()`, `loadCorrectionsOnStartup()`.
+Status: Working. `enterEditMode()` snapshots raw OCR into `corrections[pageNum].original` on first open. `exitEditMode()` saves to `corrections[pageNum].pass1`. Uses `<textarea id="text-editor">` — `#text-output` is hidden during editing. Navigating away auto-saves.
+
+---
+
+**Inline Text Editing — Pass 2**
+Functions: Same as Pass 1 with `pass2` key, `showPass2RevertBar()`.
+Status: Working with a minor flow issue. `showOCRPage()` unconditionally calls `enterEditMode()` at the end. Pages with no OCR content return early before reaching that call — those pages arrive with edit mode OFF in Pass 2 role, inconsistently.
+
+---
+
+**Role Switcher (Pass 1 / Pass 2)**
+Functions: `applyRole()`, `roleSelect` change handler, `currentRole` module variable.
+Status: Working. Saves role to `localStorage('p44_dev_role')` and restores on startup. Changes flag button labels, shows/hides Pass 2 banner, updates nav button behaviour.
+
+---
+
+**Pass 2 Banner and Queue**
+Functions: `updatePass2Banner()`, `updatePass2NavButtons()`, `getVerifiedPages()`.
+Status: Working. Banner shows count of Pass 1-verified pages ready for review. Nav buttons in Pass 2 mode skip to next/previous verified page.
+
+---
+
+**Pass 2 Change Summary Panel**
+Status: **Not implemented.** No change-summary panel exists in HTML or JS. No function builds a diff or change log. Only `#pass2-revert-bar` exists with two action buttons — no summary of what was changed.
+
+---
+
+**Revert to Pass 1 / Revert to Original**
+Functions: `showPass2RevertBar()`. Two buttons injected dynamically into `#pass2-revert-bar`.
+Status: Working in Pass 2 mode when page has a `pass1` correction. "Revert to Pass 1" sets `textEditor.value` to `corr.pass1`. "Revert to Original OCR" requires typing `REVERT`, then clears `state.corrections[pageNum]`. Neither revert auto-saves — user must navigate away or toggle Edit to persist.
+
+---
+
+**Correction Storage Model**
+Current structure:
+```javascript
+state.corrections = {
+  [pageNum]: {
+    original: string,   // raw OCR snapshot, taken once on first edit-open
+    pass1:    string,   // Pass 1 editor's text (absent if unchanged from original)
+    pass2:    string,   // Pass 2 editor's text (absent if unchanged from pass1/original)
+  }
+}
+```
+Status: Consistent across all functions. `showOCRPage()` reads `pass2 || pass1 || original || rawContent` (Pass 2) or `pass1 || original || rawContent` (Pass 1). `exitEditMode()` prunes entry entirely when neither `pass1` nor `pass2` is present. `postCorrection()` sends `{ page: N, correction: corrObj }`.
+Migration: `loadCorrectionsOnStartup()` migrates old flat-string corrections to `{ pass1: oldString }`. Migration runs on startup only and does not write migrated data back to the server.
+
+---
+
+**Keyboard Navigation**
+Function: `document.addEventListener('keydown', ...)`.
+Status: **Broken on Escape.** `ArrowRight/l`, `ArrowLeft/j`, `v/r/e` work. Escape handler references `zoomLens` — a variable never declared anywhere. Throws `ReferenceError: zoomLens is not defined` every time Escape is pressed. Click-to-zoom (Phase 3) was never implemented so the element was never created.
+
+---
+
+**Clear All Data Button**
+Function: `clearAllLocalData()`.
+Status: Working for local state. Wipes all five localStorage keys, resets state, resets role to Pass 1. Calls `fetch('/clear-all-data', { method: 'POST' })` but this endpoint does not exist in `launch_viewer.py`. Failure is silently swallowed. Server-side `flags.json` and `corrections.json` are NOT cleared.
+
+---
+
+**Sample Data / Load Sample Button**
+Functions: `loadSampleData()`, `buildSampleImageDataURL()`, `SAMPLE_TEXT`, `SAMPLE_JSON`.
+Status: Working. Status bar message still references "bidirectional highlighting" — a removed feature.
+
+---
+
+**Calgary Highlanders Auto-Load**
+Function: `loadCalgaryHighlanders()`, `DOMContentLoaded` handler.
+Status: Working when served via `launch_viewer.py`.
+
+---
+
+### 2. Dead Code
+
+| Item | Location | Reason |
+|------|----------|--------|
+| `state.scale` (= 1.0) | State object | No scale dropdown exists in HTML. Never written after init, never read in any calculation. Vestigial. |
+| `state.ocrRawFull` | `loadOCRText()` | Set for plain-text/sample mode but never read again anywhere. |
+| `const base = ...` in `enterEditMode()` | Line ~1108 | Declared but never referenced. Orphan from a refactor. |
+| `#scale-select` CSS block | CSS | No element with `id="scale-select"` exists in HTML. |
+| `#text-output[contenteditable="true"]` CSS rule | CSS | Editor uses `<textarea id="text-editor">` not contenteditable on `#text-output`. Never fires. |
+| `zoomLens` reference | JS line ~1200 | Used in Escape handler but never declared. Causes runtime ReferenceError. |
+| Stale status message | `loadSampleData()` | References removed bidirectional highlighting feature. |
+| `p44_diff_legend_dismissed` in `clearAllLocalData()` | JS | Key is never written anywhere in the codebase. Only cleared. |
+
+---
+
+### 3. State Audit
+
+| Property | Type | Status |
+|----------|------|--------|
+| `mode` | `'pdf' \| 'image' \| 'none'` | Active |
+| `pdfDoc` | PDF.js document or null | Active |
+| `currentPage` | number | Active |
+| `totalPages` | number | Active |
+| `scale` | number (1.0, never changes) | **Dead** — initialized but never written or read in any live code path |
+| `zoomScale` | number | Active |
+| `wordData` | array | Active |
+| `imageNaturalW` | number | Active (image mode only) |
+| `imageNaturalH` | number | Active (image mode only) |
+| `rendering` | boolean | Active |
+| `ocrPages` | `{ [pageNum]: string }` | Active |
+| `ocrRawFull` | string | **Dead** — written in plain-text path, never read again |
+| `lastRenderTime` | number | Active |
+| `flags` | object | Active |
+| `corrections` | object | Active |
+| `editMode` | boolean | Active |
+| `confidenceData` | object | Active |
+
+---
+
+### 4. localStorage Audit
+
+| Key | Written by | Read by | Notes |
+|-----|-----------|---------|-------|
+| `p44-flags` | `saveFlags()` | `loadFlagsOnStartup()` | Fallback when server unavailable |
+| `p44-corrections` | `postCorrection()` fallback | `loadCorrectionsOnStartup()` | Also written during startup migration |
+| `p44_pass2_flags` | `savePass2Flags()` | `loadPass2FlagsOnStartup()` | **Only storage for Pass 2 flags — no server endpoint** |
+| `p44_dev_role` | `applyRole()` | Startup IIFE | Persists role across refreshes |
+| `p44_diff_legend_dismissed` | **Never written** | **Never read** | Only appears in `clearAllLocalData()` as a `removeItem()` call. Feature was removed without cleanup. |
+
+---
+
+### 5. Known Broken or Incomplete
+
+1. **`zoomLens` ReferenceError (hard crash on Escape).** `zoomLens` is never declared. Every Escape keypress throws `ReferenceError`.
+
+2. **Wheel zoom intercepts all scroll, not Ctrl+scroll.** No `e.ctrlKey` check. Left panel cannot be scrolled with mouse wheel in PDF mode.
+
+3. **`updateFlagsSummary()` silently ignores Pass 2 flags.** Footer counts only `state.flags` (Pass 1). Pass 2 flags never included in summary even when in Pass 2 mode.
+
+4. **`savePass2Flags()` never saves to server.** Pass 2 flags are lost if browser storage is cleared or a different machine is used.
+
+5. **`/clear-all-data` POST endpoint does not exist in `launch_viewer.py`.** Server files not cleared when user clicks "Clear all data".
+
+6. **`state.ocrRawFull` is populated but never used.** Latent issue if plain-text path is extended.
+
+7. **`p44_diff_legend_dismissed` cleared but never written.** Dead localStorage reference.
+
+8. **`showPass2RevertBar()` adds fresh event listeners on each call.** Low risk but possible accumulation in edge cases.
+
+9. **Corrections migration does not update server file.** `corrections.json` on disk stays in old format after migration. Re-runs from stale file on every server restart.
+
+10. **Stale status message in `loadSampleData()`.** References removed bidirectional highlighting feature.
+
+---
+
+### 6. Data Model Audit — `state.corrections`
+
+```javascript
+state.corrections = {
+  1: {
+    original: "raw OCR text at moment editor first opened",
+    pass1:    "Pass 1 volunteer's saved text",   // absent if unchanged from original
+    pass2:    "Pass 2 editor's saved text",       // absent if unchanged from pass1/original
+  }
+}
+```
+
+Consistent across all read/write sites. `original` is preserved in memory but pruned from persisted object if both `pass1` and `pass2` are absent. So `original` is a transient in-session anchor, not guaranteed to be in stored JSON.
+
+---
+
+## Part 2 — Nathan/Claude Session History Audit
+
+*What actually happened during development — the decisions made, why things broke, and what to do differently in the rebuild.*
+
+---
+
+### The fundamental architectural mistake
+
+**`textOutput` was used as display container, edit surface, and diff viewer simultaneously.**
+
+These three roles have incompatible requirements:
+- Display container needs rendered HTML (`innerHTML`)
+- Edit surface needs plain text capture (`innerText`)
+- Diff viewer needs highlight spans (`innerHTML` with CSS classes)
+
+`innerHTML` → `innerText` on a div containing tables, headings, and diff spans is unpredictable. Tables become tab-separated text. Headings lose formatting. Diff spans get their text content extracted without markup. This single decision caused the majority of bugs encountered:
+- HTML tags appearing as literal text in corrections
+- Table structure being lost when pages were edited
+- Pass 1 edits bleeding onto Pass 2 pages
+- False-positive diffs from whitespace mismatches
+- Confidence indicator text being captured into corrections
+
+The fix — a separate `<textarea id="text-editor">` that is shown/hidden while `#text-output` is hidden/shown — was the correct architectural decision but took too long to reach because symptoms were being patched instead of root causes diagnosed.
+
+**In the rebuild: display and editing must be separate elements from day one.**
+
+---
+
+### The markdown/HTML rendering dead end
+
+olmOCR was asked to output "clean markdown" but war diary pages are tabular. The model correctly used HTML tables because markdown tables cannot handle multi-line cells with `<br>` tags. This produced inconsistent output — some pages as prose, some as HTML tables, some mixed.
+
+A hand-rolled `convertMarkdown()` parser was built to handle this. It required repeated patching:
+- First version missed indented HTML blocks (`<tr>` lines starting with spaces)
+- Second version added depth tracking for HTML blocks
+- Third version was still producing inconsistent output
+
+Six separate debugging sessions were spent on markdown rendering issues that had nothing to do with the actual feature being worked on at the time.
+
+**In the rebuild: OCR prompt outputs plain text only. No markdown parser needed. `<pre>` display of plain text. Zero rendering complexity.**
+
+---
+
+### The diff system — built and abandoned three times
+
+**Attempt 1:** Text matching — highlighted all instances of the same word. "The" highlighted every "The" on the page. Fundamental design flaw — no unique link between words.
+
+**Attempt 2:** `contentEditable` + `innerText` diff — caused cross-page bleed (page N edits appearing on page N+1) and false positives from HTML whitespace.
+
+**Attempt 3:** `stripHTML()` on both sides — `stripHTML(markdown)` and `stripHTML(innerText)` produce different whitespace patterns for the same content even when text is identical. False positives on every page.
+
+All three attempts failed for the same root reason: the input to the diff was contaminated HTML, not clean text. With plain text OCR this is now permanently solved. A simple word-array diff will work cleanly.
+
+**In the rebuild: diff runs on plain text strings only. No stripHTML needed. Build it once.**
+
+---
+
+### Features added before foundations were stable
+
+The build sequence was:
+1. PDF rendering ✓
+2. OCR loading ✓
+3. Sync scroll ✓ (partially broken by zoom later)
+4. Zoom — broke scroll
+5. Flags ✓
+6. Edit mode — introduced `contentEditable` architectural flaw
+7. Confidence display ✓
+8. Pass 2 mode + diff — built on broken edit foundation
+9. Role switcher ✓
+10. Change summary — never actually implemented despite multiple prompts
+11. Revert system — partially implemented
+
+Each feature after #6 was built on top of a broken edit/save loop. Every new feature added complexity that made the underlying bugs harder to isolate.
+
+**In the rebuild: get edit/save 100% correct before building Pass 2 mode. Get Pass 2 mode working before building change summary. No feature ships until the feature before it is solid.**
+
+---
+
+### Copilot context loss in large files
+
+When everything lived in one 2,176-line HTML file, Copilot would:
+- Apply changes in the wrong function (similar names, wrong location)
+- Miss functions entirely (outside its context window)
+- Partially implement features (add HTML but forget JS, or vice versa)
+- Introduce new bugs while fixing old ones (touching code it wasn't asked to touch)
+
+Splitting into HTML/CSS/JS reduced this significantly. The remaining issues were smaller in scope and easier to diagnose.
+
+**In the rebuild: three files from day one. Each prompt specifies exactly which file and which function. No prompt touches more than one file at a time unless unavoidable.**
+
+---
+
+### The OCR pipeline decision history
+
+Original prompt: "Output clean markdown." → Inconsistent markdown+HTML output → hand-rolled parser → cascading failures.
+
+Revised prompt: "Plain text only, no tables, transcribe war diary entries as prose." → Model made editorial decisions about what was "diary content" → omitted text that humans should see.
+
+Final prompt: "Transcribe ALL text visible on the page. Do not decide what is important." → Correct.
+
+**In the rebuild: use the final prompt. Never ask the model to interpret or select content.**
+
+---
+
+### What was never built but was promised
+
+These features were designed in full detail during the session but never actually implemented:
+
+1. **Pass 2 change summary panel** — `buildChangeSummary()` function spec written, HTML/CSS specified, Copilot prompt written. Never confirmed working in a screenshot.
+
+2. **`/clear-all-data` server endpoint** — specified for `launch_viewer.py`. Copilot prompt written. Endpoint confirmed missing by audit.
+
+3. **Pass 2 flags saved to server** — asymmetry identified but never fixed.
+
+4. **`/clear-all-data` in `launch_viewer.py`** — POST handler specified but never confirmed added.
+
+5. **Scroll-to-region approximate highlighting** — discussed as a replacement for word-level cross-highlighting. Never implemented. Deprioritized in favour of other features.
+
+---
+
+### Persistent user frustrations (verbatim from session)
+
+- *"stuff keeps breaking"* — after every Copilot fix session
+- *"are we just applying bandaids?"* — after the fourth markdown patch
+- *"it broke the output"* — diff showing tags as literal text
+- *"now its marking stuff that wasnt changed as changed"* — false-positive diff on every word
+- *"still getting the same old errors"* — after file split, still hitting edit/save bugs
+- *"I am still getting the page numbers bleeding into the output"* — `## Page N` appearing in editor
+
+---
+
+## Part 3 — Rebuild Specification
+
+### What to keep from the old codebase (working, clean, reuse as-is)
+
+- `loadPDFBuffer()` and `renderCurrentPage()` — solid, no changes needed
+- `buildPDFWordOverlay()` and `makeWordBox()` — solid
+- `parseOCRByPage()` — clean, keep exactly as-is
+- `loadConfidenceOnStartup()` and all confidence display functions — working
+- `saveFlags()` / `loadFlagsOnStartup()` / `setFlag()` / `updateFlagButtons()` — working
+- `applyRole()` / `getVerifiedPages()` / `updatePass2Banner()` — working
+- `goToPage()` — working, keep the edit-save guard
+- Sync scroll logic — working (fix wheel zoom first)
+- CSS variables and visual design — keep the aesthetic
+
+---
+
+### What to rebuild from scratch
+
+**Edit system — new architecture:**
+```
+#text-output    — display only, never editable, always innerHTML
+#text-editor    — <textarea>, plain text only, shown only in edit mode
+                  pre-populated from corrections.pass1 or corrections.original
+                  hidden when not editing
+```
+
+No `contentEditable` anywhere. Ever.
+
+**Correction data model — keep the object structure, rebuild the plumbing:**
+```javascript
+corrections[pageNum] = {
+  original: string,  // snapshotted once on first edit-open
+  pass1:    string,  // saved by Pass 1 on exit
+  pass2:    string,  // saved by Pass 2 on exit
+}
+```
+
+**Change summary — build it once, correctly:**
+- Plain text word-array diff: `original.split(/\s+/)` vs `pass1.split(/\s+/)`
+- Output a simple list: changed / added / deleted words
+- Read-only panel above textarea in Pass 2 mode
+- No inline highlighting — just a clean change log
+
+**`launch_viewer.py` — add missing endpoints:**
+- `/save-corrections` — write `corrections.json`
+- `/save-flags` — write `flags.json` (already exists, verify)
+- `/save-pass2-flags` — write `pass2_flags.json` (new)
+- `/clear-all-data` — delete all three JSON files
+
+---
+
+### Build order for the rebuild
+
+1. HTML/CSS shell — layout only, no JS
+2. PDF loading and page navigation
+3. OCR loading and plain text display (`<pre>`)
+4. Confidence display
+5. Flags (Pass 1)
+6. **Edit mode — Pass 1 only, textarea architecture. Do not proceed until 100% correct.**
+7. Corrections storage and server persistence
+8. Role switcher and Pass 2 mode
+9. Pass 2 change summary panel
+10. Revert system
+11. Sync scroll (fix wheel zoom at same time)
+12. Keyboard navigation (fix `zoomLens` reference)
+13. Clear all data (with working server endpoint)
+
+**Each step must be confirmed working before the next step starts.**
+
+---
+
+### localStorage keys for the rebuild
+
+| Key | Purpose | Server backup |
+|-----|---------|--------------|
+| `p44-flags` | Pass 1 flags | `flags.json` |
+| `p44-pass2-flags` | Pass 2 flags | `pass2_flags.json` |
+| `p44-corrections` | All corrections (object model) | `corrections.json` |
+| `p44-role` | Current role | None needed |
+
+Remove: `p44_diff_legend_dismissed` (feature gone), `p44_dev_role` (rename to `p44-role` for consistency).
+
+---
+
+*End of audit. This document is the complete reference for the rebuild.*