# P44 OCR Viewer — Full Rebuild Audit
*Combined: Copilot codebase audit + Nathan/Claude session history audit*
*Date: May 2026 — Pre-rebuild reference document*

---

## Purpose of this document

This document captures everything known about the current `p44-ocr-viewer` codebase — what works, what is broken, what is dead, and critically **why** things broke — so the rebuild starts from a complete picture rather than repeating the same mistakes.

---

## Part 1 — Copilot Codebase Audit

### 1. Feature Inventory

**PDF Rendering and Page Navigation**
Functions: `loadPDFBuffer()`, `renderCurrentPage()`, `buildPDFWordOverlay()`, `makeWordBox()`, `goToPage()`, `_pendingPage` queue.
Status: Working. DPR scaling, viewport fitting, zoom multiplier, and pending-page queue are all correct. `goToPage()` guards out-of-range values, debounces re-renders, and saves edits before navigating.

---

**Ctrl+Scroll Zoom on the Left Panel**
Functions: `wheel` event on `imagePanelBody`, `updateZoomLabel()`, `btnZoomReset` click handler.
Status: **Broken.** The handler has no `e.ctrlKey` check. It calls `e.preventDefault()` unconditionally on every wheel event when a PDF is loaded. In PDF mode, the user cannot scroll the left panel with the mouse wheel at all — every wheel movement zooms instead of scrolls. Sync-scroll from the left panel via wheel is also broken in PDF mode as a result.

---

**Synchronized Scrolling**
Functions: `onImgScroll()`, `onTxtScroll()`, `_syncScrolling` / `_syncActive` flags, `btnSyncScroll` click handler.
Status: Partially working. The proportional scroll logic and rAF anti-loop are correct. Toggle button works. However because the left-panel wheel handler kills normal scroll-via-wheel in PDF mode, users can only trigger left→right sync via the scrollbar. Right→left sync works normally.

---

**OCR Text Loading and Per-Page Rendering**
Functions: `loadOCRText()`, `parseOCRByPage()`, `showOCRPage()`, `escH()`.
Status: Working. `parseOCRByPage()` splits on `## Page N` headings into `state.ocrPages`. `showOCRPage()` reads a single page, checks for corrections, renders as `<pre>`. Startup race (OCR arrives before PDF) is handled correctly.

---

**Confidence Scoring Display (Per-Page)**
Functions: `buildConfidenceIndicatorHTML()`, `updateNavConfidenceColour()`, `confTier()`, `confTierLabel()`, `loadConfidenceOnStartup()`.
Status: Working. Loads from `REAL_CONFIDENCE_PATH` on startup. Per-page indicator bar with score, tier colour, uncertain word pills, and notes is injected above OCR text. Nav input/buttons change border colour to match tier.

---

**Confidence Summary Panel**
Functions: `buildConfidenceSummaryHTML()`, `btn-conf-summary` click handler, `goToPage(0)` special case.
Status: Working. Shows High/Med/Low counts, average score, distribution bar, and 10 lowest-scoring pages as clickable pills.

---

**Flag System (Pass 1 Flags)**
Functions: `setFlag()`, `updateFlagButtons()`, `updateFlagsSummary()`, `saveFlags()`, `loadFlagsOnStartup()`, `clearPass1Queue()`.
Status: Working. Pass 1 flags stored in `state.flags`. POSTed to `/save-flags` with localStorage fallback. Loaded from `/flags.json` on startup with localStorage fallback.
Note: Pass 2 flags stored only in `pass2Flags` module variable and saved only to `localStorage('p44_pass2_flags')` — never to the server. Asymmetry with Pass 1.

---

**Inline Text Editing — Pass 1**
Functions: `enterEditMode()`, `exitEditMode()`, `btnEdit` click handler, `updateCorrectionIndicator()`, `postCorrection()`, `loadCorrectionsOnStartup()`.
Status: Working. `enterEditMode()` snapshots raw OCR into `corrections[pageNum].original` on first open. `exitEditMode()` saves to `corrections[pageNum].pass1`. Uses `<textarea id="text-editor">` — `#text-output` is hidden during editing. Navigating away auto-saves.

---

**Inline Text Editing — Pass 2**
Functions: Same as Pass 1 with `pass2` key, `showPass2RevertBar()`.
Status: Working with a minor flow issue. `showOCRPage()` unconditionally calls `enterEditMode()` at the end. Pages with no OCR content return early before reaching that call — those pages arrive with edit mode OFF in Pass 2 role, inconsistently.

---

**Role Switcher (Pass 1 / Pass 2)**
Functions: `applyRole()`, `roleSelect` change handler, `currentRole` module variable.
Status: Working. Saves role to `localStorage('p44_dev_role')` and restores on startup. Changes flag button labels, shows/hides Pass 2 banner, updates nav button behaviour.

---

**Pass 2 Banner and Queue**
Functions: `updatePass2Banner()`, `updatePass2NavButtons()`, `getVerifiedPages()`.
Status: Working. Banner shows count of Pass 1-verified pages ready for review. Nav buttons in Pass 2 mode skip to next/previous verified page.

---

**Pass 2 Change Summary Panel**
Status: **Not implemented.** No change-summary panel exists in HTML or JS. No function builds a diff or change log. Only `#pass2-revert-bar` exists with two action buttons — no summary of what was changed.

---

**Revert to Pass 1 / Revert to Original**
Functions: `showPass2RevertBar()`. Two buttons injected dynamically into `#pass2-revert-bar`.
Status: Working in Pass 2 mode when page has a `pass1` correction. "Revert to Pass 1" sets `textEditor.value` to `corr.pass1`. "Revert to Original OCR" requires typing `REVERT`, then clears `state.corrections[pageNum]`. Neither revert auto-saves — user must navigate away or toggle Edit to persist.

---

**Correction Storage Model**
Current structure:
```javascript
state.corrections = {
  [pageNum]: {
    original: string,   // raw OCR snapshot, taken once on first edit-open
    pass1:    string,   // Pass 1 editor's text (absent if unchanged from original)
    pass2:    string,   // Pass 2 editor's text (absent if unchanged from pass1/original)
  }
}
```
Status: Consistent across all functions. `showOCRPage()` reads `pass2 || pass1 || original || rawContent` (Pass 2) or `pass1 || original || rawContent` (Pass 1). `exitEditMode()` prunes entry entirely when neither `pass1` nor `pass2` is present. `postCorrection()` sends `{ page: N, correction: corrObj }`.
Migration: `loadCorrectionsOnStartup()` migrates old flat-string corrections to `{ pass1: oldString }`. Migration runs on startup only and does not write migrated data back to the server.

---

**Keyboard Navigation**
Function: `document.addEventListener('keydown', ...)`.
Status: **Broken on Escape.** `ArrowRight/l`, `ArrowLeft/j`, `v/r/e` work. Escape handler references `zoomLens` — a variable never declared anywhere. Throws `ReferenceError: zoomLens is not defined` every time Escape is pressed. Click-to-zoom (Phase 3) was never implemented so the element was never created.

---

**Clear All Data Button**
Function: `clearAllLocalData()`.
Status: Working for local state. Wipes all five localStorage keys, resets state, resets role to Pass 1. Calls `fetch('/clear-all-data', { method: 'POST' })` but this endpoint does not exist in `launch_viewer.py`. Failure is silently swallowed. Server-side `flags.json` and `corrections.json` are NOT cleared.

---

**Sample Data / Load Sample Button**
Functions: `loadSampleData()`, `buildSampleImageDataURL()`, `SAMPLE_TEXT`, `SAMPLE_JSON`.
Status: Working. Status bar message still references "bidirectional highlighting" — a removed feature.

---

**Calgary Highlanders Auto-Load**
Function: `loadCalgaryHighlanders()`, `DOMContentLoaded` handler.
Status: Working when served via `launch_viewer.py`.

---

### 2. Dead Code

| Item | Location | Reason |
|------|----------|--------|
| `state.scale` (= 1.0) | State object | No scale dropdown exists in HTML. Never written after init, never read in any calculation. Vestigial. |
| `state.ocrRawFull` | `loadOCRText()` | Set for plain-text/sample mode but never read again anywhere. |
| `const base = ...` in `enterEditMode()` | Line ~1108 | Declared but never referenced. Orphan from a refactor. |
| `#scale-select` CSS block | CSS | No element with `id="scale-select"` exists in HTML. |
| `#text-output[contenteditable="true"]` CSS rule | CSS | Editor uses `<textarea id="text-editor">` not contenteditable on `#text-output`. Never fires. |
| `zoomLens` reference | JS line ~1200 | Used in Escape handler but never declared. Causes runtime ReferenceError. |
| Stale status message | `loadSampleData()` | References removed bidirectional highlighting feature. |
| `p44_diff_legend_dismissed` in `clearAllLocalData()` | JS | Key is never written anywhere in the codebase. Only cleared. |

---

### 3. State Audit

| Property | Type | Status |
|----------|------|--------|
| `mode` | `'pdf' \| 'image' \| 'none'` | Active |
| `pdfDoc` | PDF.js document or null | Active |
| `currentPage` | number | Active |
| `totalPages` | number | Active |
| `scale` | number (1.0, never changes) | **Dead** — initialized but never written or read in any live code path |
| `zoomScale` | number | Active |
| `wordData` | array | Active |
| `imageNaturalW` | number | Active (image mode only) |
| `imageNaturalH` | number | Active (image mode only) |
| `rendering` | boolean | Active |
| `ocrPages` | `{ [pageNum]: string }` | Active |
| `ocrRawFull` | string | **Dead** — written in plain-text path, never read again |
| `lastRenderTime` | number | Active |
| `flags` | object | Active |
| `corrections` | object | Active |
| `editMode` | boolean | Active |
| `confidenceData` | object | Active |

---

### 4. localStorage Audit

| Key | Written by | Read by | Notes |
|-----|-----------|---------|-------|
| `p44-flags` | `saveFlags()` | `loadFlagsOnStartup()` | Fallback when server unavailable |
| `p44-corrections` | `postCorrection()` fallback | `loadCorrectionsOnStartup()` | Also written during startup migration |
| `p44_pass2_flags` | `savePass2Flags()` | `loadPass2FlagsOnStartup()` | **Only storage for Pass 2 flags — no server endpoint** |
| `p44_dev_role` | `applyRole()` | Startup IIFE | Persists role across refreshes |
| `p44_diff_legend_dismissed` | **Never written** | **Never read** | Only appears in `clearAllLocalData()` as a `removeItem()` call. Feature was removed without cleanup. |

---

### 5. Known Broken or Incomplete

1. **`zoomLens` ReferenceError (hard crash on Escape).** `zoomLens` is never declared. Every Escape keypress throws `ReferenceError`.

2. **Wheel zoom intercepts all scroll, not Ctrl+scroll.** No `e.ctrlKey` check. Left panel cannot be scrolled with mouse wheel in PDF mode.

3. **`updateFlagsSummary()` silently ignores Pass 2 flags.** Footer counts only `state.flags` (Pass 1). Pass 2 flags never included in summary even when in Pass 2 mode.

4. **`savePass2Flags()` never saves to server.** Pass 2 flags are lost if browser storage is cleared or a different machine is used.

5. **`/clear-all-data` POST endpoint does not exist in `launch_viewer.py`.** Server files not cleared when user clicks "Clear all data".

6. **`state.ocrRawFull` is populated but never used.** Latent issue if plain-text path is extended.

7. **`p44_diff_legend_dismissed` cleared but never written.** Dead localStorage reference.

8. **`showPass2RevertBar()` adds fresh event listeners on each call.** Low risk but possible accumulation in edge cases.

9. **Corrections migration does not update server file.** `corrections.json` on disk stays in old format after migration. Re-runs from stale file on every server restart.

10. **Stale status message in `loadSampleData()`.** References removed bidirectional highlighting feature.

---

### 6. Data Model Audit — `state.corrections`

```javascript
state.corrections = {
  1: {
    original: "raw OCR text at moment editor first opened",
    pass1:    "Pass 1 volunteer's saved text",   // absent if unchanged from original
    pass2:    "Pass 2 editor's saved text",       // absent if unchanged from pass1/original
  }
}
```

Consistent across all read/write sites. `original` is preserved in memory but pruned from persisted object if both `pass1` and `pass2` are absent. So `original` is a transient in-session anchor, not guaranteed to be in stored JSON.

---

## Part 2 — Nathan/Claude Session History Audit

*What actually happened during development — the decisions made, why things broke, and what to do differently in the rebuild.*

---

### The fundamental architectural mistake

**`textOutput` was used as display container, edit surface, and diff viewer simultaneously.**

These three roles have incompatible requirements:
- Display container needs rendered HTML (`innerHTML`)
- Edit surface needs plain text capture (`innerText`)
- Diff viewer needs highlight spans (`innerHTML` with CSS classes)

`innerHTML` → `innerText` on a div containing tables, headings, and diff spans is unpredictable. Tables become tab-separated text. Headings lose formatting. Diff spans get their text content extracted without markup. This single decision caused the majority of bugs encountered:
- HTML tags appearing as literal text in corrections
- Table structure being lost when pages were edited
- Pass 1 edits bleeding onto Pass 2 pages
- False-positive diffs from whitespace mismatches
- Confidence indicator text being captured into corrections

The fix — a separate `<textarea id="text-editor">` that is shown/hidden while `#text-output` is hidden/shown — was the correct architectural decision but took too long to reach because symptoms were being patched instead of root causes diagnosed.

**In the rebuild: display and editing must be separate elements from day one.**

---

### The markdown/HTML rendering dead end

olmOCR was asked to output "clean markdown" but war diary pages are tabular. The model correctly used HTML tables because markdown tables cannot handle multi-line cells with `<br>` tags. This produced inconsistent output — some pages as prose, some as HTML tables, some mixed.

A hand-rolled `convertMarkdown()` parser was built to handle this. It required repeated patching:
- First version missed indented HTML blocks (`<tr>` lines starting with spaces)
- Second version added depth tracking for HTML blocks
- Third version was still producing inconsistent output

Six separate debugging sessions were spent on markdown rendering issues that had nothing to do with the actual feature being worked on at the time.

**In the rebuild: OCR prompt outputs plain text only. No markdown parser needed. `<pre>` display of plain text. Zero rendering complexity.**

---

### The diff system — built and abandoned three times

**Attempt 1:** Text matching — highlighted all instances of the same word. "The" highlighted every "The" on the page. Fundamental design flaw — no unique link between words.

**Attempt 2:** `contentEditable` + `innerText` diff — caused cross-page bleed (page N edits appearing on page N+1) and false positives from HTML whitespace.

**Attempt 3:** `stripHTML()` on both sides — `stripHTML(markdown)` and `stripHTML(innerText)` produce different whitespace patterns for the same content even when text is identical. False positives on every page.

All three attempts failed for the same root reason: the input to the diff was contaminated HTML, not clean text. With plain text OCR this is now permanently solved. A simple word-array diff will work cleanly.

**In the rebuild: diff runs on plain text strings only. No stripHTML needed. Build it once.**

---

### Features added before foundations were stable

The build sequence was:
1. PDF rendering ✓
2. OCR loading ✓
3. Sync scroll ✓ (partially broken by zoom later)
4. Zoom — broke scroll
5. Flags ✓
6. Edit mode — introduced `contentEditable` architectural flaw
7. Confidence display ✓
8. Pass 2 mode + diff — built on broken edit foundation
9. Role switcher ✓
10. Change summary — never actually implemented despite multiple prompts
11. Revert system — partially implemented

Each feature after #6 was built on top of a broken edit/save loop. Every new feature added complexity that made the underlying bugs harder to isolate.

**In the rebuild: get edit/save 100% correct before building Pass 2 mode. Get Pass 2 mode working before building change summary. No feature ships until the feature before it is solid.**

---

### Copilot context loss in large files

When everything lived in one 2,176-line HTML file, Copilot would:
- Apply changes in the wrong function (similar names, wrong location)
- Miss functions entirely (outside its context window)
- Partially implement features (add HTML but forget JS, or vice versa)
- Introduce new bugs while fixing old ones (touching code it wasn't asked to touch)

Splitting into HTML/CSS/JS reduced this significantly. The remaining issues were smaller in scope and easier to diagnose.

**In the rebuild: three files from day one. Each prompt specifies exactly which file and which function. No prompt touches more than one file at a time unless unavoidable.**

---

### The OCR pipeline decision history

Original prompt: "Output clean markdown." → Inconsistent markdown+HTML output → hand-rolled parser → cascading failures.

Revised prompt: "Plain text only, no tables, transcribe war diary entries as prose." → Model made editorial decisions about what was "diary content" → omitted text that humans should see.

Final prompt: "Transcribe ALL text visible on the page. Do not decide what is important." → Correct.

**In the rebuild: use the final prompt. Never ask the model to interpret or select content.**

---

### What was never built but was promised

These features were designed in full detail during the session but never actually implemented:

1. **Pass 2 change summary panel** — `buildChangeSummary()` function spec written, HTML/CSS specified, Copilot prompt written. Never confirmed working in a screenshot.

2. **`/clear-all-data` server endpoint** — specified for `launch_viewer.py`. Copilot prompt written. Endpoint confirmed missing by audit.

3. **Pass 2 flags saved to server** — asymmetry identified but never fixed.

4. **`/clear-all-data` in `launch_viewer.py`** — POST handler specified but never confirmed added.

5. **Scroll-to-region approximate highlighting** — discussed as a replacement for word-level cross-highlighting. Never implemented. Deprioritized in favour of other features.

---

### Persistent user frustrations (verbatim from session)

- *"stuff keeps breaking"* — after every Copilot fix session
- *"are we just applying bandaids?"* — after the fourth markdown patch
- *"it broke the output"* — diff showing tags as literal text
- *"now its marking stuff that wasnt changed as changed"* — false-positive diff on every word
- *"still getting the same old errors"* — after file split, still hitting edit/save bugs
- *"I am still getting the page numbers bleeding into the output"* — `## Page N` appearing in editor

---

## Part 3 — Rebuild Specification

### What to keep from the old codebase (working, clean, reuse as-is)

- `loadPDFBuffer()` and `renderCurrentPage()` — solid, no changes needed
- `buildPDFWordOverlay()` and `makeWordBox()` — solid
- `parseOCRByPage()` — clean, keep exactly as-is
- `loadConfidenceOnStartup()` and all confidence display functions — working
- `saveFlags()` / `loadFlagsOnStartup()` / `setFlag()` / `updateFlagButtons()` — working
- `applyRole()` / `getVerifiedPages()` / `updatePass2Banner()` — working
- `goToPage()` — working, keep the edit-save guard
- Sync scroll logic — working (fix wheel zoom first)
- CSS variables and visual design — keep the aesthetic

---

### What to rebuild from scratch

**Edit system — new architecture:**
```
#text-output    — display only, never editable, always innerHTML
#text-editor    — <textarea>, plain text only, shown only in edit mode
                  pre-populated from corrections.pass1 or corrections.original
                  hidden when not editing
```

No `contentEditable` anywhere. Ever.

**Correction data model — keep the object structure, rebuild the plumbing:**
```javascript
corrections[pageNum] = {
  original: string,  // snapshotted once on first edit-open
  pass1:    string,  // saved by Pass 1 on exit
  pass2:    string,  // saved by Pass 2 on exit
}
```

**Change summary — build it once, correctly:**
- Plain text word-array diff: `original.split(/\s+/)` vs `pass1.split(/\s+/)`
- Output a simple list: changed / added / deleted words
- Read-only panel above textarea in Pass 2 mode
- No inline highlighting — just a clean change log

**`launch_viewer.py` — add missing endpoints:**
- `/save-corrections` — write `corrections.json`
- `/save-flags` — write `flags.json` (already exists, verify)
- `/save-pass2-flags` — write `pass2_flags.json` (new)
- `/clear-all-data` — delete all three JSON files

---

### Build order for the rebuild

1. HTML/CSS shell — layout only, no JS
2. PDF loading and page navigation
3. OCR loading and plain text display (`<pre>`)
4. Confidence display
5. Flags (Pass 1)
6. **Edit mode — Pass 1 only, textarea architecture. Do not proceed until 100% correct.**
7. Corrections storage and server persistence
8. Role switcher and Pass 2 mode
9. Pass 2 change summary panel
10. Revert system
11. Sync scroll (fix wheel zoom at same time)
12. Keyboard navigation (fix `zoomLens` reference)
13. Clear all data (with working server endpoint)

**Each step must be confirmed working before the next step starts.**

---

### localStorage keys for the rebuild

| Key | Purpose | Server backup |
|-----|---------|--------------|
| `p44-flags` | Pass 1 flags | `flags.json` |
| `p44-pass2-flags` | Pass 2 flags | `pass2_flags.json` |
| `p44-corrections` | All corrections (object model) | `corrections.json` |
| `p44-role` | Current role | None needed |

Remove: `p44_diff_legend_dismissed` (feature gone), `p44_dev_role` (rename to `p44-role` for consistency).

---

*End of audit. This document is the complete reference for the rebuild.*