# Server & Data API — Context for Claude

## What this is

FastAPI server (`bristlenose serve`) that serves a React SPA (React Router, pathname-based routes) over HTTP, provides REST API endpoints for researcher state, and stores all pipeline output + researcher edits in SQLite. In serve mode, `app.py` replaces `<!-- bn-app -->` markers in the rendered HTML with `<div id="bn-app-root">`; React Router handles all client-side navigation (tab switching, session drill-down, back/forward). The SPA catch-all route (`GET /report/{path:path}`) returns the same transformed HTML for all `/report/*` paths. Transcript HTML files from `bristlenose render` are still served directly (route defined before the catch-all for priority).

## Architecture

```
bristlenose/server/
  app.py          — FastAPI factory (create_app), dev-mode HTML injection
  db.py           — SQLAlchemy engine, session factory, SQLite with WAL + FK
  models.py       — 24 ORM tables (see docs/design-serve-milestone-1.md)
  importer.py     — Pipeline JSON → SQLite on startup (idempotent upsert)
  admin.py        — SQLAdmin browser (dev-only, /admin/)
  autocode.py     — AutoCode engine (taxonomy, batching, async job runner)
  routes/
    health.py     — GET /api/health (status/version + footer links/feedback config)
    sessions.py   — GET /api/projects/{id}/sessions (React sessions table, includes source_folder_uri)
    quotes.py     — GET /api/projects/{id}/quotes (quotes grouped by section/theme)
    data.py       — 12 data API endpoints (Phase 1 researcher state sync)
    autocode.py   — 7 AutoCode endpoints (start, status, proposals, accept/deny)
    dev.py        — Dev-only endpoints (visual diff, system info)
  codebook/       — YAML codebook templates (garrett, norman, uxr, plato)
  static/         — Vite build output (React Router SPA bundle)
```

## Data API (Phase 1)

The data API replaces localStorage with HTTP.  Each endpoint mirrors one localStorage key.  The vanilla JS modules call `apiPut()` after every `localStorage.setItem()` — fire-and-forget background sync.

### Endpoints

| Method | Path | localStorage key | JS module | DB tables |
|--------|------|------------------|-----------|-----------|
| GET/PUT | `/projects/{id}/people` | `bristlenose-names` | `names.js` | Person, SessionSpeaker |
| GET/PUT | `/projects/{id}/edits` | `bristlenose-edits` | `editing.js` | QuoteEdit, HeadingEdit |
| GET/PUT | `/projects/{id}/tags` | `bristlenose-tags` | `tags.js` | QuoteTag, TagDefinition |
| GET/PUT | `/projects/{id}/hidden` | `bristlenose-hidden` | `hidden.js` | QuoteState |
| GET/PUT | `/projects/{id}/starred` | `bristlenose-starred` | `starred.js` | QuoteState |
| GET/PUT | `/projects/{id}/deleted-badges` | `bristlenose-deleted-badges` | `tags.js` | DeletedBadge |

### Data flow

```
User clicks star → JS updates DOM + in-memory map
                 → localStorage.setItem() (immediate, synchronous)
                 → if (isServeMode()) apiPut('/starred', map) (async, fire-and-forget)
                                        ↓
                                   FastAPI route
                                        ↓
                                   SQLite upsert
```

**localStorage remains the source of truth** for the current page session.  The API is a background sync — if it fails, a toast shows but no rollback occurs.

### Serve-mode detection

The server injects `<script>window.BRISTLENOSE_API_BASE = '/api/projects/1';</script>` into the HTML at serve time (`app.py:_mount_dev_report`).  When absent (static HTML opened from disk), `isServeMode()` returns false and all API calls are silently skipped.

### Quote DOM ID ↔ DB mapping

The static HTML uses DOM IDs like `q-p1-123` (format: `q-{participant_id}-{int(start_timecode)}`).  The API accepts these DOM-style IDs and resolves them server-side:

- **`_parse_dom_quote_id("q-p1-123")`** → `("p1", 123)`
- **`_resolve_quote(db, project_id, "q-p1-123")`** → queries `Quote` where `participant_id="p1"` and `start_timecode` is in `[123, 124)` (the `int()` truncation means we need a range match)

This is in `routes/data.py`.  The same format is generated by `render/quote_format.py` (`f"q-{quote.participant_id}-{int(quote.start_timecode)}"`).

### Bulk maps, not individual PATCH

All endpoints use GET/PUT of full state maps (not per-item PATCH).  This matches the localStorage pattern — each `set()` writes the entire JSON blob.  Tradeoff: larger payloads but simpler migration diff and no partial-state bugs.

### Tag definitions

User-defined tags auto-create `TagDefinition` rows.  New tags are assigned to an "Ungrouped" `CodebookGroup` (auto-created if needed by `_get_or_create_ungrouped()`).  The codebook UI can later reassign them to real groups.

### Hidden + Starred share QuoteState

Both `is_hidden` and `is_starred` live on the same `QuoteState` row (one per quote).  The PUT endpoints update only their respective field — they don't clobber each other.

## api-client.js

The JS-side HTTP abstraction (`bristlenose/theme/js/api-client.js`).  Loaded second in `_JS_FILES` (after `storage.js`, before everything else).

- **`isServeMode()`** — `typeof BRISTLENOSE_API_BASE === 'string'`
- **`apiGet(path)`** — fetch GET, returns JSON or null on error
- **`apiPut(path, data)`** — fetch PUT with JSON body, shows toast on error via `showToast()` (optional dependency from `csv-export.js`)

All calls are **fire-and-forget promises**.  The JS modules don't `await` them.

## Health API contract (footer config)

`GET /api/health` returns:

- `status` (unchanged)
- `version` (unchanged)
- `links.github_issues_url`
- `feedback.enabled`
- `feedback.url`

This payload is shared by serve mode and export mode (embedded in `BRISTLENOSE_EXPORT.health`).

Default values:

- `links.github_issues_url` = `https://github.com/cassiocassio/bristlenose/issues/new`
- `feedback.enabled` = `true`
- `feedback.url` = `https://cassiocassio.co.uk/feedback.php`

Optional env overrides:

- `BRISTLENOSE_GITHUB_ISSUES_URL`
- `BRISTLENOSE_FEEDBACK_ENABLED` (`1/true/yes/on` truthy; everything else false)
- `BRISTLENOSE_FEEDBACK_URL`

## Patterns to follow

### Adding a new data endpoint

1. Add the route to `routes/data.py` — follow `get_hidden`/`put_hidden` as the simplest template
2. Use `_get_db(request)`, `_check_project()`, `try/finally db.close()`
3. For quote-keyed data: use `_resolve_quote()` and `_quote_dom_id()` to translate between DOM IDs and DB rows
4. Add tests in `tests/test_serve_data_api.py` — the `client` fixture gives you a TestClient with the smoke-test data pre-imported (4 quotes: `q-p1-10`, `q-p1-26`, `q-p1-46`, `q-p1-66`)
5. If the JS needs to call it: add `if (isServeMode()) apiPut(...)` after the corresponding `store.set()` call

### Adding a new JS module that syncs state

1. Add `isServeMode, apiPut` to the `/* global ... */` comment
2. After every `yourStore.set(data)` call, add: `if (isServeMode()) apiPut('/your-endpoint', data);`
3. That's it.  `apiPut` prepends `BRISTLENOSE_API_BASE` internally — **never** add the base URL at the call site (double-prefix bug)

### DB dependency injection

```python
def _get_db(request: Request) -> Session:
    return request.app.state.db_factory()
```

The factory is stored on `app.state` during `create_app()`.  Each route creates its own session and **must close it** in a `finally` block.

## React island integration

React islands replace Jinja2 content at serve time via **marker-based substitution** — `<!-- bn-{name} -->` comments in `render/report.py` become `<div id="bn-{name}-root">` at serve time (regex replacement in `app.py:serve_report_html()`). Browser mounts React into the div via `main.tsx`.

**When adding a new island:** register it in the renderer overlay CSS (4 places) so it shows green (React) not blue (Jinja2) under the dev-only overlay.

See `docs/design-react-islands.md` for the 6-step "Adding a new island" checklist, current islands table, and renderer overlay details.

## Dev loop (live JS reload)

**Vanilla JS is stripped in serve mode (Step 8).** `_strip_vanilla_js()` in `app.py` uses the `_JS_MARKER` boundary to remove all concatenated module code from the IIFE, keeping only `window.*` globals (`BRISTLENOSE_VIDEO_MAP`, `BRISTLENOSE_PLAYER_URL`, `BRISTLENOSE_ANALYSIS`) that React reads.  This runs in both dev and prod paths via `_transform_report_html()`.  The `_JS_FILES` list in `render/theme_assets.py` stays for `bristlenose render` (offline HTML).

**CSS changes are live in dev mode.** `_mount_dev_report()` registers a `/report/assets/bristlenose-theme.css` route that calls `_load_default_css()` on every request — no caching, no render step. Edit `bristlenose/theme/*.css` and refresh. This route is defined before the catch-all, so it takes priority over `FileResponse(output_dir / path)`.

**What still requires re-render:** changes to the Jinja2 HTML template itself, changes to data variables (quote map, analysis data). CSS no longer requires re-render in dev mode.

**Python changes** are handled by uvicorn's `--reload` (WatchFiles) — editing `data.py`, `app.py`, etc. triggers an automatic server restart.

## Names architecture (YAML canonical, DB materialized)

`people.yaml` is the single source of truth for participant names. The SQLite `Person` table is a materialized view, populated from YAML on import and kept in sync via write-through.

### Import path (YAML → DB)

`_import_speakers()` in `importer.py` reads `people.yaml` via `_load_people_for_import()` and populates `Person.full_name`, `short_name`, `role_title`, `persona`, `notes` from matching entries. On re-import (server restart), `_update_persons_from_people()` fills empty Person fields from YAML without overwriting non-empty values — so browser edits (via `PUT /people`) always win over YAML.

### Write-through path (DB → YAML)

`PUT /people` in `routes/data.py` writes to DB first (for UI responsiveness), then calls `_write_through_people_yaml()` which atomically updates `people.yaml` (tempfile + rename). This means pipeline re-runs see browser edits. Write-through is best-effort — if it fails, a warning is logged but the API still succeeds.

### Lifecycle

```
Pipeline run → people.yaml (names from LLM/labels/manual edit)
                    ↓
bristlenose serve → importer reads YAML → Person rows in DB
                    ↓
Browser edit → PUT /people → DB (immediate) + YAML (atomic write-through)
                    ↓
Pipeline re-run → reads YAML → sees browser edits → preserves them
```

### International names (`_extract_given_name` in people.py)

`suggest_short_names()` uses `_extract_given_name()` which handles:
- **CJK ideographic** (田中由紀) — uses full name as short name (no splitting)
- **Honorific prefixes** (Dr., Prof.) — strips via `_strip_honorific()` before taking first token
- **Family-first names** (Tanaka Yuki, Park Ji-hyun) — checks first token against `_FAMILY_FIRST_SURNAMES` (337 entries), takes second token if matched
- Everything else — first token heuristic (works for Western, Spanish double surname, Russian patronymic, Arabic, mononym)

Known false positive: westernized order "Wei Zhang" (where "Wei" is also in `_FAMILY_FIRST_SURNAMES`) flips incorrectly. Acceptable — researcher corrects via inline edit.

## Multi-project scope rules

The server currently loads one project (project ID 1). Multi-project is future work (`docs/design-multi-project.md`), but the data model is already designed for it. **These rules prevent us painting ourselves into corners:**

- **Instance-scoped tables** (no `project_id`): `Person`, `CodebookGroup`, `TagDefinition`. These are shared across projects by design. **Never add `project_id` to these tables** — cross-project identity and codebook reuse depend on them being global
- **Project-scoped tables** (have `project_id` FK): `Quote`, `Session`, `ScreenCluster`, `ThemeGroup`, and all researcher-state tables. **Every new analysis/state table must include `project_id`**
- **Frontend project ID**: read from `data-project-id` attribute via `useProjectId()` hook or `apiBase()` helper. **Never hardcode `/api/projects/1/...` in new code** — use `apiBase()` which reads the injected `BRISTLENOSE_API_BASE` global. Existing hardcoded locations are tracked in the design doc
- **Person rows are created per-import, not deduped** — two "John Smith" rows from different projects is correct. Merging is a future human-driven action via `person_links` table, not an automatic process
- **`SessionSpeaker`** joins Person↔Session (with speaker code + role). It has no `project_id` because it inherits project scope through Session. This is correct — don't add `project_id` to it
- **Per-project SQLite files** — each project gets its own DB at `<output_dir>/.bristlenose/bristlenose.db`. Cross-project data (person links, app settings) will live in the instance DB at `~/.config/bristlenose/bristlenose.db`. Don't store cross-project relationships in per-project DBs

## Middleware stack

Three middleware layers in `create_app()`, added in this order (Starlette processes in reverse — last added wraps outermost):

1. **BearerTokenMiddleware** — auth on `/api/*` and `/media/*`
2. **CORSMiddleware** — blocks all cross-origin requests
3. **GZipMiddleware** (`minimum_size=500`) — compresses text responses (JSON, HTML, CSS, JS). ~83% reduction on the quotes endpoint. Media files opt out via `Content-Encoding: identity` on the `FileResponse` to avoid wasting CPU on incompressible codecs and breaking byte-range video seeking. BREACH is not a concern — no user input is reflected in token-bearing responses

## Localhost access control

`middleware.py` — `BearerTokenMiddleware` validates `Authorization: Bearer <token>` on `/api/*` and `/media/*` routes. Token generated per server instance in `create_app()` via `secrets.token_urlsafe(32)`, stored on `app.state.auth_token`. Injected into SPA HTML via `json.dumps()`, printed to stdout for desktop app. Exempt: `/api/health`, `/api/docs`, `/report/*`, `/static/*`. Tests use `AuthTestClient` (from `tests/conftest.py`) which auto-injects the auth header. Design doc: `docs/design-localhost-auth.md`.

## Gotchas

- **Alembic: `create_all()` + `stamp("head")` skips data migrations on fresh DBs** — `init_db()` calls `create_all()` (which creates the latest schema from models.py) then stamps at head. This means future migrations that do data transforms (not just DDL) must guard against "fresh DB, nothing to transform." The `002_person_uuid` migration (S3) must handle both paths: upgrade from 001 (transform existing int→UUID) and fresh DB (Person.id is already UUID from models.py, nothing to do)
- **`/media/` must NOT require bearer token auth** — HTML `<video>` and `<audio>` elements fetch their `src` URL without custom headers. If `/media/` is in `_AUTH_REQUIRED_PREFIXES`, media loads get 401'd and the player shows "Cannot play this format" (`MediaError code: 4`). This is NOT a codec issue — it's an auth issue. **Diagnostic**: check Network tab for 401 on the `/media/...` URL BEFORE investigating codecs. `/media/` is protected by path-traversal guard, extension allowlist, localhost binding, and CORS instead. This has been misdiagnosed as a format problem three times — don't add `/media/` back to the auth prefixes
- **Quote timecode range match** — DOM ID uses `int(start_timecode)` which truncates.  A quote at 123.45s becomes `q-p1-123`.  The resolver queries `start_timecode >= 123 AND start_timecode < 124` to handle this
- **PUT replaces entire state** — PUT `/hidden` with `{}` unhides everything.  PUT `/tags` with `{}` deletes all tags.  This is intentional (mirrors localStorage `setItem`)
- **TagDefinition foreign key** — `codebook_group_id` is non-nullable.  New tags go to the "Ungrouped" group.  Don't use `codebook_group_id=0` — it violates the FK constraint
- **`Session` name collision** — SQLAlchemy's `Session` (the DB session) clashes with the `Session` model.  Import the model as `SessionModel`: `from bristlenose.server.models import Session as SessionModel`
- **`db.close()` in finally** — every route must close its session.  Leaking sessions exhausts the SQLite connection pool
- **Project ID is always 1** — the current server only loads one project at startup.  `BRISTLENOSE_API_BASE` is hardcoded to `/api/projects/1`.  Multi-project support is future work
- **Offline graceful degradation** — static HTML files (opened from disk, not served) have no `BRISTLENOSE_API_BASE` global.  `isServeMode()` returns false, all API calls skip silently, localStorage works as before
- **No `await` on API calls** — the JS modules fire `apiPut()` without awaiting.  This means rapid clicks send multiple overlapping PUTs.  The server handles this safely because each PUT replaces the full state (last writer wins)
- **Always use `apiGet()`/`apiPost()` from `frontend/src/utils/api.ts`** — never raw `fetch()` for `/api/*` endpoints.  The auth middleware requires a bearer token on every request.  The `api.ts` helpers add the `Authorization` header automatically from `window.__BRISTLENOSE_AUTH_TOKEN__`.  Raw `fetch()` gets 401'd silently (this broke the TOC sidebar in v0.14.1)
- **Ruff ignores JS files** — `ruff check .` skips `.js` files.  Don't pass JS files to ruff directly (it tries to parse them as Python and generates hundreds of false errors)
- **IIFE wrapper around the script block** — `render/report.py` wraps data variables + concatenated JS in `(function() { ... })();`.  In serve mode, `_strip_vanilla_js()` removes the module code but preserves the IIFE structure (globals + closing `})();`).  In the static render path, the full IIFE is intact.  If you break the IIFE closing, every init function silently fails — no console error, just a dead page
- **Baked HTML requires re-render to pick up new JS** — the static HTML report on disk contains a snapshot of the JS from the last `bristlenose render`.  In production (no `--dev`), editing `.js` source files has no effect until you re-render.  The live reload only works in `--dev` mode.  If something works in dev but not in the static file, the static file is stale
- **`pip install -e .` and render are separate steps** — an editable install makes Python changes visible immediately, but the rendered HTML is a separate artifact.  Adding a new JS file to `_JS_FILES` requires both `pip install -e .` (so Python sees the new list) and `bristlenose render` (so the static HTML includes it).  In dev mode, only a server restart is needed (uvicorn picks up the Python change, live reload reads the new file)
- **Importer reads people.yaml** — `_import_speakers()` requires `output_dir` parameter to find `people.yaml`. On re-import, `_update_persons_from_people()` only fills empty Person fields — never overwrites non-empty values. This means browser edits survive server restarts
- **Write-through is best-effort** — `_write_through_people_yaml()` in `routes/data.py` logs a warning on failure but doesn't fail the API request. The DB is always updated; YAML update is secondary
- **Codebook tab stale counts on initial load** — the CodebookPanel re-fetches via MutationObserver when its parent `.bn-tab-panel` gains `.active`.  This covers the race where vanilla JS `PUT /tags` hasn't finished when the panel first mounts.  Will be unnecessary once tag writes move from localStorage PUT to React CRUD
- **Source file paths lose subdirectory in transcript headers** — the pipeline writes `# Source: filename.mov` (just `.path.name`, no subdirectory) into transcript headers (`merge_transcript.py`, `render_output.py`). The importer reads this and resolves the file against `project_dir`. If the media file lives in a subdirectory (e.g. `interviews/`), the direct path doesn't exist. `_import_source_files()` handles this by scanning one level of subdirectories under `project_dir` — mirroring `ingest.discover_files()`. If video playback returns 404 in serve mode: (1) check `Session.source_file` in the DB — if it's missing the subdirectory, delete `bristlenose.db` and restart to force re-import; (2) check the server is running from the correct directory (worktree). The video-map API (`GET /api/projects/{id}/video-map`) converts DB paths to `/media/` URIs via `_file_to_media_uri()` in `sessions.py`. `PlayerContext.tsx` fetches this in SPA mode (no IIFE globals exist because the SPA HTML is generated directly, not from baked HTML)

## Tests

48 tests in `tests/test_serve_quotes_api.py` (response shape, grouping, field values, researcher state round-trips via data PUT endpoints, speaker name resolution, error handling).  37 happy-path tests in `tests/test_serve_data_api.py`, 57 stress tests in `tests/test_serve_data_api_stress.py`, 43 dashboard tests in `tests/test_serve_dashboard_api.py`, 36 codebook CRUD tests in `tests/test_serve_codebook_api.py`.  All use the smoke-test fixture at `tests/fixtures/smoke-test/input/` which provides:
- 1 project, 1 session (`s1`), 2 speakers (`m1`, `p1`)
- 4 quotes: `q-p1-10` (Dashboard/confusion), `q-p1-26` (Dashboard/frustration), `q-p1-46` (Search/delight), `q-p1-66` (Onboarding/frustration)

Stress tests cover: Unicode (emoji, CJK, RTL, combining chars), large payloads (10 KB edits, 50 tags), rapid sequential PUTs (fast clicking), empty/whitespace values, malformed DOM IDs, duplicate tags and badges (unique constraint dedup), cross-endpoint interactions, and idempotency

### Future: Playwright E2E tests (post-React migration)

After the React migration, Playwright E2E tests will cover the full browser → JS → API → DB path. Currently deferred because E2E tests target DOM selectors which all change during migration. The React components will emit `data-testid` attributes from day one to provide stable selectors. See `docs/design-reactive-ui.md` "Testing strategy" section for the full plan.

## React migration status

Migration complete through M5 (Feb 2026): 14 primitives + 2 infrastructure, 8 islands, 330+ Python serve tests. Next islands: transcript page, analysis page. Toast is the only unbuilt infrastructure primitive (build when first consumer needs it). See `docs/design-react-migration-status.md` for the primitive inventory (16 rows), island inventory (8 rows), and CSS alignment summary.

## AutoCode (LLM-assisted tag application)

`autocode.py` engine + `routes/autocode.py` (7 endpoints) + two ORM tables (`AutoCodeJob`, `ProposedTag`). First feature to call LLMs from serve mode — uses `asyncio.create_task()`, no Celery/Redis. Cloud-only (prompt weight ~14K-17K tokens doesn't fit Ollama's 4K context). Unique constraint `(project_id, framework_id)` — one job per codebook per project.

**Gotcha:** `LLMClient(settings)` takes only settings, not an LLMUsageTracker — creates its own internally.

See `docs/design-autocode.md` for the 7-step workflow, API endpoints, and testing notes.

## Reference docs

- **Domain model rationale**: `docs/design-serve-milestone-1.md`
- **Migration architecture**: `docs/design-serve-migration.md`
- **React component library** (16 primitives, build sequence, coverage matrix): `docs/design-react-component-library.md` — **read this before building any React component**
- **Codebook island design** (audit, decisions, CSS cleanup): `docs/design-codebook-island.md`
- **React migration plan**: `docs/design-reactive-ui.md`
- **Phase-by-phase walkthrough**: `docs/react-migration-walkthrough.md`
