# Fix missing trajectories Implementation Plan

I'm using the writing-plans skill to create the implementation plan.

Goal: Restore visible party trajectories in the Explorer "Partij Trajectories" tab by adding validation/inspection helpers, making centroid computation tolerant of missing windows (emit NaN gaps), and adding an automatic MP-level fallback (top-K) with a debug expander and hover raw-values preserved.

Design: thoughts/shared/designs/2026-03-30-fix-missing-trajectories-design.md

Architecture: Small, focused changes in explorer_helpers.py (pure helpers + unit tests) and explorer.py (UI wiring and plotting policy). Keep helper logic independent of Streamlit so tests run in CI without heavy deps. Provide a graceful MP fallback and compact diagnostics exposed behind a collapsed expander.

Tech Stack: Python 3.x, pytest, Streamlit (manual UI verification), Plotly (already used). Tests must run in CI with duckdb / streamlit optional — unit tests only use pure Python/numpy.

---

## Dependency Graph

```
Batch 1 (parallel): 1.1, 1.2 [foundation - no deps]
Batch 2 (parallel): 2.1, 2.2 [core - depends on batch 1]
Batch 3 (parallel): 3.1, 3.2 [integration - depends on batch 2]
```

---

## Decisions / gap-filling (explicit)
- EXPLORER_MP_FALLBACK_COUNT environment variable: integer, default 20. Used to choose top-K MPs when party centroids are absent.
- Top-K definition: by seat_count when available; when seat_count unavailable, fall back to party axis activity (mean magnitude) via load_party_axis_scores if needed. I will implement MP fallback using seat_count if present in mp_metadata; otherwise use party axis magnitude from load_party_axis_scores.
- Validation rules (inspect_positions_for_issues): detect empty positions_by_window, windows_count mismatch across MPs, sample of mismatched mp ids, parties_with_centroid_counts dictionary. Reason: these are the most likely causes of empty traces.
- compute_party_centroids behavior: returns per-party arrays aligned to windows (list of floats or np.nan), metadata per-party containing counts and missing indices. Guarantees empty lists (never None).

---

## Batch 1: Foundation (parallel - 2 implementers)

All tasks in this batch have NO dependencies and can run simultaneously.

### Task 1.1: Add inspector helper
**File:** `explorer_helpers.py`
**Test:** `tests/test_inspect_positions_for_issues.py`
**Depends:** none

Helpers to add (names only):
- inspect_positions_for_issues(positions_by_window: Dict[str, Dict[str, Tuple[float,float]]], party_map: Dict[str,str]) -> Dict[str, Any]

What it returns (documented in test expectations):
- windows_count: int
- window_labels: list[str] (sorted sample of window keys)
- mp_id_set: set[str] (set of entity ids seen across windows)
- party_map_count: int (len(party_map))
- parties_with_centroid_counts: Dict[str, int] (mapping party -> number of windows with a centroid)
- mismatched_mp_ids_sample: list[str] (sample of ids present in positions but not in party_map, up to 10)

Tests to add (exact assertions):
- tests/test_inspect_positions_for_issues.py (unit):
  - Construct synthetic positions_by_window with 3 windows, with some MPs missing in some windows and some mp ids that aren't in party_map. Assert returned windows_count == 3, party_map_count equals len(party_map), parties_with_centroid_counts entries for expected parties, and mismatched_mp_ids_sample contains the expected missing keys.

Verify:
- Run: `pytest tests/test_inspect_positions_for_issues.py -q`
- Expected: PASS

Commit message: `feat(explorer): add inspect_positions_for_issues helper + test`

### Task 1.2: Add compute_party_centroids (per-window aligned arrays)
**File:** `explorer_helpers.py` (same file; add new function)
**Test:** `tests/test_compute_party_centroids.py`
**Depends:** none

Helper to add (name only):
- compute_party_centroids(positions_by_window: Dict[str, Dict[str, Tuple[float,float]]], party_map: Dict[str,str], windows: List[str]) -> Tuple[Dict[str, List[float]], Dict[str, Any]]

Behavior contract (for implementer):
- Return party_centroids: dict[party -> list[float|np.nan]] aligned to the provided windows order. For a party and window where no MPs present, insert np.nan at that index.
- Return metadata: {"per_party_counts": {party: int}, "total_windows": int, "parties": sorted_list}
- Guarantees: never return None; party lists can be empty list but must have length == len(windows) for parties present in `parties` list.

Tests to add (exact assertions):
- tests/test_compute_party_centroids.py (unit):
  - Case A: full coverage — every party has coords in every window -> assert no np.nan and lengths equal windows count.
  - Case B: partial coverage -> assert np.nan present at expected indices and metadata.per_party_counts match counts.
  - Case C: no parties (empty positions_by_window) -> party_centroids == {} and metadata.total_windows == len(windows)

Verify:
- Run: `pytest tests/test_compute_party_centroids.py -q`
- Expected: PASS

Commit message: `feat(explorer): add compute_party_centroids to produce aligned per-party arrays`

---

## Batch 2: Core Modules (parallel - 2 implementers)
All tasks depend on Batch 1.

### Task 2.1: Modify explorer.py to use helpers and add MP fallback
**File:** `explorer.py` (modify function build_trajectories_tab only)
**Test:** `tests/test_build_trajectories_tab_fallback.py`
**Depends:** 1.1, 1.2

Changes to make (high-level, exact function to modify):
- modify build_trajectories_tab(db_path: str, window_size: str) to:
  - early: call inspect_positions_for_issues(positions_by_window, party_map) and render the compact DEBUG expander content (same keys as the inspector returns). Keep the expander collapsed by default.
  - replace existing per-window centroid construction with compute_party_centroids(...) which returns aligned arrays containing np.nan placeholders.
  - relax party-selection filtering: treat a party as plottable if it has >= 1 non-nan centroid (previous code required full coverage). This ensures partial traces still render with gaps.
  - preserve hover customdata to include raw centroid values (already present in code) — ensure when centroids contain np.nan for raw values we still populate customdata with (np.nan, np.nan).
  - If no party centroids (empty dict or all-party centroid vectors are entirely nan), trigger MP fallback: plot top-K MPs (EXPLORER_MP_FALLBACK_COUNT, default 20) as per design. This fallback must show a small banner message in Dutch: "Partijcentroiden niet beschikbaar — tonen individuele MP-trajecten als fallback." and provide a toggle (st.checkbox) to expand to show the full top-K list.

Notes / gap-filling decisions (explicit):
- EXPLORER_MP_FALLBACK_COUNT: implement read via int(os.getenv("EXPLORER_MP_FALLBACK_COUNT", "20"))
- For selecting top-K MPs: use seat_count if present in mp_metadata (query `mp_metadata` for a seat_count-like field). If unavailable, choose MPs with most non-empty positions across windows. Implementer decision: compute activity = number of windows with a valid (non-None) position and sort descending.

Tests to add (integration, shims-friendly):
- tests/test_build_trajectories_tab_fallback.py
  - Scenario 1 (party centroids present): Provide a fake positions_by_window and party_map fixture with at least one party having centroids in multiple windows and assert that when build_trajectories_tab is invoked (call the internal plotting branch with a test harness) it adds at least one trace (fig.data length > 0) and trace names match selected parties.
  - Scenario 2 (no party centroids): Provide positions_by_window where party_map is empty or all MPs map to Unknown; assert the MP fallback path is chosen (method returns or builds fig with MPs) and that the banner message string appears in returned metadata or printed UI stub. Since Streamlit is not easily invoked in unit tests, structure the UI branch so the plotting logic returns fig when called from tests — write the test to import a small internal helper (e.g., build_trajectories_figure_for_test) if necessary. If refactor needed, keep it minimal: extract plotting assembly to a private helper _assemble_trajectories_figure(...) that returns (fig, trace_count, banner_text) so tests can assert fig traces without needing Streamlit.

Verify (unit/integration):
- Run: `pytest tests/test_build_trajectories_tab_fallback.py -q`
- Expected: PASS

Commit message: `feat(explorer): use inspector & compute_party_centroids; add MP top-K fallback and debug expander`

### Task 2.2: Add/adjust unit tests for hover/raw values and NaN handling
**File:** `tests/test_explorer_helpers.py` (update) and `tests/test_explorer_chart.py` (add test)
**Depends:** 1.2

Changes/tests to add (exact tests):
- tests/test_explorer_helpers.py: add a test verifying compute_party_centroids produces np.nan for missing windows and that hover customdata creation uses (float, float) or (np.nan, np.nan) consistently.
- tests/test_explorer_chart.py: add a small unit test that constructs a go.Figure via the new plotting helper (see 2.1) and asserts:
  - traces exist when parties have partial coverage
  - customdata arrays length equals x/y arrays length
  - hovertemplate contains both smoothed and raw placeholder markers (strings like 'x (raw)')

Verify:
- Run: `pytest tests/test_explorer_helpers.py::test_compute_party_centroids_nan_handling -q`
- Run: `pytest tests/test_explorer_chart.py::test_partial_party_traces -q`
- Expected: PASS

Commit message: `test(explorer): add tests for NaN gaps and hover customdata preservation`

---

## Batch 3: Integration & Manual UI checks (parallel - 2 implementers)
Depends on Batch 2

### Task 3.1: Integration test (shim-friendly) for three scenarios
**File:** `tests/integration/test_trajectories_ui_integration.py`
**Test:** the file above
**Depends:** 2.1, 2.2

Tests to add (exact scenarios):
- Scenario A (full party centroids): positions_by_window with full coverage — assert plot built uses party traces; simulate user selection to include at least one party; assert fig.data length >= 1.
- Scenario B (party centroids missing): party_map empty — assert MP fallback chosen and number of plotted MP traces == EXPLORER_MP_FALLBACK_COUNT or the available MPs if fewer.
- Scenario C (partial centroids): party centroids partial across windows — assert traces exist and customdata shows np.nan at missing indices.

Test harness notes: tests should import small pure helpers from explorer.py that assemble figures without calling st.plotly_chart or other Streamlit side-effects. If necessary, add a small refactor in explorer.py: `_assemble_trajectory_figure_for_tests(positions_by_window, party_centroids, selected_parties, windows, smooth_alpha, ...) -> go.Figure, metadata` and call that from build_trajectories_tab. Tests then call this helper. Keep the helper private and minimal.

Verify:
- Run: `pytest tests/integration/test_trajectories_ui_integration.py -q`
- Expected: PASS

Commit message: `test(integration): trajectories UI integration scenarios (full/partial/missing)`

### Task 3.2: Manual Streamlit verification steps (documented)
**File:** none (manual steps below); include in PR description.
**Depends:** 2.1

Manual verification (Streamlit):
1. Start Streamlit: `streamlit run explorer.py --server.headless true` (or run locally with a test DB path)
2. Open the app in browser (usually http://localhost:8501). Go to tab "Partij Trajectories".
3. Scenario: normal DB with party centroids
   - Select a recent window_size (e.g., quarterly or annual as appropriate)
   - Ensure default parties (CDA, D66, VVD) appear and trajectories are visible.
   - Hover on a trace point: verify hover shows both smoothed and raw centroid values (x (smoothed), x (raw)).
   - Open the DEBUG expander (collapsed by default) and confirm it shows `windows (count)`, `windows sample`, `party_map entries`, `parties with centroids`, `sample centroid window counts per party`.
4. Scenario: simulate missing party centroids (set party_map to {} or use a DB snapshot with missing mp_metadata)
   - The app should show the fallback banner: "Partijcentroiden niet beschikbaar — tonen individuele MP-trajecten als fallback." and render MP trajectories (top-K). There should be a checkbox to expand the top-K list.
5. Scenario: partial centroids
   - For a party missing centroids in some windows, its trace should appear but with gaps (line discontinuity where NaNs present). Hover customdata at gap points should show raw value `nan` or a placeholder.

Streamlit-specific acceptance criteria:
- traces drawn when at least one party has >=1 centroid
- MP fallback automatically displayed (banner + plotted MP traces) when no party centroids
- DEBUG expander shows diagnostics described above
- Hover shows raw centroid values even when smoothing is applied

---

## Files to create / modify (one-file-per-task mapping)

Batch 1
- Modify: `explorer_helpers.py` — add functions:
  - inspect_positions_for_issues
  - compute_party_centroids
- Add test: `tests/test_inspect_positions_for_issues.py`
- Add test: `tests/test_compute_party_centroids.py`

Batch 2
- Modify: `explorer.py` — function build_trajectories_tab; optional small private helper `_assemble_trajectory_figure_for_tests` (single-file change)
- Add test: `tests/test_build_trajectories_tab_fallback.py`
- Update/add tests: `tests/test_explorer_helpers.py` (augment), `tests/test_explorer_chart.py`

Batch 3
- Add test: `tests/integration/test_trajectories_ui_integration.py`

---

## Verification commands (unit & CI)
- Unit test single file: `pytest tests/test_inspect_positions_for_issues.py -q`
- Unit test compute party centroids: `pytest tests/test_compute_party_centroids.py -q`
- Trajectories fallback unit tests: `pytest tests/test_build_trajectories_tab_fallback.py -q`
- Integration tests (shim-friendly): `pytest tests/integration/test_trajectories_ui_integration.py -q`
- Run full test suite: `pytest -q`

Manual Streamlit checks: follow steps in Task 3.2 above. Recommended quick dev workflow:
- Start streamlit: `streamlit run explorer.py --server.headless true`
- Use the URL printed in console (usually http://localhost:8501) and perform the manual steps.

---

## Blocked / Unblocked checklist

- [ ] Blocker: Access to a representative DB fixture (small DuckDB or JSON fixture) that contains windows, svd_vectors and mp_metadata. Without it, integration/manual checks are limited. (Mitigation: tests use synthetic positions_by_window and party_map fixtures — unblocked for unit tests.)
- [ ] Blocker: If MP seat_count is required from DB and not present in test fixtures, fallback selection will use activity-based ranking. (Mitigation: implement activity fallback.)
- [x] Unblocked: Adding pure helpers in explorer_helpers.py (unit tests cover behavior without Streamlit/duckdb)
- [x] Unblocked: Modifying build_trajectories_tab to call helpers and add banner + expander (code-local change)
- [ ] Optional: Agree on EXPLORER_MP_FALLBACK_COUNT envvar default (I set default 20). If you want a different default, tell me now.

If any of the above blockers remain, proceed with unit tests and open a PR discussion for integration DB fixtures.

---

## Estimated timeline (hours)

- Task 1.1 (inspect_positions_for_issues + unit test): 1.5 h
- Task 1.2 (compute_party_centroids + unit tests): 3.0 h
- Task 2.1 (explorer.py changes: wiring, MP fallback, debug expander): 4.0 h
- Task 2.2 (tests for hover/NaN handling): 2.0 h
- Task 3.1 (integration tests / small refactor helper): 2.5 h
- Task 3.2 (manual Streamlit QA and documentation): 1.5 h
- PR polish, CI tweaks, and addressing review comments: 2.0 h

Total: 16.5 hours (approx)

---

## PR checklist / deliverables
- [ ] Unit tests for inspector and centroids pass
- [ ] build_trajectories_tab updated with debug expander and fallback
- [ ] Integration tests for three scenarios pass (or documented reason for partial coverage)
- [ ] Manual Streamlit QA steps documented in PR and verified locally
- [ ] Add mention of EXPLORER_MP_FALLBACK_COUNT to README or environment docs (optional follow-up)

---

If you'd like, I can now (A) produce the concrete test contents and minimal helper implementations as separate micro-tasks (one file + one test per task) ready for implementers, or (B) proceed to create and apply the code changes in this repo. Which do you prefer?