diff --git a/thoughts/shared/designs/2026-03-30-fix-missing-trajectories-design.md b/thoughts/shared/designs/2026-03-30-fix-missing-trajectories-design.md new file mode 100644 index 0000000..7b9260a --- /dev/null +++ b/thoughts/shared/designs/2026-03-30-fix-missing-trajectories-design.md @@ -0,0 +1,102 @@ +--- +date: 2026-03-30 +topic: "fix-missing-trajectories" +status: draft +--- + +## Problem Statement + +We're seeing empty/absent party trajectories in the Explorer "Partij Trajectories" tab despite compute_2d_axes producing windows and many parties having centroids. The UI shows no visible traces for selected parties in some runs, making the feature unreliable for end users. + +**Root hypothesis:** either (A) selected parties have only missing/None centroid values at plot time, (B) a runtime exception (eg float(None)) aborts trace creation silently, or (C) label/party normalization mismatch filters out traces. We need a low-risk, diagnostic-first fix to reveal which of these is happening and restore visible traces quickly. + +## Constraints + +- Preserve public function names and locations: **compute_2d_axes, classify_axes, load_positions, _build_party_axis_figure, build_trajectories_tab, build_compass_tab, _spline_smooth**. +- Avoid large refactors; prefer reversible, minimal changes that surface diagnostics. +- Do not expose internal modal tokens ("As 1"/"As 2") to end users; use axis_classifier.display_label_for_modal(...) or choose_trajectory_title() where appropriate. +- Visual traces should remain smoothed; hover must include raw centroid values for auditability. + +## Chosen Approach (what we'll implement) + +I'm choosing a **minimal triage-first approach**: add precise diagnostics and defensive conversions around plotting, so we either restore visible traces immediately or produce deterministic diagnostics that reveal the real data mismatch. + +**Why:** low risk, fastest feedback loop. This will either fix simple runtime errors (safe float conversion, exceptions while adding traces) or provide clear evidence that deeper normalization changes are required. + +**Key changes:** +- Add a small helper: **safe_float(x)** — converts numeric-like values to floats, maps None/NaN/invalid -> float('nan') without raising. +- In build_trajectories_tab/_build_party_axis_figure: + - Wrap per-party fig.add_trace(...) in try/except and log the exception with party id/name to the DEBUG expander instead of aborting the whole plot. + - Emit per-selected-party diagnostics into the existing DEBUG expander: number of raw centroids, counts of non-NaN coordinates, example first 5 raw xs/ys, and lengths per window. + - Replace direct float(...) casts on raw centroid values used in hover/customdata with safe_float. + - Ensure per-MP fallback plotting path still exists and can be forced via EXPLORER_FORCE_SHOW_TRAJECTORIES for diagnosis. +- Add unit tests for safe_float and targeted integration tests that assert traces are created when centroids contain NaNs and when party_map exists. + +## Alternatives Considered + +1) Full normalization sweep: align party centroids to global windows (fill missing with NaN) and accept parties with at least one non-NaN value. + - Pros: robust long-term fix, canonical data shape. + - Cons: larger change surface, higher risk, slower to validate in production data. + +2) Refactor plotting pipeline to use a normalized DataFrame (rows=windows, cols=parties) and build traces from that canonical shape. + - Pros: clearer data flow, easier testing. + - Cons: larger refactor, touches many modules. + +I considered both but rejected them for immediate work because we need quick deterministic diagnostics to determine if these larger efforts are warranted. + +## Architecture (high-level) + +**Inputs:** positions_by_window (from compute_2d_axes), party_map, selected_parties. + +**Flow:** +- compute_2d_axes -> positions_by_window +- load_positions / helpers -> party-centroid dicts keyed by party +- build_trajectories_tab calls _build_party_axis_figure to build per-party traces +- _build_party_axis_figure uses smoothing helpers (_spline_smooth) to produce visible traces and also builds hover customdata with raw centroid values (smoothed coords for the trace, raw values in customdata) + +**Intervention points:** build_trajectories_tab and _build_party_axis_figure (small helper additions and safe conversion), plus tests and diagnostic output in the DEBUG expander. + +## Components and Responsibilities + +- **safe_float helper:** convert inputs to float or return float('nan') safely. Centralized to avoid repeated float(None) errors. +- **Diagnostic emitter:** small utility used by build_trajectories_tab to format and write per-party diagnostic rows to the DEBUG expander. +- **Plotly trace wrapper:** per-party try/except around fig.add_trace that writes exception details to diagnostics instead of failing silently. +- **Unit + integration tests:** verify hover customdata creation, safe_float behaviour, trajectories rendered with partial centroids, and UI label mapping does not emit "As 1"/"As 2". + +## Data Flow (detailed) + +- compute_2d_axes produces windows (time labels) and canonical positions_by_window. +- load_positions consumes positions_by_window and returns a mapping party -> list of centroids (one per window) where centroids may contain None/NaN for missing windows. +- build_trajectories_tab selects parties and for each party calls _build_party_axis_figure which: + - extracts raw xs_raw, ys_raw arrays aligned to windows + - computes smoothed xs_plot, ys_plot via _spline_smooth + - builds Plotly trace using xs_plot/ys_plot for the line and includes xs_raw/ys_raw in customdata with safe_float conversion + - adds the trace inside a try/except and emits any exception + raw samples to debug + +## Error Handling + +- Use safe_float to prevent float(None) and similar runtime TypeErrors when building hover/customdata. +- Use per-party try/except to avoid a single-party failure blanking the whole chart; log the error and continue plotting other parties. +- Show structured diagnostics in the existing DEBUG expander with these fields: party name, windows_count, raw_centroid_count, non_nan_count, sample_raw_xs, sample_raw_ys, exception (if any). + +## Testing Strategy + +- Unit tests: + - safe_float: None -> nan, '1.23' -> 1.23 (if strings are expected), invalid -> nan + - UI label helpers: axis_classifier.display_label_for_modal(...) and choose_trajectory_title() do not return raw "As 1"/"As 2" + +- Integration tests (lightweight): + - Build a synthetic positions_by_window with some None / NaN holes and assert _build_party_axis_figure returns a Plotly trace object (or equivalent structure) and that customdata contains numeric/NaN values not exceptions. + - Test that build_trajectories_tab's DEBUG expander receives the expected diagnostic entries for a party with missing centroids. + +- Manual verification steps (later): run full Streamlit with duckdb/plotly installed and open Explorer -> Trajectories to confirm traces are visible for typical parties and inspect the DEBUG expander. + +## Open Questions + +- Are there other UI locations still exposing raw modal labels? We should sweep the repo and tests already added help with this, but it may not be exhaustive. +- Do we want safe_float to try to coerce numeric strings? My proposal is **no coercion** (only pass-through numeric types and map others -> nan) unless tests show string encodings exist in centroid data. +- If diagnostics show that many parties are missing centroids entirely, we'll need the full normalization sweep (alternative #1). + +--- + +I'm proceeding to create the design doc. Interrupt if you want changes.