---
date: 2026-03-31
topic: "diagnose-no-plot-trajectories"
status: draft
---

## Problem Statement

We need to restore visible party trajectories in the Explorer "Partij Trajectories" tab so the Plotly chart shows non-empty traces for realistic windows, and provide opt-in diagnostics that explain why traces are missing.

**Why:** Users see an empty chart in some environments/windows. This could be caused by upstream data gaps, malformed coordinates, strict filtering in helpers, or unhandled exceptions in the plotting helper. We must gather evidence, fix the actual cause, and avoid changing production behavior unless debug is explicitly enabled.


## Constraints

- Keep changes minimal and reversible; prefer instrumentation and small helper fixes over large refactors.
- Diagnostics must be opt-in (EXPLORER_DEBUG_TRAJECTORIES env var and UI checkbox).
- Helpers must be import-safe and pure so unit tests run without heavy GUI/DB dependencies.
- Use project's environment management (uv) for local runs and CI — do not call pip directly.


## Approach (chosen)

I recommend a **diagnostic-first** approach followed by targeted small fixes. Steps:

- Add a small, dedicated diagnostic writer script that emits a structured JSON diagnostics artifact for representative windows from data/motions.db.
- Improve input validation and normalization in load_positions / compute_2d_axes (coerce numeric strings, treat 'nan'/'None' consistently, ignore out-of-range coords) so helpers are robust to malformed rows.
- Keep current gates that avoid plotting when inputs are invalid, but record precise diagnostics into module-level _last_trajectories_diagnostics and the CLI JSON output.
- Add unit tests for the normalization logic and for inspector behaviors; add a small integration diagnostic test that runs via uv and checks trace_count > 0 for a known-good sample window.

Reasoning: we already have instrumentation capturing stages (load_positions_empty, no_mp_positions, select_helper_exception, trace_count). Gathering structured evidence will let us pick a minimal fix (data normalization or filter tweak) without risky behaviour changes.


## Alternatives considered

- Aggressive fallback rendering: render approximated centroids when traces are empty. Rejected because it may mask data quality issues and mislead users.
- Upstream data repair: fix svd pipeline / DB rows before Explorer. Good long-term, but requires cross-team coordination and longer cycle — we should diagnose first.


## Architecture

**High-level:** The Explorer plotting pipeline remains the same; we add a diagnostics writer and a small normalization layer.

- Data source: data/motions.db (svd_vectors and party maps)
- Pipeline: get_uniform_dim_windows -> compute_2d_axes -> load_positions -> inspect_positions_for_issues -> compute_party_centroids -> select_trajectory_plot_data -> Plotly fig
- Diagnostics: module-level _last_trajectories_diagnostics plus a CLI script that runs representative windows and writes JSON artifacts to thoughts/shared/diagnostics/YYYY-MM-DD-trajectories-diagnostics.json


## Components and responsibilities

- Diagnostic CLI (scripts/save_trajectories_diagnostics.py):
  - Run a configurable sample of windows, call compute_2d_axes, load_positions, inspect_positions_for_issues, select_trajectory_plot_data.
  - Emit structured JSON with per-window diagnostics and aggregated summary.

- Normalization helpers (explorer_helpers.normalize_positions):
  - Coerce numeric strings to floats, coerce common null tokens to NaN, clamp improbable values, and return a normalized positions_by_window structure.
  - Pure, import-safe, and covered by unit tests.

- Instrumentation (explorer._last_trajectories_diagnostics):
  - Record stage, window id, counts (n_windows, n_entities per window), mp_positions_count, any helper exceptions/tracebacks, and sample rows.

- UI changes (pages/2_Explorer.py):
  - Add an opt-in debug checkbox that enables detailed diagnostics in the UI when checked (or when EXPLORER_DEBUG_TRAJECTORIES=1).
  - Do not change default plotting or filtering behavior when debug is disabled.

- Tests
  - Unit tests for normalization and inspector.
  - Diagnostic integration test run via uv (non-flaky, uses a small sample or DB fixture).


## Data Flow

1. Caller requests trajectories tab (build_trajectories_tab).
2. call get_uniform_dim_windows(DB) -> returns window descriptors.
3. For each sampled window, compute_2d_axes(window) -> returns raw positions_by_window (possibly malformed).
4. normalize_positions(positions_by_window) -> cleaned positions_by_window.
5. inspect_positions_for_issues(positions_by_window) -> returns diagnostics (missing coords, string values, NaNs, zero-length paths).
6. compute_party_centroids(positions_by_window) -> party centroids and mp_positions.
7. select_trajectory_plot_data(centroids, mp_positions, options) -> returns fig, trace_count, banner_text. On exception capture diagnostics.
8. If trace_count == 0 -> do not call st.plotly_chart; show friendly message and, if debug enabled, show the collected diagnostics and link to the saved JSON artifact.


## Error Handling

- Capture exceptions at helper boundaries and record to select_trajectory_plot_data._last_diagnostics and module _last_trajectories_diagnostics. Do not raise to Streamlit UI unless debug is enabled.
- Normalize inputs proactively to reduce exception surface (avoid type errors from strings/None).
- If a helper raises, return a safe empty fig and banner that suggests enabling diagnostics.
- JSON diagnostics writer writes atomically (write to a .tmp file then rename) to avoid partial files being consumed.


## Testing Strategy

- Unit tests (fast, import-safe):
  - normalize_positions handles strings, 'nan', None, and clamps extremes.
  - inspect_positions_for_issues detects empty windows, NaNs-only windows, and malformed coordinate types.
  - select_trajectory_plot_data returns (fig, trace_count>0) for a known-good small sample and sets diagnostics correctly when trace_count==0.

- Integration tests (run under uv in CI or locally):
  - Diagnostic CLI can be executed via uv run and creates a JSON diagnostic artifact for a small sample; test asserts artifact exists and is valid JSON with expected fields.

- Manual verification:
  - Run EXPLORER_DEBUG_TRAJECTORIES=1 uv run python scripts/save_trajectories_diagnostics.py --db data/motions.db --out thoughts/shared/diagnostics/<date>.json
  - Open the Explorer locally and reproduce an empty-chart scenario; enable debug checkbox and view diagnostics.


## Open Questions

1. Do we prefer automatic normalization (silently fixing data) or conservative behavior (report and require upstream fix)? My recommendation: auto-normalize common, unambiguous issues (strings -> numbers, common null tokens) and surface anything ambiguous in diagnostics.
2. Where should diagnostic artifacts live long-term? thoughts/shared/diagnostics is fine for short-term; consider a single diagnostics/ bucket for CI artifacts.
3. Which windows should the diagnostics CLI sample by default? I propose sampling: 1) first 10 windows, 2) 10 windows evenly spaced, and 3) one window that previously produced empty result if known.


I'm proceeding to create the design doc. Interrupt if you want changes.