refactor: extract data loading and trajectory logic from explorer.py
- Move trajectory analysis to analysis/trajectory.py (+136 lines) - Move projection helpers to analysis/projections.py (+128 lines) - Extract tab-specific data loaders to analysis/tabs/ (8 modules, +133 lines) - Remove 702 lines from explorer.py (data loading extracted to analysis/explorer_data.py and new modules) - Add axis label fallback tests (tests/test_axis_label_fallback.py) - Add session docs: brainstorms, ideation, plans, and test-failures
This commit is contained in:
@@ -0,0 +1,118 @@
|
||||
---
|
||||
date: 2026-04-04
|
||||
topic: explorer-refactor
|
||||
---
|
||||
|
||||
# Explorer.py Refactor: Extract to analysis/
|
||||
|
||||
## Problem Frame
|
||||
|
||||
explorer.py is 3715 lines with 39 functions mixing:
|
||||
- Data loading (DuckDB queries)
|
||||
- Business logic (SVD projections, trajectory alignment)
|
||||
- UI rendering (Streamlit components)
|
||||
|
||||
This makes the file:
|
||||
- Hard to navigate (no clear boundaries)
|
||||
- Hard to test (requires Streamlit + DuckDB)
|
||||
- Hard to review (changes affect everything)
|
||||
|
||||
**Goal**: Improve navigability by extracting computation-heavy logic to `analysis/`, leaving explorer.py as a UI orchestration layer.
|
||||
|
||||
## Requirements
|
||||
|
||||
### Data Layer
|
||||
|
||||
- **R1.1**: Create `analysis/explorer_data.py` containing all data loading functions currently in explorer.py:
|
||||
- `get_available_windows()`
|
||||
- `get_uniform_dim_windows()`
|
||||
- `load_positions()`
|
||||
- `load_party_map()`
|
||||
- `load_active_mps()`
|
||||
- `load_party_axis_scores()`
|
||||
- `load_party_scores_all_windows()`
|
||||
- `load_party_scores_all_windows_aligned()`
|
||||
- `load_party_mp_vectors()`
|
||||
- `load_scree_data()`
|
||||
- `load_motions_df()`
|
||||
|
||||
- **R1.2**: All extracted functions must be callable without Streamlit imports (no `@st.cache_data`, no `st.*` calls)
|
||||
|
||||
- **R1.3**: Functions return pure Python data structures (DataFrames, dicts, lists) - no Plotly figures
|
||||
|
||||
### Business Logic Layer
|
||||
|
||||
- **R2.1**: Move computation functions to `analysis/` modules based on domain:
|
||||
- `_should_swap_axes()`, `_swap_axes()` → `analysis/axis_utils.py` (new)
|
||||
- `compute_party_discipline()` → `analysis/trajectories.py`
|
||||
- Trajectory computation functions → `analysis/trajectories.py`
|
||||
- SVD projection functions → `analysis/svd_labels.py` or new `analysis/projections.py`
|
||||
|
||||
- **R2.2**: Computations must be pure functions (no IO, deterministic outputs)
|
||||
|
||||
### UI Layer (explorer.py)
|
||||
|
||||
- **R3.1**: explorer.py becomes a thin orchestration layer:
|
||||
- Imports from `analysis/explorer_data.py` for data
|
||||
- Imports from `analysis/` modules for computations
|
||||
- Contains only Streamlit UI code and `@st.cache_data` wrappers
|
||||
|
||||
- **R3.2**: Render functions (`_render_*`) stay in explorer.py (they're UI-only)
|
||||
|
||||
- **R3.3**: Tab-building functions (`build_*_tab()`) stay in explorer.py but delegate to imported functions
|
||||
|
||||
### Import Safety
|
||||
|
||||
- **R4.1**: New `analysis/` modules must not import from `explorer.py` (no circular dependencies)
|
||||
|
||||
- **R4.2**: `analysis/explorer_data.py` may import from `database.py` (already exists)
|
||||
|
||||
### Testing
|
||||
|
||||
- **R5.1**: Extracted data functions should be testable with mocked DuckDB connections
|
||||
|
||||
- **R5.2**: Extracted computation functions should be pure and testable without database
|
||||
|
||||
## Success Criteria
|
||||
|
||||
- explorer.py reduced to under 1500 lines (from 3715)
|
||||
- No function in explorer.py exceeds 100 lines
|
||||
- Clear module boundaries: data → computation → UI
|
||||
- All extracted functions have docstrings with type hints
|
||||
- No circular imports between `analysis/` and `explorer/`
|
||||
|
||||
## Scope Boundaries
|
||||
|
||||
**Included:**
|
||||
- Data loading functions
|
||||
- Computation/transformation logic
|
||||
- Clear separation of concerns
|
||||
|
||||
**Excluded:**
|
||||
- UI rendering functions (they can stay in explorer.py)
|
||||
- Database schema changes
|
||||
- New features or behavior changes
|
||||
- Test suite updates (handled separately)
|
||||
|
||||
## Key Decisions
|
||||
|
||||
- **Domain-based splitting**: Computation goes to relevant `analysis/` module, not all to one file
|
||||
- **Import direction**: `explorer.py` imports from `analysis/`, never vice versa
|
||||
- **Preserve function signatures**: Refactoring shouldn't change public APIs
|
||||
|
||||
## Dependencies / Assumptions
|
||||
|
||||
- `database.py` provides `MotionDatabase` singleton - data functions will use this
|
||||
- `explorer_helpers.py` pattern is already established - follow its conventions
|
||||
- Streamlit caching (`@st.cache_data`) stays in explorer.py as the orchestration layer
|
||||
|
||||
## Outstanding Questions
|
||||
|
||||
### Deferred to Planning
|
||||
- [ ] [Implementation] Should `_load_mp_vectors_by_party()` and variants be merged or kept separate?
|
||||
- [ ] [Implementation] Should we create `analysis/projections.py` or extend existing `analysis/axis_classifier.py`?
|
||||
- [ ] [Implementation] How to handle the `_cached_bootstrap_cis()` function - move to analysis or keep as cache wrapper?
|
||||
|
||||
## Next Steps
|
||||
|
||||
→ `/ce:plan` for structured implementation planning
|
||||
@@ -0,0 +1,77 @@
|
||||
---
|
||||
date: 2026-04-05
|
||||
topic: right-wing-party-axis-validation
|
||||
---
|
||||
|
||||
# Right-Wing Party Axis Validation
|
||||
|
||||
## Problem Frame
|
||||
|
||||
The project convention states that PVV, FVD, JA21, and SGP must appear on the RIGHT side of all axes in visualizations (AGENTS.md). This is the #1 documented convention with zero automated enforcement. A single test prevents regression when SVD labels change or new components are added.
|
||||
|
||||
## Requirements
|
||||
|
||||
**R1. Canonical party sets defined once, imported everywhere**
|
||||
- Define `CANONICAL_RIGHT = frozenset({"PVV", "FVD", "JA21", "SGP"})` in `analysis/config.py`
|
||||
- Define `CANONICAL_LEFT = frozenset({"SP", "PvdA", "GL", "GroenLinks", "GroenLinks-PvdA", "DENK", "PvdD", "Volt"})` in `analysis/config.py` — matches svd_labels.py LEFT_PARTIES exactly
|
||||
- All code that checks political orientation (svd_labels.py, political_axis.py) imports from config instead of defining inline
|
||||
|
||||
**R2. Validation test loads real data from DuckDB**
|
||||
- Test file: `tests/test_axis_political_orientation.py`
|
||||
- Uses existing data loading functions (`load_party_scores_all_windows_aligned` from `analysis/explorer_data.py`)
|
||||
- No synthetic data — validates against actual `party_axis_scores` table
|
||||
|
||||
**R3. 2D political compass orientation check (statistical, not per-party)**
|
||||
- `party_axis_scores` table has `x_axis_aligned` (component 1) and `y_axis_aligned` (component 2)
|
||||
- For each window, validate both axes using **mean scores**:
|
||||
- **Axis 1 (x)**: Compute mean of `CANONICAL_RIGHT` x-values and mean of `CANONICAL_LEFT` x-values. Assert `right_mean > left_mean`
|
||||
- **Axis 2 (y)**: Same for y-values. Assert `right_mean > left_mean`
|
||||
- "Right on right" means the **average** right party is right of the **average** left party — individual parties may deviate slightly (e.g., one right party slightly negative is fine)
|
||||
- `compute_flip_direction` already implements this logic (compares group means) — use it
|
||||
- Skips parties not present in a given window (graceful, not a failure)
|
||||
|
||||
**R4. `compute_flip_direction` consistency check**
|
||||
- After loading data, call `compute_flip_direction(1, party_scores)` and `compute_flip_direction(2, party_scores)` per window
|
||||
- Assert both return `False` (no flip needed) when data is already correctly oriented
|
||||
- If either returns `True`, the data violates the convention and the test fails with a clear message
|
||||
|
||||
**R5. Clear failure messages**
|
||||
- When orientation check fails, report: window, axis (x/y), right_mean, left_mean, difference
|
||||
- Example: `"Window '2021-2023', x-axis: right_mean=-0.12, left_mean=0.08 (right parties on LEFT side — flip direction=True)"`
|
||||
|
||||
## Success Criteria
|
||||
|
||||
- Test runs as part of `pytest` suite (`.venv/bin/python -m pytest tests/test_axis_political_orientation.py`)
|
||||
- Test passes with current data (convention currently holds — this establishes the baseline)
|
||||
- If convention is violated in future data, test fails with actionable message
|
||||
- Test works for all windows in the database (not just current)
|
||||
- Statistical check (mean-based) — test passes even if individual parties deviate slightly from group mean
|
||||
|
||||
## Scope Boundaries
|
||||
|
||||
- **Not included**: Testing unaligned scores (only aligned scores are validated — these are what users see)
|
||||
- **Not included**: VVD, NSC, BBB, CDA, ChristenUnie — these are center parties, not right-wing per AGENTS.md convention
|
||||
- **Not included**: Per-party strict sign checks (statistical mean check is sufficient and more robust)
|
||||
- **Not included**: Updating `political_axis.py` — R1 only updates `svd_labels.py` to import from config; `political_axis.py` uses different party sets for PCA centroid orientation and is out of scope
|
||||
|
||||
## Key Decisions
|
||||
|
||||
- **Canonical sets match AGENTS.md for right, svd_labels.py for left**: `CANONICAL_RIGHT = {PVV, FVD, JA21, SGP}` matches AGENTS.md exactly. `CANONICAL_LEFT = {SP, PvdA, GL, GroenLinks, GroenLinks-PvdA, DENK, PvdD, Volt}` matches svd_labels.py LEFT_PARTIES exactly.
|
||||
- **Single unified source of truth in config.py**: `CANONICAL_RIGHT` and `CANONICAL_LEFT` frozensets go in `config.py` — it's a prerequisite for the test to work correctly. Only `svd_labels.py` is updated to import from config; `political_axis.py` is out of scope (uses party sets for PCA centroid orientation, not the same usage).
|
||||
- **Aligned scores only**: Unaligned scores may vary across windows due to Procrustes alignment drift; aligned scores are the stable, user-facing representation.
|
||||
- **Statistical (mean-based) validation, not per-party**: The orientation check compares group means, not individual party scores. A single right party being slightly negative is not a failure — the mean right score must exceed the mean left score.
|
||||
|
||||
## Dependencies / Assumptions
|
||||
|
||||
- DuckDB database is populated with `party_axis_scores` table with `x_axis_aligned` and `y_axis_aligned` columns (verified)
|
||||
- `analysis/explorer_data.py` functions work correctly (already tested)
|
||||
- `_PARTY_NORMALIZE` already exists in `config.py` (lines 247-256) — use it for party name alias normalization
|
||||
- `config.py` currently lacks `CANONICAL_RIGHT`/`CANONICAL_LEFT` frozensets — these must be added as part of R1
|
||||
- `compute_flip_direction()` in `svd_labels.py` currently uses inline `RIGHT_PARTIES`/`LEFT_PARTIES` — must be updated to import from config after R1
|
||||
|
||||
## Outstanding Questions
|
||||
|
||||
All resolved. Key decisions documented above.
|
||||
|
||||
## Next Steps
|
||||
→ `/ce:plan` for structured implementation planning
|
||||
Reference in New Issue
Block a user