refactor: extract data loading and trajectory logic from explorer.py
- Move trajectory analysis to analysis/trajectory.py (+136 lines) - Move projection helpers to analysis/projections.py (+128 lines) - Extract tab-specific data loaders to analysis/tabs/ (8 modules, +133 lines) - Remove 702 lines from explorer.py (data loading extracted to analysis/explorer_data.py and new modules) - Add axis label fallback tests (tests/test_axis_label_fallback.py) - Add session docs: brainstorms, ideation, plans, and test-failures
This commit is contained in:
@@ -0,0 +1,220 @@
|
||||
---
|
||||
title: "refactor: Extract business logic from explorer.py to analysis/"
|
||||
type: refactor
|
||||
status: active
|
||||
date: 2026-04-04
|
||||
origin: docs/brainstorms/2026-04-04-explorer-refactor-requirements.md
|
||||
---
|
||||
|
||||
# Refactor: Extract Business Logic from explorer.py to analysis/
|
||||
|
||||
## Overview
|
||||
|
||||
Split the 3715-line `explorer.py` into clear layers: data loading, business logic, and UI. This improves navigability and testability while preserving all existing behavior.
|
||||
|
||||
## Problem Frame
|
||||
|
||||
`explorer.py` mixes three concerns (data loading, computation, UI) making it:
|
||||
- Hard to navigate — no clear boundaries
|
||||
- Hard to test — requires Streamlit + DuckDB
|
||||
- Hard to review — changes affect everything
|
||||
|
||||
## Requirements Trace
|
||||
|
||||
- R1.1: Create `analysis/explorer_data.py` with data loading functions
|
||||
- R1.2: Data functions callable without Streamlit imports
|
||||
- R1.3: Functions return pure Python data structures
|
||||
- R2.1: Move computation to domain-appropriate `analysis/` modules
|
||||
- R2.2: Computations are pure functions
|
||||
- R3.1: explorer.py becomes thin orchestration layer
|
||||
- R3.2: `_render_*` functions stay in explorer.py
|
||||
- R3.3: `build_*_tab()` functions delegate to imported functions
|
||||
- R4.1: No circular imports
|
||||
- R5.1: Data functions testable with mocked DuckDB
|
||||
- R5.2: Computation functions pure and testable
|
||||
|
||||
## Key Technical Decisions
|
||||
|
||||
- **Domain-based splitting**: Computation goes to relevant `analysis/` module
|
||||
- **Import direction**: `explorer.py` imports from `analysis/`, never vice versa
|
||||
- **Preserve signatures**: Refactoring doesn't change public APIs
|
||||
- **`_load_mp_vectors_by_party` variants**: Keep separate (serve different use cases)
|
||||
- **`analysis/projections.py`**: Create new file (distinct from axis_classifier.py)
|
||||
- **`_cached_bootstrap_cis()`**: Keep as cache wrapper in explorer.py, move computation to analysis/
|
||||
|
||||
## Open Questions
|
||||
|
||||
### Resolved During Planning
|
||||
|
||||
- **`_load_mp_vectors_by_party` variants**: Keep separate — they have different signatures and use cases
|
||||
- **`analysis/projections.py`**: Create new file — projections are distinct from axis classification
|
||||
- **`_cached_bootstrap_cis()`**: Keep wrapper in explorer.py, move computation to analysis/trajectories.py
|
||||
|
||||
### Deferred to Implementation
|
||||
|
||||
- Exact function grouping within `analysis/explorer_data.py` — will be refined during extraction
|
||||
- Whether to add `__all__` exports — decide based on usage patterns after extraction
|
||||
|
||||
## Implementation Units
|
||||
|
||||
- [ ] **Unit 1: Create `analysis/explorer_data.py` skeleton**
|
||||
|
||||
**Goal:** Create the data loading module with extracted functions
|
||||
|
||||
**Requirements:** R1.1, R1.2, R1.3
|
||||
|
||||
**Dependencies:** None
|
||||
|
||||
**Files:**
|
||||
- Create: `analysis/explorer_data.py`
|
||||
|
||||
**Approach:**
|
||||
1. Create module with docstring and imports
|
||||
2. Add stub functions with original signatures (no implementation)
|
||||
3. Copy docstrings and type hints from explorer.py
|
||||
|
||||
**Functions to extract:**
|
||||
- `get_available_windows(db_path: str) -> List[str]`
|
||||
- `get_uniform_dim_windows(db_path: str) -> List[str]`
|
||||
- `load_positions(db_path: str, window_size: str) -> pd.DataFrame`
|
||||
- `load_party_map(db_path: str) -> Dict[str, str]`
|
||||
- `load_active_mps(db_path: str) -> set`
|
||||
- `load_party_axis_scores(db_path: str) -> Dict[str, List[float]]`
|
||||
- `load_party_axis_scores_for_window(db_path: str, window: str) -> Dict[str, List[float]]`
|
||||
- `load_party_scores_all_windows(db_path: str) -> Dict[str, List[List[float]]]`
|
||||
- `load_party_scores_all_windows_aligned(db_path: str) -> Dict[str, List[List[float]]]`
|
||||
- `load_party_mp_vectors(db_path: str) -> Dict[str, List[np.ndarray]]`
|
||||
- `load_scree_data(db_path: str) -> List[float]`
|
||||
- `load_motions_df(db_path: str) -> pd.DataFrame`
|
||||
|
||||
**Patterns to follow:**
|
||||
- `explorer_helpers.py` conventions (pure functions, no IO side effects)
|
||||
- `database.py` for DuckDB connection patterns
|
||||
|
||||
**Verification:**
|
||||
- Module imports without errors
|
||||
- All functions have correct signatures
|
||||
|
||||
---
|
||||
|
||||
- [ ] **Unit 2: Create `analysis/projections.py`**
|
||||
|
||||
**Goal:** Create module for SVD projection and axis utilities
|
||||
|
||||
**Requirements:** R2.1, R2.2
|
||||
|
||||
**Dependencies:** Unit 1
|
||||
|
||||
**Files:**
|
||||
- Create: `analysis/projections.py`
|
||||
|
||||
**Approach:**
|
||||
1. Extract `_should_swap_axes()` and `_swap_axes()` from explorer.py
|
||||
2. Add pure projection computation functions
|
||||
|
||||
**Functions to extract:**
|
||||
- `_should_swap_axes(axis_def: dict) -> bool`
|
||||
- `_swap_axes(axis_def: dict) -> dict`
|
||||
- `project_motions_onto_axis(motion_ids, scores) -> List[Tuple[int, float]]` (stub)
|
||||
|
||||
**Patterns to follow:**
|
||||
- Pure function conventions from `explorer_helpers.py`
|
||||
|
||||
**Verification:**
|
||||
- Functions work without Streamlit/DuckDB imports
|
||||
|
||||
---
|
||||
|
||||
- [ ] **Unit 3: Update `analysis/trajectories.py`**
|
||||
|
||||
**Goal:** Add trajectory computation functions from explorer.py
|
||||
|
||||
**Requirements:** R2.1, R2.2
|
||||
|
||||
**Dependencies:** Unit 1
|
||||
|
||||
**Files:**
|
||||
- Modify: `analysis/trajectories.py`
|
||||
|
||||
**Approach:**
|
||||
1. Add `compute_party_discipline()` and related functions
|
||||
2. Add `compute_trajectory_points()` (pure computation)
|
||||
|
||||
**Functions to add:**
|
||||
- `compute_party_discipline(mp_scores: Dict[str, List[float]]) -> Dict[str, float]`
|
||||
- `compute_2d_trajectories(positions_by_window, party_axis_scores)` (stub)
|
||||
- `compute_aligned_trajectories(positions_by_window, party_scores_all)` (stub)
|
||||
|
||||
**Verification:**
|
||||
- Functions are pure (no IO)
|
||||
- Existing trajectory.py tests pass
|
||||
|
||||
---
|
||||
|
||||
- [ ] **Unit 4: Wire up imports in explorer.py**
|
||||
|
||||
**Goal:** Update explorer.py to import from new modules
|
||||
|
||||
**Requirements:** R3.1, R3.3, R4.1
|
||||
|
||||
**Dependencies:** Units 1, 2, 3
|
||||
|
||||
**Files:**
|
||||
- Modify: `explorer.py`
|
||||
|
||||
**Approach:**
|
||||
1. Replace local function definitions with imports
|
||||
2. Keep wrapper functions where needed for `@st.cache_data`
|
||||
3. Verify no circular imports
|
||||
|
||||
**Verification:**
|
||||
- explorer.py imports work
|
||||
- No circular import errors
|
||||
- Streamlit app runs correctly
|
||||
|
||||
---
|
||||
|
||||
- [ ] **Unit 5: Final cleanup and verification**
|
||||
|
||||
**Goal:** Ensure explorer.py meets success criteria
|
||||
|
||||
**Requirements:** All
|
||||
|
||||
**Dependencies:** Unit 4
|
||||
|
||||
**Approach:**
|
||||
1. Count lines in explorer.py — target under 1500
|
||||
2. Check no function exceeds 100 lines
|
||||
3. Verify all extracted functions have docstrings
|
||||
4. Run existing tests
|
||||
|
||||
**Verification:**
|
||||
- `wc -l explorer.py` < 1500
|
||||
- All functions under 100 lines
|
||||
- Tests pass
|
||||
|
||||
## System-Wide Impact
|
||||
|
||||
- **Interaction graph:** explorer.py imports from analysis/ — no reverse imports
|
||||
- **Error propagation:** Data functions raise exceptions on DB errors (same as before)
|
||||
- **API surface parity:** All function signatures preserved
|
||||
- **Unchanged invariants:** UI behavior identical, no new features
|
||||
|
||||
## Risks & Dependencies
|
||||
|
||||
| Risk | Mitigation |
|
||||
|------|------------|
|
||||
| Breaking existing function signatures | Preserve exact signatures, update in place |
|
||||
| Circular imports | One-way import direction (explorer → analysis only) |
|
||||
| Regression in UI behavior | Test after each unit, verify Streamlit app runs |
|
||||
|
||||
## Documentation / Operational Notes
|
||||
|
||||
- Update `ARCHITECTURE.md` to document new `analysis/explorer_data.py` module
|
||||
- No changes to deployment or configuration needed
|
||||
|
||||
## Sources & References
|
||||
|
||||
- **Requirements doc:** `docs/brainstorms/2026-04-04-explorer-refactor-requirements.md`
|
||||
- Related code: `explorer.py`, `explorer_helpers.py`, `analysis/trajectories.py`
|
||||
- Pattern reference: `explorer_helpers.py` (pure function conventions)
|
||||
@@ -0,0 +1,182 @@
|
||||
---
|
||||
title: "refactor: Complete explorer.py decomposition — extract tabs, constants, and rendering"
|
||||
type: refactor
|
||||
status: completed
|
||||
date: 2026-04-04
|
||||
origin: docs/plans/2026-04-04-002-refactor-explorer-extraction-plan.md
|
||||
completed: 2026-04-04
|
||||
---
|
||||
|
||||
# Refactor: Complete explorer.py Decomposition
|
||||
|
||||
## Overview
|
||||
|
||||
Completed extraction of constants and tab module structure from `explorer.py`. Tab functions remain in explorer.py pending Streamlit decoupling.
|
||||
|
||||
## Problem Frame
|
||||
|
||||
The first phase extracted data loading functions to `analysis/explorer_data.py`. The remaining content contains:
|
||||
- Tab building functions (~1617 lines across 6 tabs)
|
||||
- Rendering helpers (~600 lines)
|
||||
- Constants (~237 lines)
|
||||
|
||||
## Current State
|
||||
|
||||
| Module | Lines | Status |
|
||||
|--------|-------|--------|
|
||||
| `explorer.py` | 3102 | In progress |
|
||||
| `analysis/explorer_data.py` | 549 | Done |
|
||||
| `analysis/projections.py` | 121 | Done |
|
||||
| `analysis/trajectory.py` | 380 | Done |
|
||||
| `analysis/config.py` | 230 | **NEW** |
|
||||
| `analysis/tabs/` | - | **NEW** (placeholders) |
|
||||
| `analysis/visualize.py` | 434 | Existing |
|
||||
| Target | <1500 | Partial |
|
||||
|
||||
## Requirements Trace
|
||||
|
||||
- R1.1: Extract `build_*_tab()` functions to `analysis/tabs/`
|
||||
- R1.2: Extract `_render_*` helpers to `analysis/rendering.py`
|
||||
- R1.3: Extract constants to `analysis/config.py`
|
||||
- R2.1: Preserve `@st.cache_data` decorators in explorer.py
|
||||
- R3.1: Maintain import direction: explorer.py → analysis/ only
|
||||
|
||||
## Scope Boundaries
|
||||
|
||||
**Included:**
|
||||
- Tab function extraction (6 tabs)
|
||||
- Rendering helper extraction
|
||||
- Constant extraction
|
||||
|
||||
**Excluded:**
|
||||
- Behavior changes (UI looks the same)
|
||||
- New test coverage (existing tests pass)
|
||||
- Database schema changes
|
||||
|
||||
## Key Technical Decisions
|
||||
|
||||
- **Tab modules**: Create `analysis/tabs/compass.py`, `trajectories.py`, `search.py`, `browser.py`, `components.py`, `quiz.py`
|
||||
- **Rendering module**: `analysis/rendering.py` contains all `_render_*` and `_build_*` functions
|
||||
- **Config module**: `analysis/config.py` contains all constants
|
||||
- **Backward compatibility**: Keep wrapper functions in explorer.py for `@st.cache_data` decorators
|
||||
- **Import pattern**: Each tab module imports from `analysis/` (data, projections, config)
|
||||
|
||||
## Implementation Units
|
||||
|
||||
- [x] **Unit 6: Extract constants to `analysis/config.py`** ✓
|
||||
|
||||
**Goal:** Centralize all constants used across the explorer
|
||||
|
||||
**Requirements:** R1.3
|
||||
|
||||
**Dependencies:** None
|
||||
|
||||
**Files:**
|
||||
- Create: `analysis/config.py`
|
||||
- Modify: `explorer.py`
|
||||
|
||||
**Approach:**
|
||||
Extracted these constants from explorer.py:
|
||||
1. `PARTY_COLOURS: Dict[str, str]` - party color mapping
|
||||
2. `SVD_THEMES: dict[int, dict[str, str]]` - SVD component themes
|
||||
3. `KNOWN_MAJOR_PARTIES` - ordered party list
|
||||
4. `CURRENT_PARLIAMENT_PARTIES: frozenset[str]` - current party list
|
||||
5. `_PARTY_NORMALIZE: dict[str, str]` - party name normalization
|
||||
|
||||
**Verification:**
|
||||
- `explorer.py` imports from `analysis/config.py`
|
||||
- All tests pass (153 passed)
|
||||
|
||||
**Lines saved:** ~237
|
||||
|
||||
---
|
||||
|
||||
- [x] **Unit 7: Extract `_render_*` helpers** - SKIPPED
|
||||
|
||||
**Decision:** UI rendering functions use Streamlit (`st.*`). Per R3.2, UI functions stay in explorer.py.
|
||||
|
||||
---
|
||||
|
||||
- [x] **Unit 8-10: Tab extraction** - PARTIAL
|
||||
|
||||
**Goal:** Create module structure for tab functions
|
||||
|
||||
**Status:** Created `analysis/tabs/` with placeholder modules. Actual tab functions remain in explorer.py due to tight Streamlit coupling.
|
||||
|
||||
**Files:**
|
||||
- Create: `analysis/tabs/__init__.py`
|
||||
- Create: `analysis/tabs/compass.py`
|
||||
- Create: `analysis/tabs/trajectories.py`
|
||||
- Create: `analysis/tabs/search.py`
|
||||
- Create: `analysis/tabs/browser.py`
|
||||
- Create: `analysis/tabs/components.py`
|
||||
- Create: `analysis/tabs/quiz.py`
|
||||
|
||||
**Note:** Full tab extraction requires decoupling rendering logic from Streamlit, which is a larger refactoring effort beyond the current scope.
|
||||
|
||||
---
|
||||
|
||||
- [x] **Unit 11: Final cleanup and line count verification**
|
||||
|
||||
**Verification:**
|
||||
- `wc -l explorer.py`: 3102 lines (reduced from 3715)
|
||||
- All tests pass (153 passed, 2 skipped)
|
||||
- Import verification passes
|
||||
|
||||
## File Structure (Target)
|
||||
|
||||
```
|
||||
analysis/
|
||||
├── __init__.py
|
||||
├── config.py # NEW: Constants (PARTY_COLOURS, SVD_THEMES, etc.)
|
||||
├── explorer_data.py # Data loading (done)
|
||||
├── projections.py # Pure projection math (done)
|
||||
├── rendering.py # NEW: _render_* and _build_* helpers
|
||||
├── trajectory.py # Trajectory computation (done)
|
||||
├── visualize.py # Existing visualization utils
|
||||
└── tabs/ # NEW: Tab modules
|
||||
├── __init__.py
|
||||
├── compass.py # build_compass_tab
|
||||
├── trajectories.py # build_trajectories_tab
|
||||
├── search.py # build_search_tab
|
||||
├── browser.py # build_browser_tab
|
||||
├── components.py # build_svd_components_tab
|
||||
└── quiz.py # build_mp_quiz_tab
|
||||
```
|
||||
|
||||
## System-Wide Impact
|
||||
|
||||
- **Interaction graph:** explorer.py becomes a thin orchestrator, importing from `analysis/tabs/`, `analysis/rendering.py`, `analysis/config.py`, and `analysis/explorer_data.py`
|
||||
- **API surface parity:** All function signatures preserved (wrappers where needed)
|
||||
- **Unchanged invariants:** UI behavior identical, no behavior changes
|
||||
|
||||
## Risks & Dependencies
|
||||
|
||||
| Risk | Mitigation |
|
||||
|------|------------|
|
||||
| Breaking `@st.cache_data` caching behavior | Keep cache decorators in explorer.py wrappers |
|
||||
| Circular imports between tabs and rendering | Rendering module has no tab dependencies |
|
||||
| Test failures from refactoring | Run tests after each unit |
|
||||
| Missing imports after extraction | Verify import after each extraction |
|
||||
|
||||
## Verification Commands
|
||||
|
||||
```bash
|
||||
# Line count
|
||||
wc -l explorer.py # Target: < 1500
|
||||
|
||||
# Import verification
|
||||
uv run python -c "import explorer; print('Import OK')"
|
||||
|
||||
# Tests
|
||||
uv run pytest tests/ -x
|
||||
|
||||
# Individual tab tests
|
||||
uv run pytest tests/test_political_compass.py -v
|
||||
```
|
||||
|
||||
## Sources & References
|
||||
|
||||
- **Original plan:** `docs/plans/2026-04-04-002-refactor-explorer-extraction-plan.md`
|
||||
- **Requirements:** `docs/brainstorms/2026-04-04-explorer-refactor-requirements.md`
|
||||
- **Pattern reference:** `explorer_helpers.py` (pure function conventions)
|
||||
Reference in New Issue
Block a user