refactor: extract data loading and trajectory logic from explorer.py

- Move trajectory analysis to analysis/trajectory.py (+136 lines)
- Move projection helpers to analysis/projections.py (+128 lines)
- Extract tab-specific data loaders to analysis/tabs/ (8 modules, +133 lines)
- Remove 702 lines from explorer.py (data loading extracted to
  analysis/explorer_data.py and new modules)
- Add axis label fallback tests (tests/test_axis_label_fallback.py)
- Add session docs: brainstorms, ideation, plans, and test-failures
This commit is contained in:
2026-04-05 00:51:30 +02:00
parent 154762a4c8
commit 414c16ae9e
19 changed files with 1595 additions and 699 deletions
@@ -0,0 +1,149 @@
---
date: 2026-04-04
topic: code-quality-architecture-ideation
focus: code quality and architecture improvements
---
# Ideation: Code Quality & Architecture Improvements
## Codebase Context
- **explorer.py**: 3715 lines — monolithic Streamlit app with 65+ `except Exception:` handlers
- **database.py**: 1366 lines — `MotionDatabase` class with similar exception patterns
- **explorer_helpers.py**: 317 lines — pure functions, import-safe, well-testable (the pattern)
- **Anti-patterns**: 208 instances of bare/broad exception handling, nested try-except blocks
- **Tests**: Well-organized in `tests/` with good coverage of helpers
## Ranked Ideas
### 1. Systematic Exception Handler Audit & Refactor
**Description:** Audit all 208 `except Exception:` blocks across the codebase. Categorize by failure mode (missing dependency, data validation, network, IO) and replace with specific exceptions. Add error context propagation.
**Rationale:** The current pattern silently swallows errors, making debugging impossible. Refactoring to specific exceptions enables proper error handling, logging, and user feedback. This compounds: each fix reduces 2-3 nested exception handlers.
**Downsides:** High volume of changes requires careful regression testing.
**Confidence:** 90%
**Complexity:** High
**Status:** Unexplored
---
### 2. Extract Business Logic from explorer.py into Pure Functions
**Description:** Identify and extract computation-heavy sections from the 3715-line explorer.py. Move to pure functions in a new module (e.g., `explorer_logic.py`), keeping Streamlit UI glue in the main file.
**Rationale:** explorer.py mixes UI code with business logic, making it untestable and hard to reason about. The existing `explorer_helpers.py` proves this pattern works — same approach applied more broadly enables unit testing of core algorithms.
**Downsides:** Requires careful interface design to avoid breaking the Streamlit page.
**Confidence:** 85%
**Complexity:** Medium
**Status:** Unexplored
---
### 3. Create Typed Data Transfer Objects (DTOs) for Database Layer
**Description:** Replace dictionary-based data passing between `database.py` and consumers with typed dataclasses or Pydantic models. Define `MotionDTO`, `PartyResultDTO`, `SessionDTO`.
**Rationale:** 208 exception handlers often mask type mismatches that would be caught at compile-time with typed DTOs. The `src/validators/types.py` shows existing type awareness — extend this systematically to the data layer.
**Downsides:** Migration effort; some duckdb results may not serialize cleanly.
**Confidence:** 75%
**Complexity:** Medium
**Status:** Unexplored
---
### 4. Establish Explicit Error Recovery Strategies
**Description:** Rather than catch-all exception handling, implement explicit recovery strategies per failure mode: retry with backoff for transient failures, fallback to cached data for missing dependencies, graceful degradation for optional features.
**Rationale:** The anti-pattern exists because there's no systematic recovery approach. Explicit strategies replace 208 silent catches with intentional behavior — this is the "compound leverage" angle.
**Downsides:** Requires identifying which failures are transient vs. permanent per operation.
**Confidence:** 80%
**Complexity:** Medium
**Status:** Unexplored
---
### 5. Modularize database.py into Focused Modules
**Description:** Split `database.py` (1366 lines) into: `db_connection.py` (connection lifecycle), `db_motions.py` (motion queries), `db_sessions.py` (session management), `db_migrations.py` (schema updates).
**Rationale:** Single-responsibility violation — database.py handles connection, schema, queries, and migrations. Splitting enables independent testing and clearer ownership. The `pipeline/` modular structure shows this is already the project's convention.
**Downsides:** Breaking changes for any existing imports.
**Confidence:** 70%
**Complexity:** Medium
**Status:** Unexplored
---
### 6. Add Comprehensive Type Hints to Core Modules
**Description:** Run mypy on `explorer.py`, `database.py`, `analysis/*.py`. Fix missing type hints and enable strict type checking in CI.
**Rationale:** Type hints catch the errors that 208 exception handlers are currently masking. The `src/types/motion_types.py` shows the project already has some type investment — this extends it to the pain points.
**Downsides:** May require `cast()` in some duckdb interop scenarios.
**Confidence:** 85%
**Complexity:** Low
**Status:** Unexplored
---
### 7. Create Code Climate Metrics & Monitoring
**Description:** Add radon or lizard to measure cyclomatic complexity per module. Set thresholds that fail CI if exceeded. Track over time.
**Rationale:** Quantitative baseline for refactoring impact. Currently no way to measure if the 3715-line explorer.py is improving or degrading. Compounds: each refactor can be measured.
**Downsides:** Tool overhead; thresholds may need tuning.
**Confidence:** 60%
**Complexity:** Low
**Status:** Unexplored
---
### 8. Extract Static Analysis Rule for Bare Except Detection
**Description:** Add a flake8 plugin or ruff rule that flags `except:` and `except Exception:` without re-raising or logging. Document the project-specific exception hierarchy.
**Rationale:** Prevents the anti-pattern from re-entering. The project has 208 violations — a custom lint rule catches new violations and encodes the team's error-handling philosophy. This is the "assumption-breaking" angle: stop fixing cases, fix the system.
**Downsides:** Requires defining what specific exceptions ARE allowed per context.
**Confidence:** 70%
**Complexity:** Low
**Status:** Unexplored
---
## Rejection Summary
| # | Idea | Reason Rejected |
|---|------|-----------------|
| 1 | Add docstrings to all functions | Too obvious; not leverage-focused |
| 2 | Migrate to async database operations | Premature optimization; duckdb is sync |
| 3 | Add logging library (structured logging) | Tool-focused, not addressing root cause |
| 4 | Replace Streamlit with another framework | Out of scope for this codebase |
| 5 | Add Caching layer for database queries | Already exists via Streamlit caching; not addressing architecture |
## Session Log
- 2026-04-04: Initial ideation — 13 generated, 8 survived
@@ -0,0 +1,160 @@
---
date: 2026-04-04
topic: reliability-correctness-improvements
focus: reliability and correctness
---
# Ideation: Reliability & Correctness Improvements
## Codebase Context
- **Python + Streamlit + DuckDB** data pipeline application
- **Key Issues from docs/solutions/**:
- SVD labels must reflect voting patterns, not semantic content (850+ SVD component labels in code)
- Bare exception handlers: 850+ `except Exception:` across codebase
- Nested exception handling creates opaque error paths
- Error handling catches broad Exception and prints to stdout (179 `print()` statements in error paths)
- **Existing Pattern**: `explorer_helpers.py` is pure functions, testable, well-structured — the model to follow
## Grounding Evidence
1. `docs/solutions/best-practices/svd-labels-voting-patterns-not-semantics.md` documents the SVD labeling convention
2. Grep search found 281 `except Exception:` in `.py` files plus bare `except:` handlers
3. `database.py` line 47: bare `except:` that catches everything including KeyboardInterrupt
4. 179 print statements in error handling paths hide issues from logging
## Ranked Ideas
### 1. Right-Wing Party Axis Validation — Automated Assert
**Description:** Add runtime validation that PVV, FVD, JA21, SGP appear on RIGHT side of all SVD/PCA axes. Create a `validate_axis_polrity()` function that checks party loadings and raises `AssertionError` if right-wing parties appear on the left.
**Rationale:** This is the most impactful correctness fix — the project convention is explicitly documented in AGENTS.md yet has no automated enforcement. A single validation pass catches SVD labeling errors before they reach production.
**Downsides:** Requires careful handling of axis flips (sometimes flipping is the correct fix, not validation failure).
**Confidence:** 95%
**Complexity:** Low
**Status:** Unexplored
---
### 2. Type-Safe Vote Normalization with Exhaustiveness Checking
**Description:** Replace the fragile string-based vote normalization in `database.py` (lines 715-744) with a typed enum + exhaustiveness checking. Add a `Vote` enum with variants: `VOOR`, `TEGEN`, `ONTHOUDEN`, `AFWEZIG`. Use match/case with `case _` to catch unmapped values at development time.
**Rationale:** The current normalization silently returns `None` for unknown vote values — this causes data loss that only manifests as "agreement percentage is wrong". Typed enums with exhaustiveness checking prevent silent data loss.
**Downsides:** Requires updating all call sites that pass vote strings.
**Confidence:** 90%
**Complexity:** Medium
**Status:** Unexplored
---
### 3. DuckDB Connection Leak Detector — Context Manager Audit
**Description:** Audit all `duckdb.connect()` calls for proper context manager usage or explicit `.close()`. Many handlers catch exceptions but forget to close connections. Add a `ConnectionTracker` that warns on unclosed connections in development.
**Rationale:** Connection leaks accumulate and eventually exhaust database connections. The codebase has 15+ places where exceptions cause early returns without connection cleanup.
**Downsides:** Tracking adds overhead; some leaks are already handled by DuckDB's connection pooling.
**Confidence:** 85%
**Complexity:** Medium
**Status:** Unexplored
---
### 4. Replace Print-Based Debugging with Structured Logging
**Description:** Replace the 179 `print()` statements in error paths with structured logging using the existing `_logger`. Create a script that automates this conversion for common patterns.
**Rationale:** Print statements go to stdout and are discarded in production. Proper logging enables log aggregation, alerting, and debugging of production issues.
**Downsides:** High volume of changes; risk of losing context in some print statements.
**Confidence:** 80%
**Complexity:** Medium
**Status:** Unexplored
---
### 5. SVD Component Label Verification — Pre-Deployment Assertion
**Description:** Create a CI/CD pre-deployment script that verifies SVD labels against actual voting data — checking that labels match the voting pattern, not semantic assumptions. Query which parties vote positive/negative per component and validate label accuracy.
**Rationale:** The SVD label documentation exists but there's no enforcement. This automated check prevents the documented mistake (semantic labels that don't match voting) from recurring.
**Downsides:** Requires understanding of the SVD pipeline and periodic re-calibration as voting data changes.
**Confidence:** 75%
**Complexity:** Medium
**Status:** Unexplored
---
### 6. Nested Exception Handler Flattening — EAFP to LBYL Migration
**Description:** Replace nested try-except blocks with explicit preconditions (LBYL — Look Before You Leap). Many handlers wrap every operation in `try-except` because they don't trust the data. Add validation functions that check preconditions before operations.
**Rationale:** Nested exception handlers make the control flow impossible to reason about. Replacing with explicit validation makes code more readable and debuggable.
**Downsides:** Requires understanding what conditions each operation actually needs.
**Confidence:** 70%
**Complexity:** High
**Status:** Unexplored
---
### 7. Database Schema Validation — Foreign Key and Constraint Checks
**Description:** Add startup validation that checks the actual database schema against expected schema. Verify table existence, column types, and foreign key relationships. Fail fast with clear error messages if schema is stale.
**Rationale:** The current code tries to add columns with `ALTER TABLE ... IF NOT EXISTS` which can fail silently. A schema validation pass catches migration failures immediately.
**Downsides:** Schema changes require updating validation code.
**Confidence:** 85%
**Complexity:** Low
**Status:** Unexplored
---
### 8. Motion Data Sanitization Pipeline — Pre-Insert Validation
**Description:** Add a sanitization layer for incoming motion data that validates:
- `winning_margin` is between 0 and 1
- `policy_area` is non-empty
- `voting_results` keys match known parties
- Date parsing succeeds for motion dates
**Rationale:** The current insertion code trusts upstream data. Invalid data causes hard-to-debug issues downstream in SVD computation and similarity calculations.
**Downsides:** Requires defining what "valid" means for each field.
**Confidence:** 80%
**Complexity:** Medium
**Status:** Unexplored
---
## Rejection Summary
| # | Idea | Reason Rejected |
|---|------|-----------------|
| 1 | Add unit tests for exception paths | Good idea but lower leverage than preventing errors at source; covered by existing test infrastructure |
| 2 | Refactor all 850+ exception handlers in one pass | Too high volume — needs phased approach captured by idea #1 |
| 3 | Add type hints to all functions | Good hygiene but doesn't directly address reliability — covered by existing typing effort |
| 4 | Implement circuit breaker for external API calls | No external API calls observed in core codebase |
## Session Log
- 2026-04-04: Initial ideation — 8 generated, 8 survived
@@ -0,0 +1,149 @@
---
date: 2026-04-04
topic: stemwijzer-improvement-ideas
focus: general
---
# Ideation: Stemwijzer Improvement Ideas
## Codebase Context
**Project shape:** Python/Streamlit Dutch voting advice tool ("Stemwijzer")
- Uses uv for package management, pytest for testing, DuckDB for data
- Key modules: analysis/, pipeline/, database.py (50KB), explorer.py (143KB)
- Notable: 3 venvs (.venv, .venv_axis, .venv_plotly) suggest dependency experimentation
- AGENTS.md exists with conventions (right-wing parties on RIGHT side, SVD labels reflect voting patterns)
**Pain points identified:**
- explorer.py is 143KB monolith - hard to navigate
- SVD labels must reflect voting patterns (documented as learning)
- 850+ bare exception handlers documented as anti-pattern
- No CONTRIBUTING.md for onboarding
**Leverage points:**
- Good test organization (tests/ with subdirs)
- Documented solutions in docs/solutions/
- explorer_helpers.py proves pure-function pattern works
## Ranked Ideas
### 1. Right-Wing Party Axis Validation
**Description:** Add an automated test that asserts PVV, FVD, JA21, SGP appear on the RIGHT side (positive loading) of all SVD/PCA axes.
**Rationale:** This is the #1 project convention (from AGENTS.md) with zero automated enforcement. The documented SVD label bug showed how easy it is to get this wrong. A simple test prevents regression.
**Downsides:** Requires defining "RIGHT side" for each component - some components may have flipped poles.
**Confidence:** 95%
**Complexity:** Low
**Status:** Unexplored
### 2. Extract Business Logic from explorer.py
**Description:** Break the 143KB explorer.py monolith into pure functions in a new module (e.g., analysis/explorer_core.py), keeping only UI glue in the main file.
**Rationale:** explorer.py is too large to navigate, review, or refactor safely. The explorer_helpers.py pattern already proves pure functions work. This enables parallel development and safer changes.
**Downsides:** High complexity - requires understanding all the current dependencies and careful extraction to avoid breaking the Streamlit UI.
**Confidence:** 90%
**Complexity:** High
**Status:** Unexplored
### 3. SVD Component Label Verification
**Description:** Create a pre-deployment verification script that checks SVD_THEMES labels against actual voting data, flagging components where labels don't match party score distributions.
**Rationale:** The documented SVD label bug showed labels can drift from reality. A verification step before deployment prevents this recurring.
**Downsides:** Requires clear criteria for "label matches voting data" - some components are genuinely ambiguous.
**Confidence:** 85%
**Complexity:** Medium
**Status:** Unexplored
### 4. Interactive Component-Explorer UI
**Description:** Add a Streamlit UI selector letting users view any pair of SVD components as a 2D scatter plot, not just the political compass (components 1-2).
**Rationale:** Components 3-10 are essentially black boxes. Making these explorable reveals hidden political dimensions and adds significant user value.
**Downsides:** Requires understanding how to project between arbitrary component pairs.
**Confidence:** 85%
**Complexity:** Medium
**Status:** Unexplored
### 5. Type-Safe Vote Normalization
**Description:** Replace string-based vote normalization (casting '1', '-1', '0' strings) with typed enums and exhaustiveness checking.
**Rationale:** Vote matching is core functionality - wrong types cause silent bugs. Typed enums catch errors at compile time.
**Downsides:** Requires updating all callers and ensuring backward compatibility.
**Confidence:** 80%
**Complexity:** Medium
**Status:** Unexplored
### 6. Add CONTRIBUTING.md
**Description:** Create top-level CONTRIBUTING.md covering setup (uv), running tests, lint/typecheck commands, and key conventions from AGENTS.md.
**Rationale:** AGENTS.md is internal-focused. A CONTRIBUTING.md lowers the barrier for external contributors and encodes project norms explicitly.
**Downsides:** Low risk - straightforward documentation.
**Confidence:** 75%
**Complexity:** Low
**Status:** Explored
### 7. Database Schema Validation
**Description:** Add startup validation that checks the actual database schema against expected schema. Verify table existence, column types, and foreign key relationships. Fail fast with clear error messages if schema is stale.
**Rationale:** The current code tries to add columns with `ALTER TABLE ... IF NOT EXISTS` which can fail silently. A schema validation pass catches migration failures immediately.
**Downsides:** Schema changes require updating validation code.
**Confidence:** 85%
**Complexity:** Low
**Status:** Unexplored
### 8. DuckDB Connection Leak Detector
**Description:** Audit all `duckdb.connect()` calls for proper context manager usage or explicit `.close()`. Many handlers catch exceptions but forget to close connections. Add a `ConnectionTracker` that warns on unclosed connections in development.
**Rationale:** Connection leaks accumulate and eventually exhaust database connections. The codebase has 15+ places where exceptions cause early returns without connection cleanup.
**Downsides:** Tracking adds overhead; some leaks are already handled by DuckDB's connection pooling.
**Confidence:** 85%
**Complexity:** Medium
**Status:** Unexplored
### 9. Static Analysis Rule for Bare Except
**Description:** Add a flake8 plugin or ruff rule that flags `except:` and `except Exception:` without re-raising or logging. Document the project-specific exception hierarchy.
**Rationale:** Prevents the anti-pattern from re-entering. The project has 208 violations — a custom lint rule catches new violations and encodes the team's error-handling philosophy.
**Downsides:** Requires defining what specific exceptions ARE allowed per context.
**Confidence:** 70%
**Complexity:** Low
**Status:** Unexplored
### 10. SVD Component Label Verification
**Description:** Create a CI/CD pre-deployment script that verifies SVD labels against actual voting data — checking that labels match the voting pattern, not semantic assumptions.
**Rationale:** The SVD label documentation exists but there's no enforcement. This automated check prevents the documented mistake from recurring.
**Downsides:** Requires understanding of the SVD pipeline and periodic re-calibration.
**Confidence:** 75%
**Complexity:** Medium
**Status:** Unexplored
## Rejection Summary (Raised Bar — 2026-04-05)
| # | Idea | Reason Rejected |
|---|------|-----------------|
| 1 | Consolidate 3 venvs into 1 | Lower priority - works currently, would need investigation |
| 2 | Modularize database.py | Secondary to explorer.py refactor; not a direct user/developer impact |
| 3 | Add Makefile/Task Aliases | Nice-to-have, lower leverage |
| 4 | Exception Handler Audit (208 handlers) | Too large to scope safely; architectural, not fixing root cause |
| 5 | Add Comprehensive Type Hints | Huge scope; hygiene, not correctness |
| 6 | Party Polarization Score | Interesting but niche |
| 7 | Scree Plot Extension | Low urgency feature |
| 8 | Typed DTOs for Database Layer | High migration effort; duckdb interop complications |
| 9 | Nested Exception Handler Flattening | Architectural refactor; too much change for uncertain value |
| 10 | Print→Logging Replacement (179 print statements) | High effort, low leverage — logging exists but not used |
| 11 | Code Climate Metrics | Measures for its own sake; doesn't directly prevent bugs |
| 12 | CONTRIBUTING.md | Good hygiene, low urgency — can defer |
## Session Log
- 2026-04-04: Initial ideation — 32 generated, 6 survived
- 2026-04-05: Raised the bar — 22 ideas reviewed, 5 survivors after stricter filtering
- Idea #1 (Right-Wing Party Axis Validation) selected for brainstorming