docs: add session ledger files and logic-errors docs

2026-04-04 18:47:05 +02:00
parent 1dbf8da3a2
commit 805cc7e284
6 changed files with 442 additions and 0 deletions
@@ -0,0 +1,85 @@
+---
+session: ses_2a6e
+updated: 2026-04-04T15:34:15.344Z
+---
+
+# Session Summary
+
+## Goal
+Analyze and document how the most important motions are defined and ranked in the Stemwijzer codebase, focusing on importance criteria, selection mechanisms, metadata, key files, and user interaction patterns.
+
+## Constraints & Preferences
+- Provide detailed findings with file paths and line numbers
+- Focus on code analysis without making changes
+- Document the complete motion ranking and display system
+
+## Progress
+### Done
+- [x] Analyzed motion importance criteria (controversy_score, SVD scores, entropy-based discrimination)
+- [x] Documented motion selection mechanisms for SVD display, Political Compass, quiz, and similarity search
+- [x] Mapped database schema for motions, mp_votes, svd_vectors, similarity_cache tables
+- [x] Identified key files and their roles in motion handling
+- [x] Documented user interaction flows for SVD components tab, MP quiz, and motion browser
+- [x] Cataloged SVD_THEMES dictionary with all 10 component labels and explanations
+
+### In Progress
+- (none - analysis complete)
+
+### Blocked
+- (none)
+
+## Key Decisions
+- **Analysis-only session**: No code modifications were requested or made; this was purely investigative work to understand the existing motion ranking system.
+
+## Next Steps
+1. Awaiting further instructions from user on what to do with the analysis (e.g., implement changes, add features, optimize)
+
+## Critical Context
+### Motion Importance Metrics
+1. **Controversy Score**: `1 - winning_margin` (0.5 = even split, higher = more controversial)
+2. **SVD Component Scores**: Absolute projection on each SVD component axis
+3. **Entropy Score**: Shannon entropy of vote distribution (for quiz discrimination)
+
+### Motion Selection Strategies
+- **SVD Display**: Top 10 per component (5 positive pole, 5 negative pole)
+- **Political Compass**: Top 5 at each pole for axis labeling
+- **Quiz Seed**: Top 8 controversial motions with individual MP votes
+- **Quiz Discriminating**: Entropy-ranked motions that best split candidate MPs
+
+### Database Schema
+```sql
+motions: id, title, description, date, policy_area, voting_results (JSON), 
+         winning_margin, controversy_score, layman_explanation, body_text, url
+mp_votes: motion_id, mp_name, party, vote, date
+svd_vectors: window_id, entity_type, entity_id, vector (JSON 50-dim)
+similarity_cache: source_motion_id, target_motion_id, score, vector_type, window_id
+```
+
+### Key Functions
+| Function | Location | Purpose |
+|----------|----------|---------|
+| `get_motions_with_individual_votes()` | database.py:660-692 | Get controversial motions with MP votes |
+| `choose_discriminating_motions()` | database.py:817-903 | Entropy-based motion selection |
+| `_top_motion_ids()` | axis_classifier.py:274-295 | Top N motions per axis pole |
+| `build_svd_components_tab()` | explorer.py:3081-3497 | UI for SVD motion display |
+| `build_mp_quiz_tab()` | explorer.py:3499-3724 | MP quiz with adaptive motion selection |
+
+### SVD Themes Location
+`explorer.py:432-762` - Dictionary `SVD_THEMES` contains labels, explanations, and party poles for components 1-10.
+
+## File Operations
+### Read
+- `/home/sgeboers/Projects/stemwijzer/analysis/axis_classifier.py`
+- `/home/sgeboers/Projects/stemwijzer/database.py`
+- `/home/sgeboers/Projects/stemwijzer/explorer.py` (partial reads at offsets 1860, 3050, 3400)
+- `/home/sgeboers/Projects/stemwijzer/pages/1_Stemwijzer.py`
+- `/home/sgeboers/Projects/stemwijzer/scripts/generate_svd_json.py`
+- `/home/sgeboers/Projects/stemwijzer/similarity/lookup.py`
+- `/home/sgeboers/Projects/stemwijzer/src/types/motion_types.py`
+- `/home/sgeboers/Projects/stemwijzer/migrations/2026_03_21__create_mp_metadata.sql`
+- `/home/sgeboers/Projects/stemwijzer/migrations/2026_03_21__create_mp_votes.sql`
+- `/home/sgeboers/Projects/stemwijzer/migrations/2026_03_21__create_svd_vectors.sql`
+- `/home/sgeboers/Projects/stemwijzer/migrations/2026-03-22-add-similarity-cache.sql`
+
+### Modified
+- (none)
@@ -0,0 +1,56 @@
+---
+session: ses_2b07
+updated: 2026-04-02T19:01:27.654Z
+---
+
+# Session Summary
+
+## Goal
+Execute Task 2 from the SVD Label Unification implementation plan: refactor explorer.py to export SVD_THEMES at module level and update analysis/svd_labels.py to import it properly.
+
+## Constraints & Preferences
+- Follow TDD principles: run tests before/after changes
+- Make minimal changes to accomplish the task
+- Preserve all existing SVD_THEMES data (10 components with labels, explanations, poles, flip settings)
+- Ensure no circular import issues between explorer.py and analysis/svd_labels.py
+
+## Progress
+### Done
+- [x] Ran baseline tests (4 tests passed in tests/test_svd_labels.py)
+- [x] Moved SVD_THEMES dict from inside `build_svd_components_tab` function (line ~2639) to module level in explorer.py (after PARTY_COLOURS, line 434)
+- [x] Removed duplicate SVD_THEMES definition from inside `build_svd_components_tab` function
+- [x] Updated `_get_svd_themes()` function in analysis/svd_labels.py to import directly from explorer module instead of using complex importlib.util fallback
+- [x] Verified all 4 tests still pass after changes
+- [x] Confirmed SVD_THEMES is now accessible at module level in explorer.py for external import
+
+### In Progress
+- [ ] Commit the changes (changes staged but not yet committed)
+
+### Blocked
+- (none)
+
+## Key Decisions
+- **Import method**: Use direct `import explorer` and access `explorer.SVD_THEMES` instead of importlib.util machinery. Rationale: Now that SVD_THEMES is at module level, the direct import is clean and the lazy runtime import in `_get_svd_themes()` prevents circular dependencies at module load time.
+- **Module placement**: Placed SVD_THEMES after PARTY_COLOURS (line 434) to keep constants together near the top of the file. Rationale: This keeps the canonical source of truth visible and maintains logical grouping with other module-level constants.
+
+## Next Steps
+1. Run full test suite to verify no regressions: `uv run pytest tests/ -v`
+2. Commit the changes: `git add explorer.py analysis/svd_labels.py && git commit -m "refactor: move SVD_THEMES to module level for import"`
+3. Proceed to Task 3: Update axis_classifier.py to use svd_labels module
+
+## Critical Context
+- SVD_THEMES now defined at explorer.py line 434 with full type annotation `dict[int, dict[str, str]]`
+- SVD_THEMES contains 10 components (1-indexed) with keys: label, explanation, positive_pole, negative_pole, flip
+- Function `_get_svd_themes()` in analysis/svd_labels.py now uses simple import pattern with global cache `_svd_themes_cache`
+- The function references in explorer.py at lines 2691 and 2719 (`SVD_THEMES.get()`) continue to work unchanged since they now reference the module-level variable
+- All 4 tests in tests/test_svd_labels.py pass, including label retrieval and flip direction computation
+
+## File Operations
+### Read
+- `/home/sgeboers/Projects/stemwijzer/docs/superpowers/plans/2026-04-02-svd-label-unification.md`
+- `/home/sgeboers/Projects/stemwijzer/explorer.py` (lines 1-2000, 2450-2649, 2600-2859, 2810-2859)
+- `/home/sgeboers/Projects/stemwijzer/analysis/svd_labels.py`
+
+### Modified
+- `/home/sgeboers/Projects/stemwijzer/explorer.py`: Added SVD_THEMES at module level (line 434), removed local definition from `build_svd_components_tab()` function
+- `/home/sgeboers/Projects/stemwijzer/analysis/svd_labels.py`: Simplified `_get_svd_themes()` to use direct import from explorer instead of importlib.util fallback
@@ -0,0 +1,55 @@
+---
+session: ses_2b4f
+updated: 2026-04-01T21:57:48.280Z
+---
+
+# Session Summary
+
+## Goal
+Analyze how the SVD Components tab in `explorer.py` computes party positions, focusing on: data loading flow, window_size default, X/Y coordinate computation, and whether positions are individual MPs or party centroids.
+
+## Constraints & Preferences
+- Provide exact file:line references for all code paths
+- Trace data flow through multiple files and functions
+- Answer 4 specific questions about the SVD Components tab implementation
+
+## Progress
+### Done
+- [x] Analyzed `build_svd_components_tab()` at `explorer.py:2449-2867`
+- [x] Traced `load_positions()` at `explorer.py:603-656` — default window_size is "quarterly"
+- [x] Traced `load_party_axis_scores()` at `explorer.py:836-853`
+- [x] Traced `_load_mp_vectors_by_party()` at `explorer.py:778-832`
+- [x] Analyzed `compute_2d_axes()` at `analysis/political_axis.py:131-476`
+- [x] Analyzed `compute_party_bootstrap_cis()` at `analysis/political_axis.py:624-695`
+- [x] Analyzed `compute_party_centroids()` at `explorer_helpers.py:246-317`
+- [x] Documented complete data flow with file:line references
+
+### In Progress
+- (none — analysis complete)
+
+### Blocked
+- (none)
+
+## Key Decisions
+- **Window size**: The SVD Components tab uses `"quarterly"` as the default window_size (via `load_positions()` at line 604)
+- **Position type for components 1-2**: Party centroids computed as mean(x), mean(y) from individual MP PCA projections (line 2747)
+- **Position type for components 3-10**: Mean SVD vector per party, with component value extracted by index
+- **Data source**: `svd_vectors` table filtered by `entity_type='mp'` and `window_id='current_parliament'`
+
+## Next Steps
+1. (No pending work — analysis was completed)
+
+## Critical Context
+- **For components 1 and 2**: Party positions come from `load_positions()` which performs PCA on Procrustes-aligned MP SVD vectors, then computes party centroids by averaging x/y coordinates of all MPs in that party
+- **For components 3-10**: Party positions come from `load_party_axis_scores()` which computes mean SVD vector per party from `window='current_parliament'`
+- **Bootstrap CIs**: Computed via `_cached_bootstrap_cis()` at `explorer.py:873-880` using `compute_party_bootstrap_cis()` from `analysis/political_axis.py`
+- **MP→Party mapping**: Via `mp_metadata` table, normalized using `_PARTY_NORMALIZE` dict at `explorer.py:471-480`
+
+## File Operations
+### Read
+- `/home/sgeboers/Projects/stemwijzer/explorer.py` (full file: lines 1-3094)
+- `/home/sgeboers/Projects/stemwijzer/analysis/political_axis.py` (full file: lines 1-695)
+- `/home/sgeboers/Projects/stemwijzer/explorer_helpers.py` (full file: lines 1-317)
+
+### Modified
+- (none)
@@ -0,0 +1,78 @@
+---
+session: ses_2b9f
+updated: 2026-04-04T16:29:25.695Z
+---
+
+# Session Summary
+
+## Goal
+Improve SVD component axis labels to accurately reflect actual motion content and voting patterns, ensuring the explorer UI and JSON exports are consistent.
+
+## Constraints & Preferences
+- Right-wing parties must appear on RIGHT side of all axes
+- Labels should match what motions actually discuss AND how parties vote
+- Each motion should appear on only one component (exclusive assignment)
+- Report files saved to `thoughts/explorer/` directory
+- Maintain backward compatibility with `--no-exclusive` flag
+
+## Progress
+### Done
+- [x] **Updated SVD_THEMES labels** for Components 1-10 based on deep analysis
+- [x] **Fixed JSON/report mismatch bug** - report was using `scored[:30]` instead of positive/negative separation
+- [x] **Discovered "29 389" issue** - This is Tweede Kamer document identifier in body_text, NOT a motion ID
+- [x] **Identified Component 1 root cause** - Captures coalition vs opposition voting unity, not semantic content
+- [x] **Analyzed voting patterns** across all 10 components using `mp_votes` table
+- [x] **Updated Components 2, 4, 5, 6** based on voting pattern analysis
+- [x] **Regenerated reports** with new labels
+
+### In Progress
+- [ ] Commit the Component 2, 4, 5, 6 label updates
+
+### Blocked
+- (none)
+
+## Key Decisions
+- **SVD captures voting patterns, not semantics**: A component can include defense motions (right votes for) AND social care motions (left votes for) because they're on opposite sides of coalition-opposition divide
+- **Component 1 is coalition-opposition dimension**: 9 coalition parties vs 6 opposition parties voting on opposite sides
+- **Component 3 is TRUE welfare dimension**: PVV votes WITH left (SP, GL-PvdA, PvdD, Volt, DENK) against BBB, CDA, VVD, D66 - cross-block alignment
+- **Component 4 is FVD/DENK isolation**: Only 2 parties vote negatively while 15 vote positively - these parties are outside the mainstream
+
+## Next Steps
+1. **Commit Component 2, 4, 5, 6 label updates**
+2. **Test the explorer** to verify labels render correctly in UI
+3. **Review Component 3** - current label "Verzorgingsstaat vs bezuinigingen" is accurate (cross-block welfare voting)
+4. **Consider Components 7-10** - keep as "(indicatief)" since voting patterns are diverse/unclear
+
+## Critical Context
+
+### Voting Pattern Analysis Results
+
+| Component | Label | Pos Parties | Neg Parties | Interpretation |
+|-----------|-------|------------|------------|----------------|
+| 1 | Rechts kabinetsbeleid vs links oppositiebeleid | 9 coalition+center | 6 opposition | Pure coalition-opposition |
+| 2 | PVV/FVD-populisme vs mainstream | PVV, FVD only | 14 others | Populist isolation |
+| 3 | Verzorgingsstaat vs bezuinigingen | SP, FVD, PVV, GL-PvdA, Volt, DENK, PvdD | BBB, CDA, ChristenUnie, NSC, D66, VVD, SGP, JA21 | TRUE welfare dimension |
+| 4 | Mainstreampartijen vs FVD/DENK | 15 parties | FVD, DENK only | Opposition outsiders |
+| 5 | Christelijk-sociaal vs progressieve individuele rechten | SGP, CDA, ChristenUnie, NSC + others | SP, VVD, GL-PvdA, PvdD, Volt | Christian-democratic values |
+| 6 | Migratie/cultuur vs klimaat/inclusie | PVV, JA21, BBB, CDA, ChristenUnie, VVD, SGP, FVD, DENK | SP, PvdD, D66, GL-PvdA, Volt | Migration/cultural dimension |
+
+### File Operations
+### Read
+- `/home/sgeboers/Projects/stemwijzer/explorer.py` (SVD_THEMES at lines 434-611)
+- `/home/sgeboers/Projects/stemwijzer/scripts/generate_svd_json.py`
+- `/home/sgeboers/Projects/stemwijzer/thoughts/explorer/top_svd_top_motions.json`
+- `/home/sgeboers/Projects/stemwijzer/thoughts/explorer/top_svd_top_motions_report.md`
+
+### Modified
+- `/home/sgeboers/Projects/stemwijzer/explorer.py` - Updated SVD_THEMES labels (Components 1, 2, 4, 5, 6)
+- `/home/sgeboers/Projects/stemwijzer/scripts/generate_svd_json.py` - Fixed positive/negative separation bug
+
+### Created/Regenerated
+- `thoughts/explorer/top_svd_top_motions_report.md` (with updated labels)
+- `thoughts/explorer/top_svd_top_motions.json` (84 rows, 10 components)
+
+### Commits
+- `33edb33` - feat: implement exclusive SVD motion assignment with label review report
+- `e77f0ec` - fix: update SVD_THEMES labels to match actual motion content
+- `bfe37c6` - fix: align report generation with JSON output for positive/negative separation
+- `f7fc908` - fix: update Component 1 label to coalition-opposition reality
@@ -0,0 +1,72 @@
+---
+session: ses_2bed
+updated: 2026-03-31T00:07:06.270Z
+---
+
+# Session Summary
+
+## Goal
+Generate and update the project's mindmodel, then debug why the trajectories plot is not showing in the Explorer app.
+
+## Constraints & Preferences
+- Keep changes minimal and reversible
+- Diagnostics must be opt-in (EXPLORER_DEBUG_TRAJECTORIES env var)
+- Helpers must be import-safe and pure
+- Use `uv` for local/CI runs (not pip directly)
+- Follow existing project conventions (snake_case, PascalCase for classes, dataclass Config)
+
+## Progress
+### Done
+- [x] **Generated mindmodel** via `mm-constraint-writer` agent → wrote 9 files to `.mindmodel/`:
+  - `manifest.yaml`, `stack/stack.yaml`, `architecture/architecture.yaml`, `conventions/conventions.yaml`, `domain/domain-glossary.yaml`, `patterns/patterns.yaml`, `anti-patterns/anti-patterns.yaml`, `dependencies/dependencies.yaml`, `constraints/README.md`
+  - Top anti-pattern: `explorer_helpers.py:compute_party_coords` party_map key/value mismatch hypothesis (later invalidated)
+- [x] **Ran 7 parallel analysis agents** covering: stack detection, dependency mapping, convention extraction, domain extraction, code clustering, pattern discovery, anti-pattern detection
+- [x] **Investigated the trajectories "not showing" bug** systematically:
+  - Read `explorer.py` (2948 lines), `explorer_helpers.py` (297 lines), `analysis/political_axis.py` (695 lines), `analysis/trajectory.py` (297 lines), `analysis/visualize.py`, `scripts/diagnose_trajectories_cli.py`
+  - Ran DB queries confirming: `svd_vectors` has entity_type values `mp` and `motion` only (NO `party` rows), `entity_type='party'` count = 0
+  - Ran `diagnose_trajectories_cli.py` — all 4 scenarios produced `party_map_count: 0`
+  - **CRITICAL FINDING**: The diagnostic script was artificially passing `load_party_map_ret={}` (empty dict) in ALL scenarios, creating a false alarm
+  - Tested with **real data** confirming `party_map` has **1036 entries** (NOT empty)
+  - Confirmed party centroids ARE computed correctly: CDA, D66, VVD, PVV, SP, GroenLinks-PvdA etc. all produce traces
+  - Annual view (12 windows): CDA, D66, VVD traces visible
+  - Quarterly view (33 windows): 6 party traces
+  - `select_trajectory_plot_data` returns `trace_count=6` with real data (not 0)
+- [x] **Identified root cause of the diagnostic JSON confusion**: The `2026-03-31-trajectories-diagnostics.json` was created by `diagnose_trajectories_cli.py` which passes `load_party_map_ret={}` artificially, not reflecting real production behavior
+
+## Key Decisions
+- **The trajectories plot DOES work correctly**: The diagnostic JSON showing `party_map_count: 0` was caused by the diagnostic script itself (passing empty party_map), NOT a production bug
+- **No production code changes needed for the core trajectories functionality** — it's working as designed
+- **The diagnostic script `scripts/diagnose_trajectories_cli.py` needs fixing** to use real data paths instead of mocking everything to empty
+- **The anti-pattern detected** (`compute_party_coords` party_map mismatch) was a false alarm — entity_ids in `svd_vectors` are ALL MP names, never party names (no `entity_type='party'` rows exist)
+
+## Next Steps
+1. **Fix `scripts/diagnose_trajectories_cli.py`** to use real data paths (`data/motions.db`) and real `load_party_map` / `load_positions` calls instead of mocking everything to empty
+2. **Re-run the fixed diagnostic script** to produce a correct `trajectories-diagnostics.json` artifact
+3. **Update the mindmodel manifest** to reflect that trajectories work correctly (remove the incorrect anti-pattern about party_map mismatch — it doesn't apply since no party-level entity_ids exist in svd_vectors)
+4. **Consider writing an integration test** that calls `select_trajectory_plot_data` with real DB data and asserts `trace_count > 0` (as the design doc planned but wasn't implemented)
+5. **Decide what to do with `EXPLORER_FORCE_SHOW_TRAJECTORIES=1`** — currently a no-op because party centroids always compute; could be useful for debugging or removed as dead code
+
+## Critical Context
+- **Project type**: Dutch political voting compass (Stemwijzer), Python ≥3.13, Streamlit, DuckDB
+- **DB state**: `mp_metadata` has 798 rows with party info; `svd_vectors` has 73,165 rows with entity_type `mp` (8,219) and `motion` (65,000+), **zero** `entity_type='party'` rows
+- **Window IDs**: 41 windows (annual + quarterly), `get_uniform_dim_windows` returns 33 that pass the dim≥25 AND cnt≥10 filter
+- **`run_app()` hardcodes `window_size = "annual"`** (not quarterly) — so the default view uses 12 windows with 3 default party traces (CDA, D66, VVD)
+- **Mismatch between mp_metadata names and svd_vectors entity_ids**: ~6 MPs in annual view have name variants that don't match party_map (e.g., `De Pater-Postma, W.L.` vs `Pater-Postma de, W.L.`), but this is minor (6 out of 612 = ~1%)
+- **Existing diagnostic JSON** at `thoughts/shared/diagnostics/2026-03-31-trajectories-diagnostics.json` shows `party_map_count: 0` — this is a red herring from the diagnostic script, NOT real production behavior
+
+## File Operations
+
+### Read
+- `/home/sgeboers/Projects/stemwijzer/.mindmodel/manifest.yaml`
+- `/home/sgeboers/Projects/stemwijzer/explorer.py` (2948 lines, lines 1–50, 210–329, 414–443, 486–535, 584–643, 641–720, 1297–1315, 1601–1800, 1800–1919, 1919–1998, 1998–2057, 210–329, 2868–2947)
+- `/home/sgeboers/Projects/stemwijzer/explorer_helpers.py` (full, 297 lines)
+- `/home/sgeboers/Projects/stemwijzer/analysis/political_axis.py` (full, 695 lines)
+- `/home/sgeboers/Projects/stemwijzer/analysis/trajectory.py` (full, 297 lines)
+- `/home/sgeboers/Projects/stemwijzer/analysis/visualize.py` (lines 30–109 for `_load_party_map`)
+- `/home/sgeboers/Projects/stemwijzer/scripts/diagnose_trajectories_cli.py` (full, 118 lines)
+- `/home/sgeboers/Projects/stemwijzer/tests/test_build_trajectories_tab_fallback.py` (full, 61 lines)
+- `/home/sgeboers/Projects/stemwijzer/thoughts/shared/designs/2026-03-31-diagnose-no-plot-trajectories-design.md`
+- `/home/sgeboers/Projects/stemwijzer/thoughts/shared/plans/2026-03-30-diagnose-no-plot-trajectories.md`
+
+### Modified
+- (none yet — verified trajectories work correctly via DB queries and Python tests)