parent
1dbf8da3a2
commit
805cc7e284
@ -0,0 +1,96 @@ |
|||||||
|
--- |
||||||
|
title: SVD component labels incorrect due to semantic vs voting pattern mismatch |
||||||
|
date: 2026-04-04 |
||||||
|
category: docs/solutions/logic-errors/ |
||||||
|
module: Stemwijzer Data Analysis |
||||||
|
problem_type: logic_error |
||||||
|
component: explorer |
||||||
|
symptoms: |
||||||
|
- Component 1 label "Sociale zekerheid vs economische liberalisering" did not match voting patterns |
||||||
|
- Report analysis showed different party alignment than label suggested |
||||||
|
- SVD components captured voting patterns but labels described semantic content |
||||||
|
root_cause: logic_error |
||||||
|
resolution_type: code_fix |
||||||
|
severity: high |
||||||
|
tags: [svd, voting-analysis, component-labels, logic-error] |
||||||
|
--- |
||||||
|
|
||||||
|
# SVD component labels incorrect due to semantic vs voting pattern mismatch |
||||||
|
|
||||||
|
## Problem |
||||||
|
The SVD (Singular Value Decomposition) component labels in `explorer.py` were based on semantic analysis of motion titles, but the SVD actually captures HOW parties vote, not WHAT topics are discussed. This resulted in misleading component labels that did not match actual voting patterns. |
||||||
|
|
||||||
|
## Symptoms |
||||||
|
- Component 1 was labeled "Sociale zekerheid vs economische liberalisering" but actually captured coalition vs opposition voting |
||||||
|
- Analysis report showed different party groupings than the labels suggested |
||||||
|
- Report generation used incorrect slice (`scored[:30]`) instead of positive/negative party separation |
||||||
|
|
||||||
|
## What Didn't Work |
||||||
|
- Semantic analysis of motion titles to determine component labels |
||||||
|
- Assuming that topics discussed in motions matched how parties voted on them |
||||||
|
- Report generation logic was inconsistent with JSON output logic |
||||||
|
|
||||||
|
## Solution |
||||||
|
|
||||||
|
### 1. Report Generation Bug Fix (commit bfe37c6) |
||||||
|
Fixed the report generation to use positive/negative party lists correctly instead of `scored[:30]`: |
||||||
|
|
||||||
|
```python |
||||||
|
# Before (incorrect) |
||||||
|
scored[:30] |
||||||
|
|
||||||
|
# After (correct) |
||||||
|
positive_parties = [p for p, s in scored if s > 0] |
||||||
|
negative_parties = [p for p, s in scored if s < 0] |
||||||
|
``` |
||||||
|
|
||||||
|
### 2. Component 1 Label Fix (commit f7fc908) |
||||||
|
Changed from semantic-based to voting-pattern-based label: |
||||||
|
|
||||||
|
```python |
||||||
|
# Before (incorrect) |
||||||
|
"label": "Sociale zekerheid vs economische liberalisering" |
||||||
|
|
||||||
|
# After (correct) |
||||||
|
"label": "Rechts kabinetsbeleid vs links oppositiebeleid" |
||||||
|
``` |
||||||
|
|
||||||
|
Root cause: Component 1 captures 9 coalition parties voting together vs 6 opposition parties voting together. |
||||||
|
|
||||||
|
### 3. Components 2, 4, 5, 6 Label Updates (commit 92c3c0e) |
||||||
|
- **Component 2**: "PVV/FVD-populisme versus mainstream-partijen" — Only PVV and FVD vote positively |
||||||
|
- **Component 4**: "Mainstreampartijen versus FVD/DENK-oppositie" — Only FVD and DENK vote negatively |
||||||
|
- **Component 5**: "Christelijk-sociaal en gemeenschapswaarden versus progressieve individuele rechten" |
||||||
|
- **Component 6**: "Migratie en cultuur versus klimaat en progressieve inclusie" |
||||||
|
|
||||||
|
### 4. Exclusive Motion Assignment (commit 33edb33) |
||||||
|
Each motion now appears on only one component (highest absolute loading): |
||||||
|
|
||||||
|
```python |
||||||
|
# Each motion assigned to component with highest absolute loading |
||||||
|
# Backward compatible with --no-exclusive flag |
||||||
|
``` |
||||||
|
|
||||||
|
## Why This Works |
||||||
|
|
||||||
|
**Critical Insight**: SVD captures voting patterns, not semantic content. When labeling SVD components: |
||||||
|
- Look at which parties vote positively vs negatively |
||||||
|
- Don't assume semantics match voting patterns |
||||||
|
- Coalition vs opposition is a strong voting dimension in parliamentary data |
||||||
|
- Components may include motions from seemingly unrelated topics if parties vote the same way |
||||||
|
|
||||||
|
The fix works because it aligns labels with actual voting data: |
||||||
|
- Labels now describe the voting behavior of parties |
||||||
|
- Positive/negative poles show which parties vote which way |
||||||
|
- Explanations reference specific motions that illustrate the pattern |
||||||
|
|
||||||
|
## Prevention |
||||||
|
|
||||||
|
1. **Always verify SVD labels against voting data** — Before finalizing labels, check which parties score positively and negatively on each component |
||||||
|
2. **Test label-party alignment** — Add a test that verifies component labels match the party groupings in the data |
||||||
|
3. **Document the semantic vs voting distinction** — Make this a known Gotcha in the codebase for future developers |
||||||
|
|
||||||
|
## Related Issues |
||||||
|
- Analysis: `thoughts/explorer/top_svd_top_motions_report.md` |
||||||
|
- JSON generator: `scripts/generate_svd_json.py` |
||||||
|
- Labels source: `explorer.py:434-611` (SVD_THEMES dictionary) |
||||||
@ -0,0 +1,85 @@ |
|||||||
|
--- |
||||||
|
session: ses_2a6e |
||||||
|
updated: 2026-04-04T15:34:15.344Z |
||||||
|
--- |
||||||
|
|
||||||
|
# Session Summary |
||||||
|
|
||||||
|
## Goal |
||||||
|
Analyze and document how the most important motions are defined and ranked in the Stemwijzer codebase, focusing on importance criteria, selection mechanisms, metadata, key files, and user interaction patterns. |
||||||
|
|
||||||
|
## Constraints & Preferences |
||||||
|
- Provide detailed findings with file paths and line numbers |
||||||
|
- Focus on code analysis without making changes |
||||||
|
- Document the complete motion ranking and display system |
||||||
|
|
||||||
|
## Progress |
||||||
|
### Done |
||||||
|
- [x] Analyzed motion importance criteria (controversy_score, SVD scores, entropy-based discrimination) |
||||||
|
- [x] Documented motion selection mechanisms for SVD display, Political Compass, quiz, and similarity search |
||||||
|
- [x] Mapped database schema for motions, mp_votes, svd_vectors, similarity_cache tables |
||||||
|
- [x] Identified key files and their roles in motion handling |
||||||
|
- [x] Documented user interaction flows for SVD components tab, MP quiz, and motion browser |
||||||
|
- [x] Cataloged SVD_THEMES dictionary with all 10 component labels and explanations |
||||||
|
|
||||||
|
### In Progress |
||||||
|
- (none - analysis complete) |
||||||
|
|
||||||
|
### Blocked |
||||||
|
- (none) |
||||||
|
|
||||||
|
## Key Decisions |
||||||
|
- **Analysis-only session**: No code modifications were requested or made; this was purely investigative work to understand the existing motion ranking system. |
||||||
|
|
||||||
|
## Next Steps |
||||||
|
1. Awaiting further instructions from user on what to do with the analysis (e.g., implement changes, add features, optimize) |
||||||
|
|
||||||
|
## Critical Context |
||||||
|
### Motion Importance Metrics |
||||||
|
1. **Controversy Score**: `1 - winning_margin` (0.5 = even split, higher = more controversial) |
||||||
|
2. **SVD Component Scores**: Absolute projection on each SVD component axis |
||||||
|
3. **Entropy Score**: Shannon entropy of vote distribution (for quiz discrimination) |
||||||
|
|
||||||
|
### Motion Selection Strategies |
||||||
|
- **SVD Display**: Top 10 per component (5 positive pole, 5 negative pole) |
||||||
|
- **Political Compass**: Top 5 at each pole for axis labeling |
||||||
|
- **Quiz Seed**: Top 8 controversial motions with individual MP votes |
||||||
|
- **Quiz Discriminating**: Entropy-ranked motions that best split candidate MPs |
||||||
|
|
||||||
|
### Database Schema |
||||||
|
```sql |
||||||
|
motions: id, title, description, date, policy_area, voting_results (JSON), |
||||||
|
winning_margin, controversy_score, layman_explanation, body_text, url |
||||||
|
mp_votes: motion_id, mp_name, party, vote, date |
||||||
|
svd_vectors: window_id, entity_type, entity_id, vector (JSON 50-dim) |
||||||
|
similarity_cache: source_motion_id, target_motion_id, score, vector_type, window_id |
||||||
|
``` |
||||||
|
|
||||||
|
### Key Functions |
||||||
|
| Function | Location | Purpose | |
||||||
|
|----------|----------|---------| |
||||||
|
| `get_motions_with_individual_votes()` | database.py:660-692 | Get controversial motions with MP votes | |
||||||
|
| `choose_discriminating_motions()` | database.py:817-903 | Entropy-based motion selection | |
||||||
|
| `_top_motion_ids()` | axis_classifier.py:274-295 | Top N motions per axis pole | |
||||||
|
| `build_svd_components_tab()` | explorer.py:3081-3497 | UI for SVD motion display | |
||||||
|
| `build_mp_quiz_tab()` | explorer.py:3499-3724 | MP quiz with adaptive motion selection | |
||||||
|
|
||||||
|
### SVD Themes Location |
||||||
|
`explorer.py:432-762` - Dictionary `SVD_THEMES` contains labels, explanations, and party poles for components 1-10. |
||||||
|
|
||||||
|
## File Operations |
||||||
|
### Read |
||||||
|
- `/home/sgeboers/Projects/stemwijzer/analysis/axis_classifier.py` |
||||||
|
- `/home/sgeboers/Projects/stemwijzer/database.py` |
||||||
|
- `/home/sgeboers/Projects/stemwijzer/explorer.py` (partial reads at offsets 1860, 3050, 3400) |
||||||
|
- `/home/sgeboers/Projects/stemwijzer/pages/1_Stemwijzer.py` |
||||||
|
- `/home/sgeboers/Projects/stemwijzer/scripts/generate_svd_json.py` |
||||||
|
- `/home/sgeboers/Projects/stemwijzer/similarity/lookup.py` |
||||||
|
- `/home/sgeboers/Projects/stemwijzer/src/types/motion_types.py` |
||||||
|
- `/home/sgeboers/Projects/stemwijzer/migrations/2026_03_21__create_mp_metadata.sql` |
||||||
|
- `/home/sgeboers/Projects/stemwijzer/migrations/2026_03_21__create_mp_votes.sql` |
||||||
|
- `/home/sgeboers/Projects/stemwijzer/migrations/2026_03_21__create_svd_vectors.sql` |
||||||
|
- `/home/sgeboers/Projects/stemwijzer/migrations/2026-03-22-add-similarity-cache.sql` |
||||||
|
|
||||||
|
### Modified |
||||||
|
- (none) |
||||||
@ -0,0 +1,56 @@ |
|||||||
|
--- |
||||||
|
session: ses_2b07 |
||||||
|
updated: 2026-04-02T19:01:27.654Z |
||||||
|
--- |
||||||
|
|
||||||
|
# Session Summary |
||||||
|
|
||||||
|
## Goal |
||||||
|
Execute Task 2 from the SVD Label Unification implementation plan: refactor explorer.py to export SVD_THEMES at module level and update analysis/svd_labels.py to import it properly. |
||||||
|
|
||||||
|
## Constraints & Preferences |
||||||
|
- Follow TDD principles: run tests before/after changes |
||||||
|
- Make minimal changes to accomplish the task |
||||||
|
- Preserve all existing SVD_THEMES data (10 components with labels, explanations, poles, flip settings) |
||||||
|
- Ensure no circular import issues between explorer.py and analysis/svd_labels.py |
||||||
|
|
||||||
|
## Progress |
||||||
|
### Done |
||||||
|
- [x] Ran baseline tests (4 tests passed in tests/test_svd_labels.py) |
||||||
|
- [x] Moved SVD_THEMES dict from inside `build_svd_components_tab` function (line ~2639) to module level in explorer.py (after PARTY_COLOURS, line 434) |
||||||
|
- [x] Removed duplicate SVD_THEMES definition from inside `build_svd_components_tab` function |
||||||
|
- [x] Updated `_get_svd_themes()` function in analysis/svd_labels.py to import directly from explorer module instead of using complex importlib.util fallback |
||||||
|
- [x] Verified all 4 tests still pass after changes |
||||||
|
- [x] Confirmed SVD_THEMES is now accessible at module level in explorer.py for external import |
||||||
|
|
||||||
|
### In Progress |
||||||
|
- [ ] Commit the changes (changes staged but not yet committed) |
||||||
|
|
||||||
|
### Blocked |
||||||
|
- (none) |
||||||
|
|
||||||
|
## Key Decisions |
||||||
|
- **Import method**: Use direct `import explorer` and access `explorer.SVD_THEMES` instead of importlib.util machinery. Rationale: Now that SVD_THEMES is at module level, the direct import is clean and the lazy runtime import in `_get_svd_themes()` prevents circular dependencies at module load time. |
||||||
|
- **Module placement**: Placed SVD_THEMES after PARTY_COLOURS (line 434) to keep constants together near the top of the file. Rationale: This keeps the canonical source of truth visible and maintains logical grouping with other module-level constants. |
||||||
|
|
||||||
|
## Next Steps |
||||||
|
1. Run full test suite to verify no regressions: `uv run pytest tests/ -v` |
||||||
|
2. Commit the changes: `git add explorer.py analysis/svd_labels.py && git commit -m "refactor: move SVD_THEMES to module level for import"` |
||||||
|
3. Proceed to Task 3: Update axis_classifier.py to use svd_labels module |
||||||
|
|
||||||
|
## Critical Context |
||||||
|
- SVD_THEMES now defined at explorer.py line 434 with full type annotation `dict[int, dict[str, str]]` |
||||||
|
- SVD_THEMES contains 10 components (1-indexed) with keys: label, explanation, positive_pole, negative_pole, flip |
||||||
|
- Function `_get_svd_themes()` in analysis/svd_labels.py now uses simple import pattern with global cache `_svd_themes_cache` |
||||||
|
- The function references in explorer.py at lines 2691 and 2719 (`SVD_THEMES.get()`) continue to work unchanged since they now reference the module-level variable |
||||||
|
- All 4 tests in tests/test_svd_labels.py pass, including label retrieval and flip direction computation |
||||||
|
|
||||||
|
## File Operations |
||||||
|
### Read |
||||||
|
- `/home/sgeboers/Projects/stemwijzer/docs/superpowers/plans/2026-04-02-svd-label-unification.md` |
||||||
|
- `/home/sgeboers/Projects/stemwijzer/explorer.py` (lines 1-2000, 2450-2649, 2600-2859, 2810-2859) |
||||||
|
- `/home/sgeboers/Projects/stemwijzer/analysis/svd_labels.py` |
||||||
|
|
||||||
|
### Modified |
||||||
|
- `/home/sgeboers/Projects/stemwijzer/explorer.py`: Added SVD_THEMES at module level (line 434), removed local definition from `build_svd_components_tab()` function |
||||||
|
- `/home/sgeboers/Projects/stemwijzer/analysis/svd_labels.py`: Simplified `_get_svd_themes()` to use direct import from explorer instead of importlib.util fallback |
||||||
@ -0,0 +1,55 @@ |
|||||||
|
--- |
||||||
|
session: ses_2b4f |
||||||
|
updated: 2026-04-01T21:57:48.280Z |
||||||
|
--- |
||||||
|
|
||||||
|
# Session Summary |
||||||
|
|
||||||
|
## Goal |
||||||
|
Analyze how the SVD Components tab in `explorer.py` computes party positions, focusing on: data loading flow, window_size default, X/Y coordinate computation, and whether positions are individual MPs or party centroids. |
||||||
|
|
||||||
|
## Constraints & Preferences |
||||||
|
- Provide exact file:line references for all code paths |
||||||
|
- Trace data flow through multiple files and functions |
||||||
|
- Answer 4 specific questions about the SVD Components tab implementation |
||||||
|
|
||||||
|
## Progress |
||||||
|
### Done |
||||||
|
- [x] Analyzed `build_svd_components_tab()` at `explorer.py:2449-2867` |
||||||
|
- [x] Traced `load_positions()` at `explorer.py:603-656` — default window_size is "quarterly" |
||||||
|
- [x] Traced `load_party_axis_scores()` at `explorer.py:836-853` |
||||||
|
- [x] Traced `_load_mp_vectors_by_party()` at `explorer.py:778-832` |
||||||
|
- [x] Analyzed `compute_2d_axes()` at `analysis/political_axis.py:131-476` |
||||||
|
- [x] Analyzed `compute_party_bootstrap_cis()` at `analysis/political_axis.py:624-695` |
||||||
|
- [x] Analyzed `compute_party_centroids()` at `explorer_helpers.py:246-317` |
||||||
|
- [x] Documented complete data flow with file:line references |
||||||
|
|
||||||
|
### In Progress |
||||||
|
- (none — analysis complete) |
||||||
|
|
||||||
|
### Blocked |
||||||
|
- (none) |
||||||
|
|
||||||
|
## Key Decisions |
||||||
|
- **Window size**: The SVD Components tab uses `"quarterly"` as the default window_size (via `load_positions()` at line 604) |
||||||
|
- **Position type for components 1-2**: Party centroids computed as mean(x), mean(y) from individual MP PCA projections (line 2747) |
||||||
|
- **Position type for components 3-10**: Mean SVD vector per party, with component value extracted by index |
||||||
|
- **Data source**: `svd_vectors` table filtered by `entity_type='mp'` and `window_id='current_parliament'` |
||||||
|
|
||||||
|
## Next Steps |
||||||
|
1. (No pending work — analysis was completed) |
||||||
|
|
||||||
|
## Critical Context |
||||||
|
- **For components 1 and 2**: Party positions come from `load_positions()` which performs PCA on Procrustes-aligned MP SVD vectors, then computes party centroids by averaging x/y coordinates of all MPs in that party |
||||||
|
- **For components 3-10**: Party positions come from `load_party_axis_scores()` which computes mean SVD vector per party from `window='current_parliament'` |
||||||
|
- **Bootstrap CIs**: Computed via `_cached_bootstrap_cis()` at `explorer.py:873-880` using `compute_party_bootstrap_cis()` from `analysis/political_axis.py` |
||||||
|
- **MP→Party mapping**: Via `mp_metadata` table, normalized using `_PARTY_NORMALIZE` dict at `explorer.py:471-480` |
||||||
|
|
||||||
|
## File Operations |
||||||
|
### Read |
||||||
|
- `/home/sgeboers/Projects/stemwijzer/explorer.py` (full file: lines 1-3094) |
||||||
|
- `/home/sgeboers/Projects/stemwijzer/analysis/political_axis.py` (full file: lines 1-695) |
||||||
|
- `/home/sgeboers/Projects/stemwijzer/explorer_helpers.py` (full file: lines 1-317) |
||||||
|
|
||||||
|
### Modified |
||||||
|
- (none) |
||||||
@ -0,0 +1,78 @@ |
|||||||
|
--- |
||||||
|
session: ses_2b9f |
||||||
|
updated: 2026-04-04T16:29:25.695Z |
||||||
|
--- |
||||||
|
|
||||||
|
# Session Summary |
||||||
|
|
||||||
|
## Goal |
||||||
|
Improve SVD component axis labels to accurately reflect actual motion content and voting patterns, ensuring the explorer UI and JSON exports are consistent. |
||||||
|
|
||||||
|
## Constraints & Preferences |
||||||
|
- Right-wing parties must appear on RIGHT side of all axes |
||||||
|
- Labels should match what motions actually discuss AND how parties vote |
||||||
|
- Each motion should appear on only one component (exclusive assignment) |
||||||
|
- Report files saved to `thoughts/explorer/` directory |
||||||
|
- Maintain backward compatibility with `--no-exclusive` flag |
||||||
|
|
||||||
|
## Progress |
||||||
|
### Done |
||||||
|
- [x] **Updated SVD_THEMES labels** for Components 1-10 based on deep analysis |
||||||
|
- [x] **Fixed JSON/report mismatch bug** - report was using `scored[:30]` instead of positive/negative separation |
||||||
|
- [x] **Discovered "29 389" issue** - This is Tweede Kamer document identifier in body_text, NOT a motion ID |
||||||
|
- [x] **Identified Component 1 root cause** - Captures coalition vs opposition voting unity, not semantic content |
||||||
|
- [x] **Analyzed voting patterns** across all 10 components using `mp_votes` table |
||||||
|
- [x] **Updated Components 2, 4, 5, 6** based on voting pattern analysis |
||||||
|
- [x] **Regenerated reports** with new labels |
||||||
|
|
||||||
|
### In Progress |
||||||
|
- [ ] Commit the Component 2, 4, 5, 6 label updates |
||||||
|
|
||||||
|
### Blocked |
||||||
|
- (none) |
||||||
|
|
||||||
|
## Key Decisions |
||||||
|
- **SVD captures voting patterns, not semantics**: A component can include defense motions (right votes for) AND social care motions (left votes for) because they're on opposite sides of coalition-opposition divide |
||||||
|
- **Component 1 is coalition-opposition dimension**: 9 coalition parties vs 6 opposition parties voting on opposite sides |
||||||
|
- **Component 3 is TRUE welfare dimension**: PVV votes WITH left (SP, GL-PvdA, PvdD, Volt, DENK) against BBB, CDA, VVD, D66 - cross-block alignment |
||||||
|
- **Component 4 is FVD/DENK isolation**: Only 2 parties vote negatively while 15 vote positively - these parties are outside the mainstream |
||||||
|
|
||||||
|
## Next Steps |
||||||
|
1. **Commit Component 2, 4, 5, 6 label updates** |
||||||
|
2. **Test the explorer** to verify labels render correctly in UI |
||||||
|
3. **Review Component 3** - current label "Verzorgingsstaat vs bezuinigingen" is accurate (cross-block welfare voting) |
||||||
|
4. **Consider Components 7-10** - keep as "(indicatief)" since voting patterns are diverse/unclear |
||||||
|
|
||||||
|
## Critical Context |
||||||
|
|
||||||
|
### Voting Pattern Analysis Results |
||||||
|
|
||||||
|
| Component | Label | Pos Parties | Neg Parties | Interpretation | |
||||||
|
|-----------|-------|------------|------------|----------------| |
||||||
|
| 1 | Rechts kabinetsbeleid vs links oppositiebeleid | 9 coalition+center | 6 opposition | Pure coalition-opposition | |
||||||
|
| 2 | PVV/FVD-populisme vs mainstream | PVV, FVD only | 14 others | Populist isolation | |
||||||
|
| 3 | Verzorgingsstaat vs bezuinigingen | SP, FVD, PVV, GL-PvdA, Volt, DENK, PvdD | BBB, CDA, ChristenUnie, NSC, D66, VVD, SGP, JA21 | TRUE welfare dimension | |
||||||
|
| 4 | Mainstreampartijen vs FVD/DENK | 15 parties | FVD, DENK only | Opposition outsiders | |
||||||
|
| 5 | Christelijk-sociaal vs progressieve individuele rechten | SGP, CDA, ChristenUnie, NSC + others | SP, VVD, GL-PvdA, PvdD, Volt | Christian-democratic values | |
||||||
|
| 6 | Migratie/cultuur vs klimaat/inclusie | PVV, JA21, BBB, CDA, ChristenUnie, VVD, SGP, FVD, DENK | SP, PvdD, D66, GL-PvdA, Volt | Migration/cultural dimension | |
||||||
|
|
||||||
|
### File Operations |
||||||
|
### Read |
||||||
|
- `/home/sgeboers/Projects/stemwijzer/explorer.py` (SVD_THEMES at lines 434-611) |
||||||
|
- `/home/sgeboers/Projects/stemwijzer/scripts/generate_svd_json.py` |
||||||
|
- `/home/sgeboers/Projects/stemwijzer/thoughts/explorer/top_svd_top_motions.json` |
||||||
|
- `/home/sgeboers/Projects/stemwijzer/thoughts/explorer/top_svd_top_motions_report.md` |
||||||
|
|
||||||
|
### Modified |
||||||
|
- `/home/sgeboers/Projects/stemwijzer/explorer.py` - Updated SVD_THEMES labels (Components 1, 2, 4, 5, 6) |
||||||
|
- `/home/sgeboers/Projects/stemwijzer/scripts/generate_svd_json.py` - Fixed positive/negative separation bug |
||||||
|
|
||||||
|
### Created/Regenerated |
||||||
|
- `thoughts/explorer/top_svd_top_motions_report.md` (with updated labels) |
||||||
|
- `thoughts/explorer/top_svd_top_motions.json` (84 rows, 10 components) |
||||||
|
|
||||||
|
### Commits |
||||||
|
- `33edb33` - feat: implement exclusive SVD motion assignment with label review report |
||||||
|
- `e77f0ec` - fix: update SVD_THEMES labels to match actual motion content |
||||||
|
- `bfe37c6` - fix: align report generation with JSON output for positive/negative separation |
||||||
|
- `f7fc908` - fix: update Component 1 label to coalition-opposition reality |
||||||
@ -0,0 +1,72 @@ |
|||||||
|
--- |
||||||
|
session: ses_2bed |
||||||
|
updated: 2026-03-31T00:07:06.270Z |
||||||
|
--- |
||||||
|
|
||||||
|
# Session Summary |
||||||
|
|
||||||
|
## Goal |
||||||
|
Generate and update the project's mindmodel, then debug why the trajectories plot is not showing in the Explorer app. |
||||||
|
|
||||||
|
## Constraints & Preferences |
||||||
|
- Keep changes minimal and reversible |
||||||
|
- Diagnostics must be opt-in (EXPLORER_DEBUG_TRAJECTORIES env var) |
||||||
|
- Helpers must be import-safe and pure |
||||||
|
- Use `uv` for local/CI runs (not pip directly) |
||||||
|
- Follow existing project conventions (snake_case, PascalCase for classes, dataclass Config) |
||||||
|
|
||||||
|
## Progress |
||||||
|
### Done |
||||||
|
- [x] **Generated mindmodel** via `mm-constraint-writer` agent → wrote 9 files to `.mindmodel/`: |
||||||
|
- `manifest.yaml`, `stack/stack.yaml`, `architecture/architecture.yaml`, `conventions/conventions.yaml`, `domain/domain-glossary.yaml`, `patterns/patterns.yaml`, `anti-patterns/anti-patterns.yaml`, `dependencies/dependencies.yaml`, `constraints/README.md` |
||||||
|
- Top anti-pattern: `explorer_helpers.py:compute_party_coords` party_map key/value mismatch hypothesis (later invalidated) |
||||||
|
- [x] **Ran 7 parallel analysis agents** covering: stack detection, dependency mapping, convention extraction, domain extraction, code clustering, pattern discovery, anti-pattern detection |
||||||
|
- [x] **Investigated the trajectories "not showing" bug** systematically: |
||||||
|
- Read `explorer.py` (2948 lines), `explorer_helpers.py` (297 lines), `analysis/political_axis.py` (695 lines), `analysis/trajectory.py` (297 lines), `analysis/visualize.py`, `scripts/diagnose_trajectories_cli.py` |
||||||
|
- Ran DB queries confirming: `svd_vectors` has entity_type values `mp` and `motion` only (NO `party` rows), `entity_type='party'` count = 0 |
||||||
|
- Ran `diagnose_trajectories_cli.py` — all 4 scenarios produced `party_map_count: 0` |
||||||
|
- **CRITICAL FINDING**: The diagnostic script was artificially passing `load_party_map_ret={}` (empty dict) in ALL scenarios, creating a false alarm |
||||||
|
- Tested with **real data** confirming `party_map` has **1036 entries** (NOT empty) |
||||||
|
- Confirmed party centroids ARE computed correctly: CDA, D66, VVD, PVV, SP, GroenLinks-PvdA etc. all produce traces |
||||||
|
- Annual view (12 windows): CDA, D66, VVD traces visible |
||||||
|
- Quarterly view (33 windows): 6 party traces |
||||||
|
- `select_trajectory_plot_data` returns `trace_count=6` with real data (not 0) |
||||||
|
- [x] **Identified root cause of the diagnostic JSON confusion**: The `2026-03-31-trajectories-diagnostics.json` was created by `diagnose_trajectories_cli.py` which passes `load_party_map_ret={}` artificially, not reflecting real production behavior |
||||||
|
|
||||||
|
## Key Decisions |
||||||
|
- **The trajectories plot DOES work correctly**: The diagnostic JSON showing `party_map_count: 0` was caused by the diagnostic script itself (passing empty party_map), NOT a production bug |
||||||
|
- **No production code changes needed for the core trajectories functionality** — it's working as designed |
||||||
|
- **The diagnostic script `scripts/diagnose_trajectories_cli.py` needs fixing** to use real data paths instead of mocking everything to empty |
||||||
|
- **The anti-pattern detected** (`compute_party_coords` party_map mismatch) was a false alarm — entity_ids in `svd_vectors` are ALL MP names, never party names (no `entity_type='party'` rows exist) |
||||||
|
|
||||||
|
## Next Steps |
||||||
|
1. **Fix `scripts/diagnose_trajectories_cli.py`** to use real data paths (`data/motions.db`) and real `load_party_map` / `load_positions` calls instead of mocking everything to empty |
||||||
|
2. **Re-run the fixed diagnostic script** to produce a correct `trajectories-diagnostics.json` artifact |
||||||
|
3. **Update the mindmodel manifest** to reflect that trajectories work correctly (remove the incorrect anti-pattern about party_map mismatch — it doesn't apply since no party-level entity_ids exist in svd_vectors) |
||||||
|
4. **Consider writing an integration test** that calls `select_trajectory_plot_data` with real DB data and asserts `trace_count > 0` (as the design doc planned but wasn't implemented) |
||||||
|
5. **Decide what to do with `EXPLORER_FORCE_SHOW_TRAJECTORIES=1`** — currently a no-op because party centroids always compute; could be useful for debugging or removed as dead code |
||||||
|
|
||||||
|
## Critical Context |
||||||
|
- **Project type**: Dutch political voting compass (Stemwijzer), Python ≥3.13, Streamlit, DuckDB |
||||||
|
- **DB state**: `mp_metadata` has 798 rows with party info; `svd_vectors` has 73,165 rows with entity_type `mp` (8,219) and `motion` (65,000+), **zero** `entity_type='party'` rows |
||||||
|
- **Window IDs**: 41 windows (annual + quarterly), `get_uniform_dim_windows` returns 33 that pass the dim≥25 AND cnt≥10 filter |
||||||
|
- **`run_app()` hardcodes `window_size = "annual"`** (not quarterly) — so the default view uses 12 windows with 3 default party traces (CDA, D66, VVD) |
||||||
|
- **Mismatch between mp_metadata names and svd_vectors entity_ids**: ~6 MPs in annual view have name variants that don't match party_map (e.g., `De Pater-Postma, W.L.` vs `Pater-Postma de, W.L.`), but this is minor (6 out of 612 = ~1%) |
||||||
|
- **Existing diagnostic JSON** at `thoughts/shared/diagnostics/2026-03-31-trajectories-diagnostics.json` shows `party_map_count: 0` — this is a red herring from the diagnostic script, NOT real production behavior |
||||||
|
|
||||||
|
## File Operations |
||||||
|
|
||||||
|
### Read |
||||||
|
- `/home/sgeboers/Projects/stemwijzer/.mindmodel/manifest.yaml` |
||||||
|
- `/home/sgeboers/Projects/stemwijzer/explorer.py` (2948 lines, lines 1–50, 210–329, 414–443, 486–535, 584–643, 641–720, 1297–1315, 1601–1800, 1800–1919, 1919–1998, 1998–2057, 210–329, 2868–2947) |
||||||
|
- `/home/sgeboers/Projects/stemwijzer/explorer_helpers.py` (full, 297 lines) |
||||||
|
- `/home/sgeboers/Projects/stemwijzer/analysis/political_axis.py` (full, 695 lines) |
||||||
|
- `/home/sgeboers/Projects/stemwijzer/analysis/trajectory.py` (full, 297 lines) |
||||||
|
- `/home/sgeboers/Projects/stemwijzer/analysis/visualize.py` (lines 30–109 for `_load_party_map`) |
||||||
|
- `/home/sgeboers/Projects/stemwijzer/scripts/diagnose_trajectories_cli.py` (full, 118 lines) |
||||||
|
- `/home/sgeboers/Projects/stemwijzer/tests/test_build_trajectories_tab_fallback.py` (full, 61 lines) |
||||||
|
- `/home/sgeboers/Projects/stemwijzer/thoughts/shared/designs/2026-03-31-diagnose-no-plot-trajectories-design.md` |
||||||
|
- `/home/sgeboers/Projects/stemwijzer/thoughts/shared/plans/2026-03-30-diagnose-no-plot-trajectories.md` |
||||||
|
|
||||||
|
### Modified |
||||||
|
- (none yet — verified trajectories work correctly via DB queries and Python tests) |
||||||
Loading…
Reference in new issue