- Added --pool-size argument (default 50) to control pool size
- Pool mode is now default; use --no-exclusive for old behavior
- Algorithm: for each component, claim top 5 positive + 5 negative from pool
- All 10 SVD components now have exactly 10 representative motions
Also removes tests that require missing dependencies (sklearn, plotly) or
missing files (.mindmodel/manifest.yaml):
- tests/mindmodel/ (2 files)
- tests/test_diagnose_no_plot_trajectories.py
- tests/test_explorer_chart.py
- tests/test_motion_drift.py
- tests/test_trajectories_pipeline_integration.py
- tests/test_trajectory_*.py (4 files)
Refs: thoughts/shared/plans/2026-04-12-svd-axis-label-alignment.md
Previously, components 1-2 in the SVD tab used Procrustes-aligned PCA
coordinates (from load_positions), which meant the SVD tab showed PCA
dimensions of the 50D aligned space rather than the actual raw SVD
components. This was a fundamental inconsistency — the SVD tab's component 2
showed completely different party ordering than the raw SVD component 2.
Changes:
- explorer.py: Unified all components 1-10 to use raw SVD values via
load_party_axis_scores_for_window(). Removed the separate
load_positions() path for components 1-2. Now all components use the
same data source (50D vectors from svd_vectors table).
- explorer.py: Updated flip computation to cover ALL components 1-10
(was range 3-11 for components 3-10 only). The compute_flip_direction
function correctly determines sign for each component.
- explorer.py: Unified rendering to always use _render_party_axis_chart_1d
(was _render_party_axis_chart for components 1-2 using 2D coords).
- explorer.py: Unified trajectory to always use load_party_scores_all_windows.
- analysis/config.py: Updated component 1 label (simplified explanation,
removed coalition-specific policy references).
- analysis/config.py: Updated component 2 label to "Nationalistisch versus
kosmopolitisch" matching raw SVD data (PVV/FVD at positive extreme,
Volt/DENK/GL-PvdA at negative extreme).
- tests: Updated test assertions to match new labels.
- scripts/validate_svd_themes.py: Verified all components pass right-wing
alignment check, config flip consistency, and theme pole consistency.
Fixes the core inconsistency: SVD tab component 2 now uses the same raw
SVD data as components 3-10, with consistent party ordering and labels.
The compass remains a separate PCA-based visualization.
Remove the '🔍 Wat bepaalt deze assen?' expander that showed individual
motion titles (➕/➖) with axis labels and variance explanation. Only the
Stemdiscipline analyse (Rice index) section remains. Also removes the
now-unused _render_axis_motions() helper function.
Delete 17 malformed YAML constraint files and 10 stale numbered
constraint files. Convert domain glossary, patterns, stack, and
anti-patterns to markdown format. Update manifest.yaml to reference
new markdown files.
Add four test files covering:
- test_config.py: SVD_THEMES structure validation
- test_explorer_labels.py: label derivation from positive/negative poles and flip
- test_svd_axis_alignment.py: right-wing centroid on RIGHT side for all axes
- test_validate_svd_themes.py: theme validation script tests
Allow analysis modules to be imported in lightweight test environments
without duckdb installed. Modules that need duckdb for actual queries
still require it at runtime, but import-time failures are handled gracefully.
Two related bugs fixed:
1. Label alignment: Removed static left_pole/right_pole from SVD_THEMES
entries. These labels assumed a fixed flip direction but could mismatch
with runtime flip computation, causing right-wing parties to appear on
the wrong side. Labels are now always derived from positive_pole,
negative_pole, and the runtime flip direction.
2. Score mismatch: Changed tijdtraject view for components 3-10 from
load_party_scores_all_windows_aligned() to load_party_scores_all_windows().
Procrustes alignment rotates the full 50-dim vector space to align
components 1-2, but this also transforms components 3-10, making their
scores incomparable with the single-window view. Per-window flip
computation already handles orientation alignment for these components.
Also updated svd_labels.py to prefer analysis.config as the canonical
source for SVD_THEMES, falling back to explorer only when config is
unavailable.
Key findings:
- Coalition started losing votes structurally from 2019
- Not that 'right' won, but that government lost
- Added government_win_rate.png visualization
- Updated analysis with party vote counts
- Add polarization_analysis.png: spread over time for all axes
- Add axis1_deep_dive.png: focus on Axis 1 (coalition vs opposition)
- Add Dutch blog post on parliamentary polarization findings
- Add script to find motions closest to semantic gravity per axis/window
- Document Axis 1 semantic shift: from administrative law (2016)
to migration/asylum policy (2026)
- Shows that 'coalition' votes on different topics over time
- Implement SVD axis stability using Lasso regression on fused embeddings
- Add overtone shift analysis to detect semantic content changes
- Implement semantic drift tracking for motion content over time
- Add party voting analysis with cross-ideological voting patterns
- Generate markdown report with visualizations
- Add comprehensive test suite with 12 passing tests
See reports/drift/report.md for analysis results.
- Add compute_overtone_shift(): tracks semantic gravity movement across windows
even when party ordering stays the same
- Update _generate_report() with overtone shift section including dimension-level
analysis and inflection point detection
- Update methodology section to reflect new metrics
- All 12 tests pass
Key finding: no axes exceed 0.7 stability threshold — semantic features
defining each SVD axis shift significantly across windows (0.06-0.51 range)
- Replace Procrustes-based stability with Ridge regression on fused embeddings
- For each SVD axis, fit Ridge: SVD_score ~ fused_embedding per window
- Compare weight vectors via max(cosine similarity, Jaccard top-100)
- Add --regression-alpha CLI argument (default 1.0)
- Keep party-based fallback for windows with < 50 motions
- Update tests for new regression-based approach
Key finding: regression weights show moderate stability (0.06-0.51)
but no axes exceed 0.7 threshold — semantic features defining each
axis shift significantly across windows
- Add scripts/motion_drift.py: analyzes SVD axis stability, semantic drift,
and cross-ideological voting patterns across annual windows
- Add analysis/motion_drift.py: core analysis functions with Procrustes
alignment fallback using party-based sign consistency
- Add matplotlib dependency for static chart generation
- Add tests/test_motion_drift.py: 12 tests covering all analysis functions
- Report output: markdown with embedded PNG charts
Key findings from real data:
- No axes are fully stable (>0.7) across 2019-2026
- All axes show moderate consistency (0.40-0.47) — stable within periods
but flip between cabinet periods (2019/2022/2026 vs 2023/2024/2025)
- Party voting analysis detects cross-ideological voting patterns
- refactoring-streamlit-data-loading.md: update test count
164/164 → 173/173 (7 new axis validation tests added)
- svd-component-labels-mismatch.md: SVD_THEMES moved from
explorer.py:434-611 → analysis/config.py:67+ per the
refactoring that extracted constants to analysis/config.py
- Add CANONICAL_RIGHT (PVV, FVD, JA21, SGP) and CANONICAL_LEFT frozensets
to analysis/config.py as the canonical source of truth
- Update analysis/svd_labels.py to import from config; re-export as
RIGHT_PARTIES/LEFT_PARTIES for backward compatibility
- Add build_window_party_scores helper to analysis/explorer_data.py
- Add 7 integration tests in tests/test_axis_political_orientation.py
validating that canonical right parties appear on the right side of SVD
axes (x=component 1, y=component 2) using real DuckDB data
- Add AGENTS.md with documented solutions reference
- Include SVD label convention (right-wing parties on right side)
- Document SVD insight: labels reflect voting patterns, not semantics
- Fix SQL verification example to use Python approach
The component captures voting unity of the right-wing coalition vs left
opposition, NOT semantic content like 'defense' or 'EU integration'.
Motions about elderly care (Dobbe) appear because the left votes for them
while the right coalition votes against - this is coalition-opposition
polarization, not policy domain.
Bug: report_per_component used scored[:args.report_top_n] which took
top N by score (all positive for components with only positive scores).
JSON correctly separated positive and negative poles.
Fix: Use same positive/negative separation logic for report as JSON.
- Each motion now assigned to exactly one component (highest absolute score)
- Added --exclusive flag (default: True) for backward compatibility
- Added markdown report generation with motion details for label review
- Added --report-top-n for report size (default: 20 per component)
- Updated JSON output with 'exclusive' flag for transparency
- Add Dutch paragraph explaining Rice index and party discipline patterns
- Analysis covers high discipline parties (PVV, SGP) vs lower discipline parties
- Explains what discipline reveals about party dynamics
- Add _load_mp_vectors_by_party_for_window() to load SVD vectors for specific windows
- Add load_party_axis_scores_for_window() cached function
- Add year selector UI for components 3-10 similar to components 1-2
- Uses get_uniform_dim_windows() to get available windows
- Changed _render_party_axis_chart_1d from horizontal bar chart to scatter plot
- Same format as components 1-2: markers on horizontal line with axis arrows- Axis labels now show correct direction with arrows (← left | right →)
- Ensures consistent visualization across all SVD components
sgeboers