The global PCA X-axis flip uses centroids averaged across all windows,
which can leave individual windows with left/right inverted (e.g. PvdA
appearing right of VVD in 2020). Mirror the existing per-window Y-axis
correction to also check and flip X values per window.
Replace hardcoded 'Links-Rechts' / 'Progressief-Conservatief' axis labels
with values from classify_axes(). Add per-year interpretation caption when
axis quality score is below the 0.65 correlation threshold.
classify_axes() correlates per-party PCA positions against party_ideologies.csv
to assign honest dynamic labels (Links-Rechts, Coalitie-Oppositie, etc.)
instead of always assuming the first PCA axis is left-right.
Add design for honest PCA axis labeling — validates each compass axis
against a party ideology reference CSV and labels dynamically (Links–Rechts,
Coalitie–Oppositie, or fallback) instead of hardcoding Left–Right always.
The global orientation check using party centroids averaged across all
windows was insufficient — individual windows (notably 2023) could still
have conservative parties above progressive ones on the Y-axis.
Added a per-window flip in compute_2d_axes (PCA branch) that checks
prog_avg_y vs cons_avg_y for each window independently and negates all
Y values in that window when cons > prog. Flipped window IDs are stored
in axis_def['y_flipped_windows'] for diagnostics.
Moved the canonical party set definitions outside the orientation try-
block so they are always in scope for the per-window correction.
Added test_per_window_y_orientation to cover the case where one window
is globally fine but locally inverted.
- Rename app to 'Motief: de stematlas' in Home.py
- Remove PCA variance caption from compass tab
- Hardcode db_path and window_size; remove sidebar inputs
- Change trajectories default to [CDA, D66, VVD]
- Move quiz to pages/1_Stemwijzer.py; wrap in st.form
- Remove quiz tab from main explorer
- Add pytest dev dep + fix test fixtures (_load_mp_vectors_for_window)
- Add test_pca_axis_orientation with proper PCA variance dominance
- Replace gtfs/bokeh deploy with motief/streamlit (port 8501)
- Update inventory to motief.sgeboers.nl
- Remove stale .drone.yml
- Add CI guard to forbid .env in repo
- Add env removal report and secrets rotation checklist
The mp_votes table contains both party-aggregate rows (e.g. 'PVV', 'NSC')
and individual MP rows (e.g. 'Aardema, M.'). Running SVD on both together
creates a block-diagonal vote matrix where party codes and individual MPs
occupy disjoint SVD dimensions — causing dim 0 to be zero for all 421 MPs.
Fix: _build_expanded_rows() converts every party-level vote to individual
MP votes using mp_metadata date ranges (active MPs on motion date). Motions
that already have individual MP records are kept as-is. A party name mapping
handles NSC/Nieuw Sociaal Contract and other canonical name variants.
Results for current_parliament: 517 individual MPs, all 8732 motions covered,
dim 0 std=23.1 (was 0.0 for all MPs). PVV/NSC/BBB on positive end, SP/GL/PvdD
on negative end — matches expected left-right political axis.
All 11 annual windows (2016-2026) re-run with the new pipeline.
- load_party_axis_scores now loads individual MP vectors and averages
per party (same data source as the political compass), so SVD axis
rankings are consistent between the two tabs. Previously it used
party-aggregate rows which gave structurally 0 signal on dim 3.
- Re-add component 4 (dim 3) to SVD_THEMES and revert comp_options
filter — with individual MPs averaged, dim 3 now shows real party
separation (the 'publieke voorzieningen vs marktwerking' axis).
- Scree plot y-axis now shows percentage of total variance instead of
raw L2-norms; hover also updated to show '% van totaal'.
Dim 3 captures within-party individual disagreement (MPs splitting
from party lines). Party-aggregate votes are structurally 0 on this
dimension, so the axis chart showed all parties at ~0 with no
discrimination. Confirmed via dry-run: identical to existing data.
Also filter comp_options to only show components with a defined theme,
so component 4 is hidden from the selectbox entirely.
current_parliament has two separate SVD data spaces mixed together.
Party vectors (entity_id without comma) carry the between-party signal
in dims 0-15. Individual MP vectors only have signal in dim 3 and
dims 16-49 (within-party variance). The axis chart uses party vectors,
so the scree must too.
Previous query used DISTINCT ON without ordering by dim, picking arbitrary
(often non-50) dim per window. Rewritten to find the dominant dim per window
(highest count) and include only windows where dominant dim = 50 with >= 10
entities. This surfaces annual windows 2016/2018/2019/2022-2026 that were
previously excluded due to mixed-dim rows from multiple pipeline runs.
- load_positions annual mode now selects actual annual window_ids
('2022', '2023', etc.) instead of Q4 quarterly approximations,
with current_parliament appended as the most-recent anchor
- Sidebar radio defaults to 'Per jaar' (annual) instead of quarterly
- Dutch labels for window size radio: 'Per jaar' / 'Per kwartaal'
- Add import re (needed for strip_paren deduplication)
- Deduplicate MPs with parenthetical first-name variants (e.g. 'Dijk, J.P.'
and 'Dijk, J.P. (Jimmy)') by stripping parens and averaging positions;
fixes 22 duplicate groups across all windows
- Replace select_slider with selectbox for time window control
- Remove non-functional 'Toon namen' checkbox and its usage
- Remove 'Min. MPs per partij' slider and party-size filter
- Add 'Weergave' radio toggle: Kamerleden (individual MPs) vs Partijen
(party centroids computed as mean x/y per party)
- Fix axis ranges: x [-1, 1], y [-0.6, 0.6]
- Party view shows party abbreviation as dot label and hover shows MP count
- Use all individual MPs (not party aggregates) for L2-norm computation;
party-aggregated vectors have near-zero values on some dims due to
Procrustes alignment, producing spurious zeros
- Sort importances descending so scree plot is properly monotonic
- Relabel x-axis as 'Rang' since dim ordering after Procrustes alignment
no longer matches original singular value order
- Add Scatter line trace connecting bar tops for elbow visibility
- Add load_scree_data() cached loader computing L2-norm of party scores
per SVD dimension as a proxy for component importance
- Add _render_scree_plot() rendering a bar chart of the first 15 components
- Insert scree plot + Dutch explanation at the top of build_svd_components_tab
- Clean up _render_party_axis_chart: remove tick numbers, axis line, grid,
and zero-line from the x-axis (pole labels remain as chart title)
Cleanup performed by assistant: removed generated caches and stale files: __pycache__, *.pyc, .pytest_cache, .ruff_cache, dummy/, test.py, read.py, reset.py, fix_database.py, thoughts/thoughts/, .github/workflows/mindmodel-validate.yml. No push performed.
Adds new SVD window 'current_parliament' covering 8732 motions and 451 MPs
(vs 7424 motions in old '2025' window, adding ~1300 motions from 2025-Q4+).
Updates explorer.py to query the new window. Regenerates top_svd_top_motions.json.
Also clarifies axis 3 explanation noting FVD's anti-American positioning.
Removes the raw_title[:80] cap on expander labels so full titles show.
Adds scripts/generate_svd_json.py to regenerate top_svd_top_motions.json
from any SVD window after a recompute.
Axes 4 and 5 had inverted sign conventions relative to actual party votes.
Diagnostic confirmed SP/PvdD scored negative on axis 4 (free trade motions)
and FVD scored negative on axis 5 (secular motions), opposite to their
voting behaviour. Fix: swap positive_pole/negative_pole for both axes and
set correct flip direction so progressive parties appear on the left.
- Add ChristenUnie colour alias and CURRENT_PARLIAMENT_PARTIES frozenset (15 parties)
- Add load_party_axis_scores() — queries party SVD vectors from window=2025, cached
- Add _render_party_axis_chart() — 1D Plotly scatter of party positions per axis
- Restructure build_svd_components_tab: replace session-state button/detail-pane with
inline st.expander per motion, split into pos/neg pole columns, batch DB query for
all 10 motions including voting_results, rendered via _render_voting_results
Smoke-tested: 15 parties loaded, all 10 axis-1 motions returned with voting data.
sgeboers