Previously the st.plotly_chart call was wrapped in 'except Exception: pass'
which silently swallowed all rendering errors. The user would see no chart
and no error message.
Now:
- Exception message is shown via st.error()
- Diagnostics JSON is shown when debug is enabled (EXPLORER_DEBUG_TRAJECTORIES=1
or UI checkbox), even when trace_count > 0
This reveals the actual root cause when the chart fails to render.
- Lock x_label/y_label to Links-Rechts / Progressief-Conservatief after
classify_axes; Procrustes sign-fixing in compute_2d_axes already ensures
the correct orientation so the heuristic _should_swap_axes call is removed
- Remove visual error bars from party axis chart; 95% CI is now shown in
hover text (party: score, N=n, 95%-BI: [low, high]) to keep the 1D
scatter clean
- Remove show_ci checkbox and parameter — CI is always accessible on hover
- Update tests to match new hover format and absence of error_x
- Extract shared helper that both load_party_axis_scores and
load_party_mp_vectors delegate to, eliminating ~40 lines of
duplicated DB query + vector parsing code
- Remove dead code in load_party_axis_scores that queried mp_metadata
twice (first without ORDER BY, then again with ORDER BY, overwriting)
- Fix _cached_bootstrap_cis parameter: remove _ prefix so Streamlit
actually hashes the input dict instead of caching with no key
- Add load_party_mp_vectors() to return raw per-MP SVD vectors by party
- Extract _build_party_axis_figure() as pure function for testability
- Modify _render_party_axis_chart to accept bootstrap_data and delegate
to the new builder
- When bootstrap_data present: show error_x bars, diamond markers for
N=1 parties, and N=count in hover text
- Wire up bootstrap computation in build_svd_components_tab via cached
_cached_bootstrap_cis wrapper
- Add 6 tests covering figure construction, bootstrap rendering, flip
behavior, and importability
Use one DuckDB write connection for the entire update loop instead of
opening/closing per row, wrapped in try/finally for proper cleanup.
Move 'import duckdb' to module level with other imports.
Enable backfilling body_text for existing motions that lack it (2016-2018 data).
New extract_besluit_id() and update_existing_motions() helpers support the
--update-existing mode, while --no-skip-details enables detail fetching during
normal downloads. Includes 7 tests covering URL parsing, DB update flow, and
argparse wiring.
Move rng initialization before the party loop so each party gets a
unique segment of the random stream instead of identical sequences.
Replace Python bootstrap loop with vectorized numpy indexing.
Pure numpy function that computes bootstrap confidence intervals for
party centroid vectors. Handles N>=2 (bootstrap), N=1 (degenerate CI),
and N=0 (excluded) cases. Uses np.random.default_rng for reproducibility.
- PC2: rename 'maatschappelijke verantwoordelijkheid' to 'institutioneel
progressivisme' (less normatively loaded), rewrite explanation with actual
party scores (CU=-59, SGP=-25, VVD=-15 — all strongly negative, not
'near the middle'), update pole descriptions
- PC3: remove speculative motivation claim about PVV, state factual
observation that PVV/SP/PvdD/GL-PvdA vote alike despite opposing PC1
- PC7-PC10: add '(indicatief)' to labels — these axes explain <4% EVR
and may be below noise level
- PC7: add explicit fragility warning in explanation
- PC8: clarify DENK/SP negative scores mean active opposition voting,
not lack of focus; note Volt N=1 unreliability
- Scree plot: soften claim that later axes are 'meaningful'
Major corrections:
- Fix PC2 factual error: CU/CDA/SGP/D66 are strongly negative (-13 to -58), not near zero
- Correct methodology: party scores use single-window SVD, not Procrustes pipeline
- Correct centering: global (after stacking), not per-window
- Fix Groep Markuszower misclassification on PC4 (positive, not negative pool)
- Fix D66/PC4-PC5 cross-reference error
- Fix PC8/DENK interpretation (negative = voting against, not absence of focus)
Additions:
- Party sizes (N=) for all 17 parties across all axes
- Party size reliability table (D66=26 to Volt=1)
- All 5 flip values documented (PC3,4,7,9,10), not just PC3
- Vector-space mismatch table (single-window scores vs Procrustes EVR)
- Cautionary '(indicatief label)' on PC7-PC10
- New follow-up steps: bootstrap CIs, dimensionality testing, varimax, external validation
- Softened causal claims (kabinetscrisis correlation, PVV motivations)
- Less normatively loaded PC2 label
- Re-ran generate_svd_json.py for current_parliament window (100 rows, 10 components)
- Computed party centroid scores per axis from 150 matched MPs
- Updated all 10 SVD_THEMES entries with accurate labels, Dutch explanations
and correct positive/negative pole party attributions
- Key findings: PC1=rechts-links, PC2=populistisch nationalisme vs mainstream,
PC3=verzorgingsstaat vs bezuinigingen, PC6=klimaat & energie,
PC8=Europese defensie-integratie
- Added axis_analysis_data.json and party_svd_scores.json as analysis artifacts
Both _load_window_ids and _load_mp_vectors_for_window only read from the DB.
Opening without read_only=True caused an IOException when Streamlit already held
a read-only lock, silently returning an empty scree plot.
Previously load_scree_data computed L2-norms per dimension on current_parliament
vectors only, giving ~11% for PC1. This was inconsistent with the compass which
uses all windows + Procrustes alignment and gets PC1=24.1%.
Added compute_svd_spectrum() helper to political_axis.py that reuses the same
alignment pipeline. load_scree_data now delegates to it. _render_scree_plot
no longer re-normalizes (inputs are already EVR percentages). Hover label
updated to 'verklaarde variantie'.
Quarterly windows (29 of 41 total) diluted PC1 explained variance ratio
from ~20% down to ~14.6%. The fix splits the vector collection loop into:
- pca_vecs: annual windows only (re.match r'^\d{4}$') -> M_pca used for SVD
- all_vecs: every window -> M used for projections onto derived axes
Centering for SVD and global_mean for projection both now use M_pca.mean(axis=0)
so axes are consistent. Falls back to all windows if no annual windows exist.
Replaces static ideology CSV as primary axis classification signal with
per-year motion projection + Dutch keyword classifier. Adds axis-swap
logic so left-right is conventionally on X when present. Adds Option C
UI expander showing top motions per axis pole.
The global PCA X-axis flip uses centroids averaged across all windows,
which can leave individual windows with left/right inverted (e.g. PvdA
appearing right of VVD in 2020). Mirror the existing per-window Y-axis
correction to also check and flip X values per window.
Replace hardcoded 'Links-Rechts' / 'Progressief-Conservatief' axis labels
with values from classify_axes(). Add per-year interpretation caption when
axis quality score is below the 0.65 correlation threshold.
classify_axes() correlates per-party PCA positions against party_ideologies.csv
to assign honest dynamic labels (Links-Rechts, Coalitie-Oppositie, etc.)
instead of always assuming the first PCA axis is left-right.
Add design for honest PCA axis labeling — validates each compass axis
against a party ideology reference CSV and labels dynamically (Links–Rechts,
Coalitie–Oppositie, or fallback) instead of hardcoding Left–Right always.
The global orientation check using party centroids averaged across all
windows was insufficient — individual windows (notably 2023) could still
have conservative parties above progressive ones on the Y-axis.
Added a per-window flip in compute_2d_axes (PCA branch) that checks
prog_avg_y vs cons_avg_y for each window independently and negates all
Y values in that window when cons > prog. Flipped window IDs are stored
in axis_def['y_flipped_windows'] for diagnostics.
Moved the canonical party set definitions outside the orientation try-
block so they are always in scope for the per-window correction.
Added test_per_window_y_orientation to cover the case where one window
is globally fine but locally inverted.
- Rename app to 'Motief: de stematlas' in Home.py
- Remove PCA variance caption from compass tab
- Hardcode db_path and window_size; remove sidebar inputs
- Change trajectories default to [CDA, D66, VVD]
- Move quiz to pages/1_Stemwijzer.py; wrap in st.form
- Remove quiz tab from main explorer
- Add pytest dev dep + fix test fixtures (_load_mp_vectors_for_window)
- Add test_pca_axis_orientation with proper PCA variance dominance
- Replace gtfs/bokeh deploy with motief/streamlit (port 8501)
- Update inventory to motief.sgeboers.nl
- Remove stale .drone.yml
- Add CI guard to forbid .env in repo
- Add env removal report and secrets rotation checklist
The mp_votes table contains both party-aggregate rows (e.g. 'PVV', 'NSC')
and individual MP rows (e.g. 'Aardema, M.'). Running SVD on both together
creates a block-diagonal vote matrix where party codes and individual MPs
occupy disjoint SVD dimensions — causing dim 0 to be zero for all 421 MPs.
Fix: _build_expanded_rows() converts every party-level vote to individual
MP votes using mp_metadata date ranges (active MPs on motion date). Motions
that already have individual MP records are kept as-is. A party name mapping
handles NSC/Nieuw Sociaal Contract and other canonical name variants.
Results for current_parliament: 517 individual MPs, all 8732 motions covered,
dim 0 std=23.1 (was 0.0 for all MPs). PVV/NSC/BBB on positive end, SP/GL/PvdD
on negative end — matches expected left-right political axis.
All 11 annual windows (2016-2026) re-run with the new pipeline.
- load_party_axis_scores now loads individual MP vectors and averages
per party (same data source as the political compass), so SVD axis
rankings are consistent between the two tabs. Previously it used
party-aggregate rows which gave structurally 0 signal on dim 3.
- Re-add component 4 (dim 3) to SVD_THEMES and revert comp_options
filter — with individual MPs averaged, dim 3 now shows real party
separation (the 'publieke voorzieningen vs marktwerking' axis).
- Scree plot y-axis now shows percentage of total variance instead of
raw L2-norms; hover also updated to show '% van totaal'.
Dim 3 captures within-party individual disagreement (MPs splitting
from party lines). Party-aggregate votes are structurally 0 on this
dimension, so the axis chart showed all parties at ~0 with no
discrimination. Confirmed via dry-run: identical to existing data.
Also filter comp_options to only show components with a defined theme,
so component 4 is hidden from the selectbox entirely.
current_parliament has two separate SVD data spaces mixed together.
Party vectors (entity_id without comma) carry the between-party signal
in dims 0-15. Individual MP vectors only have signal in dim 3 and
dims 16-49 (within-party variance). The axis chart uses party vectors,
so the scree must too.
Previous query used DISTINCT ON without ordering by dim, picking arbitrary
(often non-50) dim per window. Rewritten to find the dominant dim per window
(highest count) and include only windows where dominant dim = 50 with >= 10
entities. This surfaces annual windows 2016/2018/2019/2022-2026 that were
previously excluded due to mixed-dim rows from multiple pipeline runs.
- load_positions annual mode now selects actual annual window_ids
('2022', '2023', etc.) instead of Q4 quarterly approximations,
with current_parliament appended as the most-recent anchor
- Sidebar radio defaults to 'Per jaar' (annual) instead of quarterly
- Dutch labels for window size radio: 'Per jaar' / 'Per kwartaal'
- Add import re (needed for strip_paren deduplication)
- Deduplicate MPs with parenthetical first-name variants (e.g. 'Dijk, J.P.'
and 'Dijk, J.P. (Jimmy)') by stripping parens and averaging positions;
fixes 22 duplicate groups across all windows
- Replace select_slider with selectbox for time window control
- Remove non-functional 'Toon namen' checkbox and its usage
- Remove 'Min. MPs per partij' slider and party-size filter
- Add 'Weergave' radio toggle: Kamerleden (individual MPs) vs Partijen
(party centroids computed as mean x/y per party)
- Fix axis ranges: x [-1, 1], y [-0.6, 0.6]
- Party view shows party abbreviation as dot label and hover shows MP count
- Use all individual MPs (not party aggregates) for L2-norm computation;
party-aggregated vectors have near-zero values on some dims due to
Procrustes alignment, producing spurious zeros
- Sort importances descending so scree plot is properly monotonic
- Relabel x-axis as 'Rang' since dim ordering after Procrustes alignment
no longer matches original singular value order
- Add Scatter line trace connecting bar tops for elbow visibility
- Add load_scree_data() cached loader computing L2-norm of party scores
per SVD dimension as a proxy for component importance
- Add _render_scree_plot() rendering a bar chart of the first 15 components
- Insert scree plot + Dutch explanation at the top of build_svd_components_tab
- Clean up _render_party_axis_chart: remove tick numbers, axis line, grid,
and zero-line from the x-axis (pole labels remain as chart title)
Cleanup performed by assistant: removed generated caches and stale files: __pycache__, *.pyc, .pytest_cache, .ruff_cache, dummy/, test.py, read.py, reset.py, fix_database.py, thoughts/thoughts/, .github/workflows/mindmodel-validate.yml. No push performed.
Adds new SVD window 'current_parliament' covering 8732 motions and 451 MPs
(vs 7424 motions in old '2025' window, adding ~1300 motions from 2025-Q4+).
Updates explorer.py to query the new window. Regenerates top_svd_top_motions.json.
Also clarifies axis 3 explanation noting FVD's anti-American positioning.
Removes the raw_title[:80] cap on expander labels so full titles show.
Adds scripts/generate_svd_json.py to regenerate top_svd_top_motions.json
from any SVD window after a recompute.
Axes 4 and 5 had inverted sign conventions relative to actual party votes.
Diagnostic confirmed SP/PvdD scored negative on axis 4 (free trade motions)
and FVD scored negative on axis 5 (secular motions), opposite to their
voting behaviour. Fix: swap positive_pole/negative_pole for both axes and
set correct flip direction so progressive parties appear on the left.
- Add ChristenUnie colour alias and CURRENT_PARLIAMENT_PARTIES frozenset (15 parties)
- Add load_party_axis_scores() — queries party SVD vectors from window=2025, cached
- Add _render_party_axis_chart() — 1D Plotly scatter of party positions per axis
- Restructure build_svd_components_tab: replace session-state button/detail-pane with
inline st.expander per motion, split into pos/neg pole columns, batch DB query for
all 10 motions including voting_results, rendered via _render_voting_results
Smoke-tested: 15 parties loaded, all 10 axis-1 motions returned with voting data.
sgeboers