Commit Graph
173 Commits
Author SHA1 Message Date
sgeboers 0308d20f12 docs: add AGENTS.md with docs/solutions reference and SVD label best practice
- Add AGENTS.md with documented solutions reference
- Include SVD label convention (right-wing parties on right side)
- Document SVD insight: labels reflect voting patterns, not semantics
- Fix SQL verification example to use Python approach
2026-04-04 18:36:05 +02:00
sgeboers 92c3c0ee01 fix: update Components 2, 4, 5, 6 SVD labels based on voting pattern analysis 2026-04-04 18:29:48 +02:00
sgeboers f7fc908b58 fix: update Component 1 label to coalition-opposition reality
The component captures voting unity of the right-wing coalition vs left
opposition, NOT semantic content like 'defense' or 'EU integration'.
Motions about elderly care (Dobbe) appear because the left votes for them
while the right coalition votes against - this is coalition-opposition
polarization, not policy domain.
2026-04-04 18:25:19 +02:00
sgeboers bfe37c6806 fix: align report generation with JSON output for positive/negative separation
Bug: report_per_component used scored[:args.report_top_n] which took
top N by score (all positive for components with only positive scores).
JSON correctly separated positive and negative poles.

Fix: Use same positive/negative separation logic for report as JSON.
2026-04-04 18:19:31 +02:00
sgeboers e77f0ec9e3 fix: update SVD_THEMES labels to match actual motion content
- Component 1: EU-integratie → Defensie-uitgaven en NAVO-commitment
- Component 2: Flip False (was True), clarify populistisch vs institutioneel
- Component 4: Pragmatisch centrisme → Gematigde middenpartijen
- Component 5: Christelijk-sociaal → Gemeenschapszin en sociale zekerheid
- Component 6: Add migratie-integratie to label
- Component 8: Complete rewrite - was COMPLETELY WRONG (vaccinatie, onderwijs)
- Component 9: Decentraal bestuur → Pragmatische probleemoplossing
- Component 10: Institutioneel toezicht → Kritisch op overheidsbemoeienis
2026-04-04 18:09:46 +02:00
sgeboers 33edb334c4 feat: implement exclusive SVD motion assignment with label review report
- Each motion now assigned to exactly one component (highest absolute score)
- Added --exclusive flag (default: True) for backward compatibility
- Added markdown report generation with motion details for label review
- Added --report-top-n for report size (default: 20 per component)
- Updated JSON output with 'exclusive' flag for transparency
2026-04-04 17:55:28 +02:00
sgeboers ee8ffea6e2 fix: add health check wait to ansible deploy
Wait for Streamlit to be ready before finishing deployment to prevent 502 errors
2026-04-02 22:46:01 +02:00
sgeboers d8bee43c15 feat: add voting discipline analysis paragraph under political compass
- Add Dutch paragraph explaining Rice index and party discipline patterns
- Analysis covers high discipline parties (PVV, SGP) vs lower discipline parties
- Explains what discipline reveals about party dynamics
2026-04-02 22:03:57 +02:00
sgeboers f5f0c8d6b1 feat: add year selector for SVD components 3-10
- Add _load_mp_vectors_by_party_for_window() to load SVD vectors for specific windows
- Add load_party_axis_scores_for_window() cached function
- Add year selector UI for components 3-10 similar to components 1-2
- Uses get_uniform_dim_windows() to get available windows
2026-04-02 21:49:12 +02:00
sgeboers 5f7126f53f docs: add voting discipline analysis
Explain Rice index methodology and what it reveals about Dutch political parties
2026-04-02 21:47:08 +02:00
sgeboers abd3281044 refactor: remove Stemgedrag cohesie section and fallback axis message
- Remove voting discipline (cohesie) section from Political Compass tab
- Remove 'empirisch stempatroon zonder duidelijke ideologische richting' fallback message from axis classifier
- Clean up unused fallback template from _INTERPRETATION_TEMPLATES
2026-04-02 21:46:24 +02:00
sgeboers a5e95c33d7 refactor: use scatter plot format for SVD components 3-10
- Changed _render_party_axis_chart_1d from horizontal bar chart to scatter plot
- Same format as components 1-2: markers on horizontal line with axis arrows- Axis labels now show correct direction with arrows (← left | right →)
- Ensures consistent visualization across all SVD components
2026-04-02 21:39:09 +02:00
sgeboers fa019d8a9c test: add test for auto-flip computation for all components 2026-04-02 21:21:58 +02:00
sgeboers ed2b4c1fae test: add tests for 1D party position chart 2026-04-02 21:19:23 +02:00
sgeboers 95183fec5b test: update tests for unified SVD label system (Task 7) 2026-04-02 21:08:50 +02:00
sgeboers ba24ad4fe6 feat: auto-compute flip directions for all SVD components (Task 6) 2026-04-02 21:07:46 +02:00
sgeboers bda803089a feat: add 1D party position charts for SVD components 3-10 (Task 5) 2026-04-02 21:07:11 +02:00
sgeboers 5b3cf23d36 refactor: use svd_labels for fallback labels in explorer and axis_classifier (Task 4) 2026-04-02 21:05:30 +02:00
sgeboers 36b58ad50d refactor: use svd_labels module for fallback labels in axis_classifier (Task 3) 2026-04-02 21:02:53 +02:00
sgeboers 5b1be26050 refactor: move SVD_THEMES to module level for import (Task 2) 2026-04-02 21:02:03 +02:00
sgeboers a1c3e92fab docs: add SVD label unification implementation plan 2026-04-02 20:45:44 +02:00
sgeboers bed776f295 docs: add SVD label unification design spec 2026-04-02 20:40:56 +02:00
sgeboers c9c59dd166 feat(diagnostics): enhance trajectory diagnostic script with real data mode 2026-04-01 01:59:41 +02:00
sgeboers 7e202e15be test(trajectory): fix test quality issues 2026-04-01 01:57:59 +02:00
sgeboers 8bc43b67fd test(trajectory): add tests for plot rendering with edge cases 2026-04-01 01:56:04 +02:00
sgeboers 31e1dd4371 fix(trajectory): correct import for diagnose_trajectories 2026-04-01 01:51:14 +02:00
sgeboers 5cd031777c fix(trajectory): improve fallback handling and diagnostics when trace_count is 0 2026-04-01 01:49:14 +02:00
sgeboers 8e67b89a1d fix(trajectory): fix division by zero and None handling in name normalization 2026-04-01 01:46:13 +02:00
sgeboers 0b79709847 fix(trajectory): normalize MP names to improve party_map matching 2026-04-01 01:43:39 +02:00
sgeboers 26bdb4c61c refactor(trajectory): fix code quality issues in centroid diagnostics 2026-04-01 01:40:40 +02:00
sgeboers 7d93753530 fix(trajectory): add diagnostics to compute_party_centroids for NaN detection 2026-04-01 01:35:57 +02:00
sgeboers 385a25853c diagnose(trajectory): add diagnostics to identify why trace_count is 0 2026-04-01 01:31:49 +02:00
sgeboers 24796f97d3 test: add trajectory pipeline integration test 2026-03-31 15:08:50 +02:00
sgeboers 69208e0bf6 fix: skip second trace loop when helper succeeds to avoid duplicate traces 2026-03-31 15:04:45 +02:00
sgeboers 5d1328f824 chore: add TRAJ DEBUG print checkpoints to build_trajectories_tab 2026-03-31 14:55:08 +02:00
sgeboers 1a83f0f319 docs: add trajectory plots debugging plan 2026-03-31 02:12:49 +02:00
sgeboers 9f98dbae60 Add debug st.info before st.plotly_chart to diagnose invisible chart 2026-03-31 01:49:38 +02:00
sgeboers 72d1c20340 Show error and diagnostics when st.plotly_chart fails instead of silent pass
Previously the st.plotly_chart call was wrapped in 'except Exception: pass'
which silently swallowed all rendering errors. The user would see no chart
and no error message.

Now:
- Exception message is shown via st.error()
- Diagnostics JSON is shown when debug is enabled (EXPLORER_DEBUG_TRAJECTORIES=1
  or UI checkbox), even when trace_count > 0

This reveals the actual root cause when the chart fails to render.
2026-03-31 01:38:30 +02:00
sgeboers baee50f3a5 feat(explorer): extend diagnostic inspector to surface mp samples/counts
chore(explorer): add get_debug_trajectories_enabled helper

feat(explorer): instrument trajectories with debug diagnostics and un-silence helper exceptions
2026-03-31 00:28:14 +02:00
sgeboers 0f2db0a9be chore(explorer): add get_debug_trajectories_enabled helper 2026-03-31 00:22:26 +02:00
sgeboers 525cd157c0 docs: add diagnose-no-plot-trajectories design (2026-03-30) 2026-03-30 23:57:18 +02:00
sgeboers ce1fc86bcb docs(design): add fix-missing-trajectories design 2026-03-30 22:24:05 +02:00
sgeboers c059d5d955 Fix compass orientation and simplify CI display
- Lock x_label/y_label to Links-Rechts / Progressief-Conservatief after
  classify_axes; Procrustes sign-fixing in compute_2d_axes already ensures
  the correct orientation so the heuristic _should_swap_axes call is removed
- Remove visual error bars from party axis chart; 95% CI is now shown in
  hover text (party: score, N=n, 95%-BI: [low, high]) to keep the 1D
  scatter clean
- Remove show_ci checkbox and parameter — CI is always accessible on hover
- Update tests to match new hover format and absence of error_x
2026-03-30 18:31:39 +02:00
sgeboers b7129b3755 Extract _load_mp_vectors_by_party helper and fix cache key
- Extract shared helper that both load_party_axis_scores and
  load_party_mp_vectors delegate to, eliminating ~40 lines of
  duplicated DB query + vector parsing code
- Remove dead code in load_party_axis_scores that queried mp_metadata
  twice (first without ORDER BY, then again with ORDER BY, overwriting)
- Fix _cached_bootstrap_cis parameter: remove _ prefix so Streamlit
  actually hashes the input dict instead of caching with no key
2026-03-29 23:41:15 +02:00
sgeboers 3938eecc53 Add bootstrap CIs to party axis chart with error bars and diamond markers
- Add load_party_mp_vectors() to return raw per-MP SVD vectors by party
- Extract _build_party_axis_figure() as pure function for testability
- Modify _render_party_axis_chart to accept bootstrap_data and delegate
  to the new builder
- When bootstrap_data present: show error_x bars, diamond markers for
  N=1 parties, and N=count in hover text
- Wire up bootstrap computation in build_svd_components_tab via cached
  _cached_bootstrap_cis wrapper
- Add 6 tests covering figure construction, bootstrap rendering, flip
  behavior, and importability
2026-03-29 23:35:53 +02:00
sgeboers 88110b0aaa Fix update_existing_motions: single write connection and module-level duckdb import
Use one DuckDB write connection for the entire update loop instead of
opening/closing per row, wrapped in try/finally for proper cleanup.
Move 'import duckdb' to module level with other imports.
2026-03-29 23:28:40 +02:00
sgeboers be8887f6f8 Add --skip-details, --update-existing flags to download_past_year.py with tests
Enable backfilling body_text for existing motions that lack it (2016-2018 data).
New extract_besluit_id() and update_existing_motions() helpers support the
--update-existing mode, while --no-skip-details enables detail fetching during
normal downloads. Includes 7 tests covering URL parsing, DB update flow, and
argparse wiring.
2026-03-29 23:25:04 +02:00
sgeboers 72a8dd2721 Fix RNG re-seeding per party and vectorize bootstrap loop
Move rng initialization before the party loop so each party gets a
unique segment of the random stream instead of identical sequences.
Replace Python bootstrap loop with vectorized numpy indexing.
2026-03-29 23:17:33 +02:00
sgeboers cd8aeec997 Add compute_party_bootstrap_cis() to political_axis.py with tests
Pure numpy function that computes bootstrap confidence intervals for
party centroid vectors. Handles N>=2 (bootstrap), N=1 (degenerate CI),
and N=0 (excluded) cases. Uses np.random.default_rng for reproducibility.
2026-03-29 23:13:30 +02:00
sgeboers ef96edf478 Remove stale ad-hoc JSON analysis files
party_svd_scores.json was missing NSC and axis_analysis_data.json was
unused by the app. Both were one-off analysis artifacts.
2026-03-29 23:09:37 +02:00