Commit Graph
19 Commits
Author SHA1 Message Date
sgeboers 95c5ab9302 fix: generate interpretation string when motion path wins without ideology 2026-03-29 19:26:43 +02:00
sgeboers 1ff280e0e3 feat: restructure classify_axes — motion projection as primary label source 2026-03-29 19:20:56 +02:00
sgeboers 62daad321e fix: add outer exception handling to motion helpers in axis_classifier 2026-03-29 19:18:33 +02:00
sgeboers 96224be6ee feat: add motion-loading helpers to axis_classifier 2026-03-29 19:16:57 +02:00
sgeboers 1e52a8a8cc fix: deterministic tie handling and regex matching in _classify_from_titles 2026-03-29 19:15:31 +02:00
sgeboers 71e4b68926 fix: correct docstring for _classify_from_titles return value 2026-03-29 15:00:08 +02:00
sgeboers f8d9af7d9d feat: add _classify_from_titles keyword classifier to axis_classifier 2026-03-29 14:57:19 +02:00
sgeboers 6c4dd81723 feat: expose global_mean in compute_2d_axes axes dict 2026-03-29 14:50:05 +02:00
sgeboers 392fd3afce fix: add per-window X-axis orientation correction
The global PCA X-axis flip uses centroids averaged across all windows,
which can leave individual windows with left/right inverted (e.g. PvdA
appearing right of VVD in 2020). Mirror the existing per-window Y-axis
correction to also check and flip X values per window.
2026-03-29 01:15:21 +01:00
sgeboers 5ec1f7af75 feat: add axis classifier with party ideology reference data
classify_axes() correlates per-party PCA positions against party_ideologies.csv
to assign honest dynamic labels (Links-Rechts, Coalitie-Oppositie, etc.)
instead of always assuming the first PCA axis is left-right.
2026-03-29 01:00:55 +01:00
sgeboers 064cd059d4 fix: per-window Y-axis correction for political compass
The global orientation check using party centroids averaged across all
windows was insufficient — individual windows (notably 2023) could still
have conservative parties above progressive ones on the Y-axis.

Added a per-window flip in compute_2d_axes (PCA branch) that checks
prog_avg_y vs cons_avg_y for each window independently and negates all
Y values in that window when cons > prog. Flipped window IDs are stored
in axis_def['y_flipped_windows'] for diagnostics.

Moved the canonical party set definitions outside the orientation try-
block so they are always in scope for the per-window correction.

Added test_per_window_y_orientation to cover the case where one window
is globally fine but locally inverted.
2026-03-28 22:45:40 +01:00
sgeboers b5c14d0c65 deploy to server 2026-03-26 22:52:11 +01:00
sgeboers daa22c5e2b feat: complete parliamentary embedding pipeline with full historical coverage
- Add fused (SVD + text) embedding pipeline for annual windows 2016-2026
- Fix store_fused_embedding duplicate bug: DELETE before INSERT (idempotent)
- Add --text-batch-size CLI flag to run_pipeline.py (default 200)
- Add explicit --start-date/--end-date to download_past_year.py
- Backfill mp_votes for all motions (party-level votes, 111k new rows)
- Add similarity cache recompute: 212k rows across 9 annual windows
- Improve ai_provider retry logic, text_pipeline batching
- Improve analysis/political_axis PCA handling and visualizations
- Add diagnostic/utility scripts: compare_svd, generate_compass, inspect_axis, etc.
- Untrack data/motions.db (3.6GB binary), add to .gitignore with outputs/
- Update continuity ledger with full session state
2026-03-22 16:08:06 +01:00
sgeboers bf68e48460 fix(analysis): improve PCA handling when PC1 dominates, add pca_residual option and plot autoscaling/variance annotation 2026-03-22 00:13:56 +01:00
sgeboers 23a1234314 feat(analysis): add 2D political compass (PCA/anchor) and 2D trajectories + visualizations
- compute_2d_axes (pca, anchor) with optional L2-normalisation pre-projection
- compute_2d_trajectories: per-MP coords, step vectors, magnitudes, totals
- plot_political_compass and plot_2d_trajectories (Plotly HTML)
- tests/test_political_compass.py (synthetic unit test)
2026-03-21 23:58:38 +01:00
sgeboers 3551a82f83 feat(analysis): add 2D political compass and 2D trajectories
- compute_2d_axes (PCA + anchor)
- compute_2d_trajectories
- plot_political_compass, plot_2d_trajectories
- unit test: tests/test_political_compass.py
2026-03-21 23:53:55 +01:00
sgeboers f7d806dc3a fix(analysis): add Procrustes alignment and normalize vectors for drift computation
SVD sign/rotation is arbitrary per window. Without alignment, drift was
dominated by basis flips (~1.9/step max=2.0) rather than real political movement.

- _procrustes_align_windows(): aligns each window to the previous using
  orthogonal Procrustes on common entities (scipy, falls back gracefully)
- compute_trajectories(): builds aligned window dict before per-MP drift calc,
  adds normalize=True (L2-normalise) to remove cross-window magnitude differences
  caused by varying numbers of motions per quarter
- Results now in sensible range: NSC=2.28, DENK=1.90, ... PVV=0.82, FVD=0.70
- NSC large late jump (1.39 in Q4→Q1 2026) matches its parliamentary fracture
- Add outputs/trajectories_party_aligned.html with cleaned-up drift chart
2026-03-21 23:37:20 +01:00
sgeboers aa2f66ac9f feat(analysis): fetch real MP metadata, fix anchor axis for party-level actors
- fetch_mp_metadata: use real OData URL with pagination (1200 records, 5 pages)
  uses Fractie.Afkorting not NaamNL for abbreviation matching
  skips Verwijderd=true records
- upsert_mp_metadata: keep most recent membership (prefer active over ended,
  then higher Van date) so current party affiliations are not overwritten by historical
- compute_anchor_axis: anchor directly on party-level SVD entities (GroenLinks-PvdA etc)
  before falling back to mp_metadata individual MP lookup
- test_fetch_mp_metadata: fix mock for timeout kwarg + pagination + Afkorting field
- Generated anchor axis HTML for 2025-Q2 through 2026-Q1 in outputs/
2026-03-21 23:33:47 +01:00
sgeboers f2a831dfcf feat(pipeline): add orchestrator CLI, analysis modules, and ActorFractie ingestion
- pipeline/run_pipeline.py: CLI orchestrator for all 5 pipeline phases with
  --dry-run, --skip-*, --window-size, --svd-k, --start/end-date flags
- analysis/{political_axis,trajectory,clustering,visualize}.py: PCA/anchor
  ideological axis, MP drift trajectories, UMAP + KMeans clustering, Plotly HTML output
- api_client.py: capture ActorFractie per individual MP vote (comma in ActorNaam)
  into mp_vote_parties dict on each motion
- database.insert_motion: auto-insert mp_votes rows with party affiliation for
  newly ingested motions when mp_vote_parties is present
- Add scikit-learn to pyproject.toml for KMeans clustering
- tests/test_run_pipeline.py: window generation, dry-run, skip-all paths
- tests/test_analysis.py: PCA axis, anchor axis, trajectory drift, KMeans

Ref: thoughts/shared/plans/2026-03-21-parliamentary-embedding-pipeline-plan.md
2026-03-21 22:40:28 +01:00