sgeboers/motief
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
207 Commits 1 Branch 0 Tags
823df6f9ee8549763d5aaa86fa70015d35ff9a5d
Commit Graph
207 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
sgeboers 5ad83ef1be fix(tests): update test_extract_mp_votes for party-level actor inclusion 
New extract_mp_votes behavior inserts all actors (party + individual MPs),
not only comma-name MPs. Test now validates both types and their party column.
Also adds generated HTML visualizations (political axis x5 windows + trajectories).
 2026-03-21 23:25:27 +01:00
sgeboers 847b783877 fix(pipeline): fix API pagination, add skip_details fast path, bulk mp_votes insert 
- _get_voting_records returns (records, besluit_meta) tuple; paginate via Besluit?expand=Stemming (469/mo vs 8400)
- get_motions(skip_details=True) bypasses per-motion detail chain (3 HTTP calls/motion)
- extract_mp_votes rewritten: bulk DataFrame insert (80k rows in 1.9s), includes party-level actors
- run_pipeline.py fixed: pass db_path not db, handle dict/int return types
- download_past_year.py: skip_details=True default, limit-per-chunk default 50000
 2026-03-21 23:24:06 +01:00
sgeboers f2a831dfcf feat(pipeline): add orchestrator CLI, analysis modules, and ActorFractie ingestion 
- pipeline/run_pipeline.py: CLI orchestrator for all 5 pipeline phases with
  --dry-run, --skip-*, --window-size, --svd-k, --start/end-date flags
- analysis/{political_axis,trajectory,clustering,visualize}.py: PCA/anchor
  ideological axis, MP drift trajectories, UMAP + KMeans clustering, Plotly HTML output
- api_client.py: capture ActorFractie per individual MP vote (comma in ActorNaam)
  into mp_vote_parties dict on each motion
- database.insert_motion: auto-insert mp_votes rows with party affiliation for
  newly ingested motions when mp_vote_parties is present
- Add scikit-learn to pyproject.toml for KMeans clustering
- tests/test_run_pipeline.py: window generation, dry-run, skip-all paths
- tests/test_analysis.py: PCA axis, anchor axis, trajectory drift, KMeans

Ref: thoughts/shared/plans/2026-03-21-parliamentary-embedding-pipeline-plan.md
 2026-03-21 22:40:28 +01:00
sgeboers a36e6cba4e feat(pipeline): implement parliamentary embedding pipeline MVP 
- Add 4 migration files: mp_votes, mp_metadata, svd_vectors, fused_embeddings
- Extend database.py with 5 new helper methods and table init
- Add pipeline/ package: extract_mp_votes, fetch_mp_metadata, text_pipeline,
  svd_pipeline (with Procrustes alignment), fusion
- Add full test suite (17 tests) covering all pipeline modules and migrations
- Fix Procrustes alignment bug: scipy scale is a norm value, not a multiplier
- Fix DuckDB date type handling in test assertions (datetime.date vs string)
- Remove duckdb.py shim; tests now run against real duckdb + scipy via uv

Ref: thoughts/shared/plans/2026-03-21-parliamentary-embedding-pipeline-plan.md
 2026-03-21 22:31:22 +01:00
sgeboers c498c3467e update plan: replace spike with confirmed FractieZetelPersoon fetch task 2026-03-21 22:05:56 +01:00
sgeboers 0bbda408fb plan: parliamentary embedding pipeline MVP implementation plan 2026-03-21 21:57:39 +01:00
sgeboers fd73da3752 design: parliamentary embedding pipeline (late fusion SVD + text) 2026-03-21 21:54:32 +01:00