sgeboers/motief
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
240 Commits 1 Branch 0 Tags
203ae178ca50c53db37d3ce8d5bcdef6d94c2d3f
Commit Graph
240 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
sgeboers 540099f2b7 ci(mindmodel): add report-only mindmodel validation workflow 2026-03-24 22:41:33 +01:00
sgeboers d1faf2b3e4 feat(mindmodel): add CLI wrapper, edge-case tests, and manifest schema tests 2026-03-24 22:41:28 +01:00
sgeboers ed289ff582 chore(mindmodel): add sanitized read-only manifest and reviewer README 2026-03-24 22:41:24 +01:00
sgeboers f091846dc8 feat(mindmodel): add report-only validator skeleton, types, and tests 2026-03-24 22:41:15 +01:00
sgeboers dbd8cc801a docs(mindmodel): add README and tooling note; chore(tooling): add minimal pre-commit 2026-03-24 21:31:53 +01:00
sgeboers 5caebe1331 chore(ci): add mindmodel validator workflow 2026-03-24 21:30:45 +01:00
sgeboers f77875ed54 feat(mindmodel): add CLI wrapper and tests 2026-03-24 21:29:10 +01:00
sgeboers a74e6006f5 feat(mindmodel): add validator and tests 2026-03-24 21:28:17 +01:00
sgeboers 7bd7d0d18c feat(mindmodel): add checks utilities and tests 2026-03-24 21:26:38 +01:00
sgeboers 2efd7ba3a0 feat(mindmodel): add manifest loader and tests 2026-03-24 21:24:58 +01:00
sgeboers 9c82962d47 chore: add .mindmodel/ project constraints and conventions 
Generated mindmodel covering stack, architecture, domain glossary,
coding conventions, DuckDB/requests/embeddings/error-handling patterns,
anti-patterns, and explicit constraints for naming, imports, DB access,
error handling, and testing.
 2026-03-24 20:51:53 +01:00
sgeboers 504400faf2 fix: quiz seed from motions with actual individual MP votes 
Root causes:
- Seed selection sorted by controversy_score across all 28k motions, but
  only 282 have individual MP vote records. Top controversial motions only
  have party-level votes, so match_mps_for_votes always returned empty.
- global_db singleton was used for match/discriminate instead of the db_path
  passed to the tab builder.

Fixes:
- Add MotionDatabase.get_motions_with_individual_votes(k) which queries
  motions with comma-formatted mp_name votes, ordered by controversy_score
- Replace broken seed logic in build_mp_quiz_tab with this new method
- Replace global_db usages with a local MotionDatabase(db_path) instance
- Guard against motion IDs present in votes but absent from motions DataFrame
 2026-03-24 20:29:05 +01:00
sgeboers 238d9e9ec2 test: add full quiz tab test suite and fix Geen stem normalization 
- Fix match_mps_for_votes: 'Geen stem'/'no vote' now normalize to None (skipped),
  not 'afwezig' — so unanswered questions don't inflate overlap count
- Add 5 additional tests to test_match_mps.py: zero-overlap exclusion,
  empty input validation, Geen stem overlap skip, excluded motions respected
- Add tests/test_explorer_quiz.py: builder import smoke test plus 3 real-DB
  end-to-end scenarios (unique match, indistinguishable MPs, discriminating
  question reduces candidate set)
- Full suite: 73 passed, 2 skipped
 2026-03-24 20:18:20 +01:00
sgeboers eb73275f32 feat(mp-quiz): add MP quiz tab and DB helpers; add design and plan docs 2026-03-24 19:59:25 +01:00
sgeboers b09e580f65 feat: motion content enrichment pipeline hardening 
- ai_provider_wrapper: retry/fallback with exponential backoff, None sentinel for failed items
- text_pipeline: use wrapper, return 5-tuple (stored, skipped_existing, skipped_no_text, errors, failed_ids)
- similarity/compute: filter trivial 1.0 matches on identical short titles (<12 chars)
- rerun_embeddings: --retry-missing mode, calls ensure_text_embeddings_for_ids on failed ids
- sync_motion_content: per-ext_id retries, HTTPAdapter pool, --max-body-workers CLI flag, audit on failure
- qa_similarity script: samples motions, writes JSON ledger to thoughts/ledgers/
- All tests green: 61 passed, 2 skipped
 2026-03-23 21:31:39 +01:00
sgeboers aef7c45074 Refactor tests: replace sys.modules hacks with real DI + in-memory DB 
- Add db=None, embedder=None params to ai_provider_wrapper, text_pipeline, compute_similarities
- New conftest.py: FakeEmbedder, mem_db (in-memory DuckDB), fake_embedder fixtures
- Rewrite test_ai_provider_wrapper (4 tests), test_rerun_embeddings_retry (2 tests), test_similarity_compute_filter (1 test) with real implementations
- Fix rerun_embeddings tests hanging on _get_all_windows by patching it alongside _clear_embeddings
- All 53 tests pass (2 skipped), 0 sys.modules hacks in refactored files
 2026-03-23 21:21:10 +01:00
sgeboers b7350d8f87 test: rewrite test_database_audit using mem_db fixture, no disk writes required 2026-03-23 21:01:09 +01:00
sgeboers e4f2c7ff59 fix: update integration test to unpack 5-tuple from ensure_text_embeddings 2026-03-23 20:30:35 +01:00
sgeboers ebb663aa8f docs: add test refactor implementation plan 2026-03-23 20:20:34 +01:00
sgeboers 07a89a207c docs: add test refactor design - replace monkeypatching with DI + in-memory DuckDB 2026-03-23 20:15:29 +01:00
sgeboers ce27dc6ac5 chore(ledgers): record fusion+similarity run summary and JSON details 2026-03-23 17:48:39 +01:00
sgeboers 22f53840b8 fix(explorer): normalise party name 'Nieuw Sociaal Contract' -> 'NSC' 2026-03-22 23:13:45 +01:00
sgeboers cbbdc6e738 fix(explorer): cleaner trajectories, NSC support, controversy filter, voting display, URL links 2026-03-22 23:12:43 +01:00
sgeboers cd7bb3b1e0 fix(explorer): filter to uniform-dim windows before PCA to prevent np.vstack shape mismatch 2026-03-22 22:54:54 +01:00
sgeboers 2891e9ee70 feat: add StemAtlas Streamlit app, explorer, Docker deployment, blog charts 2026-03-22 22:38:17 +01:00
sgeboers daa22c5e2b feat: complete parliamentary embedding pipeline with full historical coverage 
- Add fused (SVD + text) embedding pipeline for annual windows 2016-2026
- Fix store_fused_embedding duplicate bug: DELETE before INSERT (idempotent)
- Add --text-batch-size CLI flag to run_pipeline.py (default 200)
- Add explicit --start-date/--end-date to download_past_year.py
- Backfill mp_votes for all motions (party-level votes, 111k new rows)
- Add similarity cache recompute: 212k rows across 9 annual windows
- Improve ai_provider retry logic, text_pipeline batching
- Improve analysis/political_axis PCA handling and visualizations
- Add diagnostic/utility scripts: compare_svd, generate_compass, inspect_axis, etc.
- Untrack data/motions.db (3.6GB binary), add to .gitignore with outputs/
- Update continuity ledger with full session state
 2026-03-22 16:08:06 +01:00
sgeboers a78bee9b0a feat(similarity): add precomputed similarity cache, fix fusion N+1, add 429 retry 
- Add similarity/ package (compute.py, lookup.py) with numpy-based
  pairwise cosine similarity and cached lookup
- database.py: create embeddings + similarity_cache tables in _init_database(),
  add store_similarity_batch/get_cached_similarities/clear_similarity_cache helpers
- pipeline/fusion.py: replace N+1 per-motion embedding SELECT with single
  bulk JOIN using DuckDB QUALIFY window function
- ai_provider.py: retry HTTP 429 with Retry-After header support
- migrations/2026-03-22-add-similarity-cache.sql: make executable
- Add tests for similarity compute, db helpers, and 429 retry (34 pass, 2 skip)
 2026-03-22 03:02:25 +01:00
sgeboers a248807e03 Add design: embedding-based motion similarity cache 
Precomputed top-K similarity cache replacing the naive Python-scan
search_similar(). Also covers fixes for: embeddings table missing from
_init_database, fusion N+1 query, and ai_provider 429 retry.
 2026-03-22 02:39:20 +01:00
sgeboers bf68e48460 fix(analysis): improve PCA handling when PC1 dominates, add pca_residual option and plot autoscaling/variance annotation 2026-03-22 00:13:56 +01:00
sgeboers 23a1234314 feat(analysis): add 2D political compass (PCA/anchor) and 2D trajectories + visualizations 
- compute_2d_axes (pca, anchor) with optional L2-normalisation pre-projection
- compute_2d_trajectories: per-MP coords, step vectors, magnitudes, totals
- plot_political_compass and plot_2d_trajectories (Plotly HTML)
- tests/test_political_compass.py (synthetic unit test)
 2026-03-21 23:58:38 +01:00
sgeboers 3551a82f83 feat(analysis): add 2D political compass and 2D trajectories 
- compute_2d_axes (PCA + anchor)
- compute_2d_trajectories
- plot_political_compass, plot_2d_trajectories
- unit test: tests/test_political_compass.py
 2026-03-21 23:53:55 +01:00
sgeboers f7d806dc3a fix(analysis): add Procrustes alignment and normalize vectors for drift computation 
SVD sign/rotation is arbitrary per window. Without alignment, drift was
dominated by basis flips (~1.9/step max=2.0) rather than real political movement.

- _procrustes_align_windows(): aligns each window to the previous using
  orthogonal Procrustes on common entities (scipy, falls back gracefully)
- compute_trajectories(): builds aligned window dict before per-MP drift calc,
  adds normalize=True (L2-normalise) to remove cross-window magnitude differences
  caused by varying numbers of motions per quarter
- Results now in sensible range: NSC=2.28, DENK=1.90, ... PVV=0.82, FVD=0.70
- NSC large late jump (1.39 in Q4→Q1 2026) matches its parliamentary fracture
- Add outputs/trajectories_party_aligned.html with cleaned-up drift chart
 2026-03-21 23:37:20 +01:00
sgeboers aa2f66ac9f feat(analysis): fetch real MP metadata, fix anchor axis for party-level actors 
- fetch_mp_metadata: use real OData URL with pagination (1200 records, 5 pages)
  uses Fractie.Afkorting not NaamNL for abbreviation matching
  skips Verwijderd=true records
- upsert_mp_metadata: keep most recent membership (prefer active over ended,
  then higher Van date) so current party affiliations are not overwritten by historical
- compute_anchor_axis: anchor directly on party-level SVD entities (GroenLinks-PvdA etc)
  before falling back to mp_metadata individual MP lookup
- test_fetch_mp_metadata: fix mock for timeout kwarg + pagination + Afkorting field
- Generated anchor axis HTML for 2025-Q2 through 2026-Q1 in outputs/
 2026-03-21 23:33:47 +01:00
sgeboers 5ad83ef1be fix(tests): update test_extract_mp_votes for party-level actor inclusion 
New extract_mp_votes behavior inserts all actors (party + individual MPs),
not only comma-name MPs. Test now validates both types and their party column.
Also adds generated HTML visualizations (political axis x5 windows + trajectories).
 2026-03-21 23:25:27 +01:00
sgeboers 847b783877 fix(pipeline): fix API pagination, add skip_details fast path, bulk mp_votes insert 
- _get_voting_records returns (records, besluit_meta) tuple; paginate via Besluit?expand=Stemming (469/mo vs 8400)
- get_motions(skip_details=True) bypasses per-motion detail chain (3 HTTP calls/motion)
- extract_mp_votes rewritten: bulk DataFrame insert (80k rows in 1.9s), includes party-level actors
- run_pipeline.py fixed: pass db_path not db, handle dict/int return types
- download_past_year.py: skip_details=True default, limit-per-chunk default 50000
 2026-03-21 23:24:06 +01:00
sgeboers f2a831dfcf feat(pipeline): add orchestrator CLI, analysis modules, and ActorFractie ingestion 
- pipeline/run_pipeline.py: CLI orchestrator for all 5 pipeline phases with
  --dry-run, --skip-*, --window-size, --svd-k, --start/end-date flags
- analysis/{political_axis,trajectory,clustering,visualize}.py: PCA/anchor
  ideological axis, MP drift trajectories, UMAP + KMeans clustering, Plotly HTML output
- api_client.py: capture ActorFractie per individual MP vote (comma in ActorNaam)
  into mp_vote_parties dict on each motion
- database.insert_motion: auto-insert mp_votes rows with party affiliation for
  newly ingested motions when mp_vote_parties is present
- Add scikit-learn to pyproject.toml for KMeans clustering
- tests/test_run_pipeline.py: window generation, dry-run, skip-all paths
- tests/test_analysis.py: PCA axis, anchor axis, trajectory drift, KMeans

Ref: thoughts/shared/plans/2026-03-21-parliamentary-embedding-pipeline-plan.md
 2026-03-21 22:40:28 +01:00
sgeboers a36e6cba4e feat(pipeline): implement parliamentary embedding pipeline MVP 
- Add 4 migration files: mp_votes, mp_metadata, svd_vectors, fused_embeddings
- Extend database.py with 5 new helper methods and table init
- Add pipeline/ package: extract_mp_votes, fetch_mp_metadata, text_pipeline,
  svd_pipeline (with Procrustes alignment), fusion
- Add full test suite (17 tests) covering all pipeline modules and migrations
- Fix Procrustes alignment bug: scipy scale is a norm value, not a multiplier
- Fix DuckDB date type handling in test assertions (datetime.date vs string)
- Remove duckdb.py shim; tests now run against real duckdb + scipy via uv

Ref: thoughts/shared/plans/2026-03-21-parliamentary-embedding-pipeline-plan.md
 2026-03-21 22:31:22 +01:00
sgeboers c498c3467e update plan: replace spike with confirmed FractieZetelPersoon fetch task 2026-03-21 22:05:56 +01:00
sgeboers 0bbda408fb plan: parliamentary embedding pipeline MVP implementation plan 2026-03-21 21:57:39 +01:00
sgeboers fd73da3752 design: parliamentary embedding pipeline (late fusion SVD + text) 2026-03-21 21:54:32 +01:00