From fd73da3752a47fd88f6554ff50dfb7546f8ac2cb Mon Sep 17 00:00:00 2001 From: Sven Geboers Date: Sat, 21 Mar 2026 21:54:32 +0100 Subject: [PATCH] design: parliamentary embedding pipeline (late fusion SVD + text) --- ...parliamentary-embedding-pipeline-design.md | 184 ++++++++++++++++++ 1 file changed, 184 insertions(+) create mode 100644 thoughts/shared/designs/2026-03-21-parliamentary-embedding-pipeline-design.md diff --git a/thoughts/shared/designs/2026-03-21-parliamentary-embedding-pipeline-design.md b/thoughts/shared/designs/2026-03-21-parliamentary-embedding-pipeline-design.md new file mode 100644 index 0000000..8230cff --- /dev/null +++ b/thoughts/shared/designs/2026-03-21-parliamentary-embedding-pipeline-design.md @@ -0,0 +1,184 @@ +--- +date: 2026-03-21 +topic: "Parliamentary Embedding Pipeline (Late Fusion)" +status: validated +--- + +## Problem Statement + +We want to implement the late-fusion embedding system described in EMBEDDING_ANALYSIS.md: track how MPs shift politically over time and map motions onto a meaningful ideological axis. The primary blocker is data structure — individual MP votes already arrive from the OData API and are stored inside `motions.voting_results` as a mixed JSON blob (party names + MP names together). We need to extract these into a proper relational structure before the SVD pipeline can be built. + +**Why this is the right next step:** We already have motion text, layman explanations, text embeddings infrastructure (Qwen3 via ai_provider), and DuckDB. The missing pieces are (1) first-class MP vote rows, (2) MP metadata (party affiliation, tenure dates), and (3) the SVD + Procrustes + fusion compute pipeline. + +## Constraints + +- **DuckDB only** — no pgvector, no external vector store. In-Python compute (scipy) is correct. +- **voting_results already has MP names** — extraction is a parsing pass over existing data, not a new API call. Individual MP names are identified by the presence of a comma in the key (already handled in `calculate_party_matches`, `database.py:264`). +- **Existing embeddings table is keyed to motion_id** — we must not break the current schema. SVD and fused vectors go into new tables. +- **ai_provider.get_embedding already works** — use it as-is for text embeddings; no model changes needed for MVP. +- **ibis/DuckDB preferred** over raw SQL for analysis queries (per project preferences). +- **uv** for dependency management; add `scipy`, `umap-learn`, `plotly`, `sentence-transformers` (or use existing ai_provider for embeddings). + +## Approach + +**Late-fusion pipeline in four phases:** + +1. **Extract** — parse MP-level votes out of `voting_results` JSON into an `mp_votes` table; fetch MP metadata from OData into `mp_metadata`. +2. **Compute SVD** — per time window, build sparse MP × motion matrix → SVD → Procrustes-align windows sequentially. +3. **Text embeddings** — ensure every motion has a text embedding (existing path; just fill gaps). +4. **Fuse** — concatenate aligned SVD motion vector + text embedding → store in `fused_embeddings` table. + +Alternatives considered: +- **Pure text embeddings only**: easier but loses the behavioral (voting) signal entirely. Rejected because the whole point of the plan is the fused representation. +- **Store aligned SVD + rotation matrices separately**: more flexible for recomputing, but adds complexity. MVP will store aligned vectors directly; rotation matrices are logged for debugging but not persisted. + +## Architecture + +``` +Data layer (DB): + motions (existing) + embeddings (existing — text vectors keyed to motion_id) + mp_votes (NEW — one row per MP per motion) + mp_metadata (NEW — MP name, party, entry/exit dates) + svd_vectors (NEW — per window, per entity: MP or motion) + fused_embeddings (NEW — per motion, per window: SVD + text concatenated) + +Pipeline modules (new, in pipeline/): + extract_mp_votes.py — JSON blob → mp_votes rows + fetch_mp_metadata.py — OData /Kamerlid → mp_metadata rows + svd_pipeline.py — time windows → SVD → Procrustes alignment → svd_vectors + text_pipeline.py — ensure embeddings coverage, delegates to existing summarizer + fusion.py — join svd_vectors + embeddings → fused_embeddings + +Analysis modules (new, in analysis/): + political_axis.py — first SVD component / anchor-party axis + trajectory.py — MP drift across aligned windows + clustering.py — UMAP on fused motion embeddings, thematic clusters + visualize.py — Plotly interactive trajectory and cluster plots + +CLI entry points (new): + pipeline/run_pipeline.py — orchestrate all phases with flags +``` + +## Key Components & Responsibilities + +**mp_votes table** +- Schema: `(id, motion_id, mp_name, party, vote ENUM(voor/tegen/afwezig), date, created_at)` +- Populated by `extract_mp_votes.py` doing a one-time parse of `motions.voting_results` JSON. +- Idempotent: skip motion_id if already extracted (upsert or EXISTS check). +- `party` field is left NULL initially; backfilled from `mp_metadata` after that table is populated. + +**mp_metadata table** +- Schema: `(mp_name, party, entry_date, exit_date, source_id)` +- Fetched from OData `/Kamerlid` endpoint (needs verification — see Open Questions). +- Fallback: derive approximate party affiliation from `mp_votes` rows (majority-party heuristic) if OData metadata is unavailable. + +**svd_vectors table** +- Schema: `(window_id, entity_type ENUM(mp/motion), entity_id, vector JSON, model TEXT, created_at)` +- Stores both MP and motion SVD vectors per time window, after Procrustes alignment. +- `window_id` is a string like `2024-Q1`. + +**fused_embeddings table** +- Schema: `(motion_id, window_id, vector JSON, svd_dims INT, text_dims INT, created_at)` +- Separate from the existing `embeddings` table to avoid schema conflicts. +- Vector is the concatenation of the SVD motion vector and the text embedding. + +**svd_pipeline.py** +- Groups motions by time window (quarterly default). +- Builds a sparse `scipy.sparse.csr_matrix` (MPs as rows, motions as columns, vote values encoded as +1/−1/0). +- Calls `scipy.sparse.linalg.svds(matrix, k=dims)` — `k` is configurable (default 50). +- Applies Procrustes alignment between consecutive windows using overlapping MPs as anchors. +- Logs Procrustes disparity score per transition; flags high disparity (election transitions). + +**extract_mp_votes.py** +- Reads all motions with `voting_results` JSON, parses keys: if comma in key → individual MP name, else → party/fraction name. +- Writes MP-level rows to `mp_votes`; party-level rows are ignored here (they're already used by the existing `calculate_party_matches` flow). +- Handles the three vote values: `voor` (+1), `tegen` (−1), `afwezig` (0). + +**fusion.py** +- For each motion in a window: lookup SVD motion vector from `svd_vectors`; lookup text embedding from `embeddings`. +- Concatenates vectors (simple `list + list`); stores in `fused_embeddings`. +- Skips motion if either vector is missing; logs counts. + +**analysis/ modules** +- All read-only from DB; write only to output files (HTML/PNG plots). +- `political_axis.py`: project all MP SVD vectors onto the first principal component; optionally define axis by anchor parties (e.g. VVD vs SP). +- `trajectory.py`: collect MP's aligned SVD vector per window → compute drift distance → plot trajectory over time. +- `clustering.py`: run UMAP on `fused_embeddings` per window → label with policy_area or thematic cluster. +- `visualize.py`: Plotly interactive scatter/line plots; outputs self-contained HTML. + +## Data Flow + +``` +Phase 1 — Extract + motions.voting_results (JSON, existing) + → extract_mp_votes.py + → INSERT mp_votes rows (motion_id, mp_name, vote, date) + + OData /Kamerlid + → fetch_mp_metadata.py + → INSERT mp_metadata rows (mp_name, party, entry_date, exit_date) + → UPDATE mp_votes.party via JOIN (backfill) + +Phase 2 — SVD + mp_votes (date-filtered per window) + → sparse MP × motion matrix + → scipy svds(k=50) + → raw SVD vectors per window + + Procrustes alignment: + window[t-1] aligned vectors + window[t] raw vectors + → overlapping MPs as anchors + → scipy.spatial.procrustes → rotation R + → window[t] aligned vectors + → INSERT svd_vectors rows + +Phase 3 — Text embeddings (fill gaps) + motions without embedding in embeddings table + → text_pipeline.py → ai_provider.get_embedding(body_text or description) + → INSERT embeddings rows (existing schema) + +Phase 4 — Fusion + svd_vectors (motion, window) + embeddings (motion) + → fusion.py + → INSERT fused_embeddings rows + +Phase 5 — Analysis (on demand) + fused_embeddings + mp_metadata + svd_vectors + → analysis modules + → HTML plots output +``` + +## Error Handling Strategy + +- **Extraction idempotency**: `extract_mp_votes` checks `SELECT COUNT(*) FROM mp_votes WHERE motion_id = ?` before inserting; re-runs are safe. +- **Sparse windows**: if a time window has fewer than `MIN_MOTIONS` (default 20) or `MIN_MPs` (default 10), skip SVD for that window and log a warning. Do not crash. +- **Procrustes at election transitions**: chain alignment via the last quarter of the old term and first quarter of the new term using only returning MPs. If overlap < 30%, log as HIGH_DISPARITY and store the window but flag it. +- **Missing text embeddings**: log motions skipped in fusion; the SVD-only path remains valid for those motions. +- **OData metadata unavailable**: fall back to heuristic party assignment (mp_votes majority-party per MP name). Log which MPs used fallback. +- **Replace prints with structured logging**: all pipeline modules use `logging.getLogger(__name__)` — not `print()`. + +## Testing Strategy + +- **Unit**: + - Vote parser: given sample `voting_results` JSON, assert correct MP rows extracted and party rows ignored. + - Sparse matrix builder: inject 5 MPs × 10 motions → assert matrix shape and values. + - Procrustes wrapper: inject two small aligned-then-rotated matrices → assert recovered rotation close to identity. + - Fusion: inject matching SVD and text vectors → assert concatenated output length = svd_dims + text_dims. + +- **Integration**: + - Extract → SVD → Fusion on a fixture of 50 motions (stored in `tests/fixtures/`). Monkeypatch ai_provider for text embeddings. Assert `fused_embeddings` table populated and vector dimensions correct. + +- **Regression**: + - Run pipeline on a fixed 100-motion snapshot. Assert output dimensions and row counts stable across runs. + +- **Migration tests**: + - Follow existing pattern (`tests/test_migration_embeddings.py`): apply new migration SQL to a temp DuckDB, assert expected tables and columns. + +## Open Questions + +1. **OData `/Kamerlid` endpoint availability**: does it expose party affiliation and tenure dates with the same API key/base URL we already use? If not, we need a scraping fallback for `mp_metadata`. +2. **Store rotation matrices?**: MVP stores aligned vectors directly. Should we also persist the Procrustes R matrix per window transition so we can re-project new MPs added later without full recomputation? +3. **Output target**: CLI producing HTML plots (simplest) vs. new Streamlit page vs. Jupyter notebook. Recommendation: CLI first, Streamlit page in a follow-up. +4. **Time window granularity**: quarterly is the default. Should we validate this empirically first with an annual window (larger, more stable matrices) and switch to quarterly once the pipeline is proven? +5. **SVD dimensions k**: default 50 dims for SVD. This needs to be validated against the actual data size (number of unique MPs × motions per window). A window with 100 MPs and 50 motions cannot have k=50 — needs to be `k < min(n_mps, n_motions)`. Pipeline must enforce this dynamically.