motief/thoughts/shared/designs/2026-03-21-parliamentary-embedding-pipeline-design.md

date	topic	status
2026-03-21	Parliamentary Embedding Pipeline (Late Fusion)	validated

Problem Statement

We want to implement the late-fusion embedding system described in EMBEDDING_ANALYSIS.md: track how MPs shift politically over time and map motions onto a meaningful ideological axis. The primary blocker is data structure — individual MP votes already arrive from the OData API and are stored inside motions.voting_results as a mixed JSON blob (party names + MP names together). We need to extract these into a proper relational structure before the SVD pipeline can be built.

Why this is the right next step: We already have motion text, layman explanations, text embeddings infrastructure (Qwen3 via ai_provider), and DuckDB. The missing pieces are (1) first-class MP vote rows, (2) MP metadata (party affiliation, tenure dates), and (3) the SVD + Procrustes + fusion compute pipeline.

Constraints

• DuckDB only — no pgvector, no external vector store. In-Python compute (scipy) is correct.
• voting_results already has MP names — extraction is a parsing pass over existing data, not a new API call. Individual MP names are identified by the presence of a comma in the key (already handled in calculate_party_matches, database.py:264).
• Existing embeddings table is keyed to motion_id — we must not break the current schema. SVD and fused vectors go into new tables.
• ai_provider.get_embedding already works — use it as-is for text embeddings; no model changes needed for MVP.
• ibis/DuckDB preferred over raw SQL for analysis queries (per project preferences).
• uv for dependency management; add scipy, umap-learn, plotly, sentence-transformers (or use existing ai_provider for embeddings).

Approach

Late-fusion pipeline in four phases:

1. Extract — parse MP-level votes out of voting_results JSON into an mp_votes table; fetch MP metadata from OData into mp_metadata.
2. Compute SVD — per time window, build sparse MP × motion matrix → SVD → Procrustes-align windows sequentially.
3. Text embeddings — ensure every motion has a text embedding (existing path; just fill gaps).
4. Fuse — concatenate aligned SVD motion vector + text embedding → store in fused_embeddings table.

Alternatives considered:

• Pure text embeddings only: easier but loses the behavioral (voting) signal entirely. Rejected because the whole point of the plan is the fused representation.
• Store aligned SVD + rotation matrices separately: more flexible for recomputing, but adds complexity. MVP will store aligned vectors directly; rotation matrices are logged for debugging but not persisted.

Architecture

Data layer (DB):
  motions          (existing)
  embeddings       (existing — text vectors keyed to motion_id)
  mp_votes         (NEW — one row per MP per motion)
  mp_metadata      (NEW — MP name, party, entry/exit dates)
  svd_vectors      (NEW — per window, per entity: MP or motion)
  fused_embeddings (NEW — per motion, per window: SVD + text concatenated)

Pipeline modules (new, in pipeline/):
  extract_mp_votes.py   — JSON blob → mp_votes rows
  fetch_mp_metadata.py  — OData /Kamerlid → mp_metadata rows
  svd_pipeline.py       — time windows → SVD → Procrustes alignment → svd_vectors
  text_pipeline.py      — ensure embeddings coverage, delegates to existing summarizer
  fusion.py             — join svd_vectors + embeddings → fused_embeddings

Analysis modules (new, in analysis/):
  political_axis.py     — first SVD component / anchor-party axis
  trajectory.py         — MP drift across aligned windows
  clustering.py         — UMAP on fused motion embeddings, thematic clusters
  visualize.py          — Plotly interactive trajectory and cluster plots

CLI entry points (new):
  pipeline/run_pipeline.py   — orchestrate all phases with flags

Key Components & Responsibilities

mp_votes table

• Schema: (id, motion_id, mp_name, party, vote ENUM(voor/tegen/afwezig), date, created_at)
• Populated by extract_mp_votes.py doing a one-time parse of motions.voting_results JSON.
• Idempotent: skip motion_id if already extracted (upsert or EXISTS check).
• party field is left NULL initially; backfilled from mp_metadata after that table is populated.

mp_metadata table

• Schema: (mp_name, party, entry_date, exit_date, source_id)
• Fetched from OData /Kamerlid endpoint (needs verification — see Open Questions).
• Fallback: derive approximate party affiliation from mp_votes rows (majority-party heuristic) if OData metadata is unavailable.

svd_vectors table

• Schema: (window_id, entity_type ENUM(mp/motion), entity_id, vector JSON, model TEXT, created_at)
• Stores both MP and motion SVD vectors per time window, after Procrustes alignment.
• window_id is a string like 2024-Q1.

fused_embeddings table

• Schema: (motion_id, window_id, vector JSON, svd_dims INT, text_dims INT, created_at)
• Separate from the existing embeddings table to avoid schema conflicts.
• Vector is the concatenation of the SVD motion vector and the text embedding.

svd_pipeline.py

• Groups motions by time window (quarterly default).
• Builds a sparse scipy.sparse.csr_matrix (MPs as rows, motions as columns, vote values encoded as +1/−1/0).
• Calls scipy.sparse.linalg.svds(matrix, k=dims) — k is configurable (default 50).
• Applies Procrustes alignment between consecutive windows using overlapping MPs as anchors.
• Logs Procrustes disparity score per transition; flags high disparity (election transitions).

extract_mp_votes.py

• Reads all motions with voting_results JSON, parses keys: if comma in key → individual MP name, else → party/fraction name.
• Writes MP-level rows to mp_votes; party-level rows are ignored here (they're already used by the existing calculate_party_matches flow).
• Handles the three vote values: voor (+1), tegen (−1), afwezig (0).

fusion.py

• For each motion in a window: lookup SVD motion vector from svd_vectors; lookup text embedding from embeddings.
• Concatenates vectors (simple list + list); stores in fused_embeddings.
• Skips motion if either vector is missing; logs counts.

analysis/ modules

• All read-only from DB; write only to output files (HTML/PNG plots).
• political_axis.py: project all MP SVD vectors onto the first principal component; optionally define axis by anchor parties (e.g. VVD vs SP).
• trajectory.py: collect MP's aligned SVD vector per window → compute drift distance → plot trajectory over time.
• clustering.py: run UMAP on fused_embeddings per window → label with policy_area or thematic cluster.
• visualize.py: Plotly interactive scatter/line plots; outputs self-contained HTML.

Data Flow

Phase 1 — Extract
  motions.voting_results (JSON, existing)
    → extract_mp_votes.py
    → INSERT mp_votes rows (motion_id, mp_name, vote, date)

  OData /Kamerlid
    → fetch_mp_metadata.py
    → INSERT mp_metadata rows (mp_name, party, entry_date, exit_date)
    → UPDATE mp_votes.party via JOIN (backfill)

Phase 2 — SVD
  mp_votes (date-filtered per window)
    → sparse MP × motion matrix
    → scipy svds(k=50)
    → raw SVD vectors per window

  Procrustes alignment:
    window[t-1] aligned vectors + window[t] raw vectors
    → overlapping MPs as anchors
    → scipy.spatial.procrustes → rotation R
    → window[t] aligned vectors
    → INSERT svd_vectors rows

Phase 3 — Text embeddings (fill gaps)
  motions without embedding in embeddings table
    → text_pipeline.py → ai_provider.get_embedding(body_text or description)
    → INSERT embeddings rows (existing schema)

Phase 4 — Fusion
  svd_vectors (motion, window) + embeddings (motion)
    → fusion.py
    → INSERT fused_embeddings rows

Phase 5 — Analysis (on demand)
  fused_embeddings + mp_metadata + svd_vectors
    → analysis modules
    → HTML plots output

Error Handling Strategy

• Extraction idempotency: extract_mp_votes checks SELECT COUNT(*) FROM mp_votes WHERE motion_id = ? before inserting; re-runs are safe.
• Sparse windows: if a time window has fewer than MIN_MOTIONS (default 20) or MIN_MPs (default 10), skip SVD for that window and log a warning. Do not crash.
• Procrustes at election transitions: chain alignment via the last quarter of the old term and first quarter of the new term using only returning MPs. If overlap < 30%, log as HIGH_DISPARITY and store the window but flag it.
• Missing text embeddings: log motions skipped in fusion; the SVD-only path remains valid for those motions.
• OData metadata unavailable: fall back to heuristic party assignment (mp_votes majority-party per MP name). Log which MPs used fallback.
• Replace prints with structured logging: all pipeline modules use logging.getLogger(__name__) — not print().

Testing Strategy

• Unit:

  • Vote parser: given sample voting_results JSON, assert correct MP rows extracted and party rows ignored.
  • Sparse matrix builder: inject 5 MPs × 10 motions → assert matrix shape and values.
  • Procrustes wrapper: inject two small aligned-then-rotated matrices → assert recovered rotation close to identity.
  • Fusion: inject matching SVD and text vectors → assert concatenated output length = svd_dims + text_dims.

• Integration:

  • Extract → SVD → Fusion on a fixture of 50 motions (stored in tests/fixtures/). Monkeypatch ai_provider for text embeddings. Assert fused_embeddings table populated and vector dimensions correct.

• Regression:

  • Run pipeline on a fixed 100-motion snapshot. Assert output dimensions and row counts stable across runs.

• Migration tests:

  • Follow existing pattern (tests/test_migration_embeddings.py): apply new migration SQL to a temp DuckDB, assert expected tables and columns.

Open Questions

1. OData /Kamerlid endpoint availability: does it expose party affiliation and tenure dates with the same API key/base URL we already use? If not, we need a scraping fallback for mp_metadata.
2. Store rotation matrices?: MVP stores aligned vectors directly. Should we also persist the Procrustes R matrix per window transition so we can re-project new MPs added later without full recomputation?
3. Output target: CLI producing HTML plots (simplest) vs. new Streamlit page vs. Jupyter notebook. Recommendation: CLI first, Streamlit page in a follow-up.
4. Time window granularity: quarterly is the default. Should we validate this empirically first with an annual window (larger, more stable matrices) and switch to quarterly once the pipeline is proven?
5. SVD dimensions k: default 50 dims for SVD. This needs to be validated against the actual data size (number of unique MPs × motions per window). A window with 100 MPs and 50 motions cannot have k=50 — needs to be k < min(n_mps, n_motions). Pipeline must enforce this dynamically.