feat(pipeline): implement parliamentary embedding pipeline MVP
- Add 4 migration files: mp_votes, mp_metadata, svd_vectors, fused_embeddings - Extend database.py with 5 new helper methods and table init - Add pipeline/ package: extract_mp_votes, fetch_mp_metadata, text_pipeline, svd_pipeline (with Procrustes alignment), fusion - Add full test suite (17 tests) covering all pipeline modules and migrations - Fix Procrustes alignment bug: scipy scale is a norm value, not a multiplier - Fix DuckDB date type handling in test assertions (datetime.date vs string) - Remove duckdb.py shim; tests now run against real duckdb + scipy via uv Ref: thoughts/shared/plans/2026-03-21-parliamentary-embedding-pipeline-plan.md
This commit is contained in:
@@ -0,0 +1,98 @@
|
||||
---
|
||||
date: 2026-03-19
|
||||
topic: "Stemwijzer AI & DB design"
|
||||
status: draft
|
||||
---
|
||||
|
||||
## Problem Statement
|
||||
|
||||
We need a clear, low-risk design to improve AI usage and query ergonomics in this repository. The codebase currently ingests motions, stores them in DuckDB, and generates AI-driven layman summaries via an OpenRouter/OpenAI client. There are a few maintenance issues (e.g., missing config keys, a broken reset script) and no embedding/search infrastructure.
|
||||
|
||||
**Goal:**
|
||||
- Centralize AI/LLM usage behind a provider abstraction so we can swap or prefer providers later.
|
||||
- Introduce minimal embeddings storage and search so we can add semantic features without heavy infra.
|
||||
- Prefer ibis for read/query paths where that improves clarity and maintainability (the repo already imports ibis in read.py).
|
||||
|
||||
|
||||
## Constraints
|
||||
|
||||
- Work must be incremental and non-disruptive: keep existing DuckDB schema and write paths where possible.
|
||||
- Do not add external services (vector DB) in the first iteration — store embeddings in DuckDB as JSON for now.
|
||||
- Secrets must remain environment-driven (no checked-in secrets). Add env var defaults only.
|
||||
- Keep changes small and well-tested; make it easy to roll back.
|
||||
|
||||
|
||||
## Approach (chosen)
|
||||
|
||||
I'll introduce two small layers:
|
||||
- **ai_provider**: a thin adapter that exposes get_embedding(text) and chat_completion(messages). It will use the existing OpenRouter/OpenAI path by default and can be extended to prefer other providers if/when desired.
|
||||
- **query_dal**: read-focused utilities implemented with ibis to replace direct SQL reads in the app and other read-heavy paths. Writes (insert_motion, update_user_vote) stay in database.py initially.
|
||||
|
||||
This gives the benefits of abstraction and pythonic query composition while keeping risk low.
|
||||
|
||||
|
||||
## Architecture
|
||||
|
||||
High level components (repo root):
|
||||
- api_client.py — fetches motion data from Tweede Kamer OData (unchanged)
|
||||
- scraper.py — optional HTML scraping fallback (unchanged)
|
||||
- database.py — current writes, schema initialization (add small embeddings table)
|
||||
- summarizer.py — generate layman summaries (refactor to use ai_provider)
|
||||
- app.py — Streamlit UI (switch read paths to query_dal)
|
||||
- scheduler.py — orchestrates ingestion and triggers summarization (unchanged)
|
||||
|
||||
Additions:
|
||||
- ai_provider.py — single place for LLM/embedding calls and retries
|
||||
- query_dal.py — ibis-based read helpers (get_filtered_motions, calculate_party_matches)
|
||||
- minimal embeddings table in DuckDB (motion_id, model, vector JSON, created_at)
|
||||
|
||||
|
||||
## Components and responsibilities
|
||||
|
||||
- **ai_provider**: choose provider, handle retries/backoff, return plain Python objects (list[float] embeddings, str completions). Keep error classes small and testable.
|
||||
- **database (existing)**: add store_embedding and search_similar helpers (naive in-Python cosine scan). Keep insert_motion/update_user_vote unchanged to minimize risk.
|
||||
- **query_dal**: use ibis for read queries used by Streamlit paths (get_filtered_motions, session lookups). Return parsed JSON fields.
|
||||
- **summarizer**: call ai_provider.chat_completion to get summary; update motions.layman_explanation; optionally compute embedding via ai_provider.get_embedding and store via database.store_embedding.
|
||||
- **app.py**: replace direct duckdb selects with query_dal functions.
|
||||
|
||||
|
||||
## Data Flow
|
||||
|
||||
1. Ingest: scheduler / scraper / api_client fetch motions and call database.insert_motion(motion).
|
||||
2. Summarize: summarizer calls ai_provider.chat_completion(summary prompt) → writes layman_explanation to motions table. Optionally computes embedding and writes to embeddings table.
|
||||
3. Query: Streamlit app calls query_dal.get_filtered_motions (ibis) to load motions for sessions and query_dal.calculate_party_matches for results.
|
||||
4. Semantic search (future): query_dal or app can call database.search_similar by providing an embedding computed with ai_provider.get_embedding.
|
||||
|
||||
|
||||
## Error Handling
|
||||
|
||||
- ai_provider: retries with exponential backoff for transient errors; raises a ProviderError for terminal failures so callers can decide retry semantics.
|
||||
- Summarizer: non-fatal on AI failures — store an empty/fallback summary and log the failure; surface a user-facing message in Streamlit if generating summaries fails interactively.
|
||||
- DB functions: existing try/except patterns retained; ensure connections are closed on error.
|
||||
|
||||
|
||||
## Testing Strategy
|
||||
|
||||
- Unit tests for ai_provider using mocks for HTTP/openai responses.
|
||||
- DB tests using temporary DuckDB files to verify store_embedding and search_similar behavior.
|
||||
- query_dal tests using ibis against a temporary DB file; ensure JSON fields parse correctly.
|
||||
- Summarizer tests mock ai_provider to assert DB writes happen.
|
||||
|
||||
|
||||
## Open Questions
|
||||
|
||||
- Store embeddings inside motions table vs separate embeddings table? Recommendation: separate embeddings table for clarity and easier upserts.
|
||||
- Do we want to prefer other providers (Copilot) automatically? This repo currently references OPENROUTER. If user wants Copilot preference, we can add env vars and selection logic later.
|
||||
|
||||
|
||||
## Next steps (short)
|
||||
|
||||
1. Add ai_provider.py (adapter) and tests.
|
||||
2. Add embeddings table and store/search helpers in database.py and tests.
|
||||
3. Add query_dal.py with ibis reads and tests.
|
||||
4. Refactor summarizer.py to use ai_provider and optionally store embeddings.
|
||||
5. Update Streamlit app read paths to use query_dal.
|
||||
6. Fix housekeeping bugs: reset.py references reset_database(), scraper uses undefined SCRAPING_DELAY — address these small fixes in a separate patch.
|
||||
|
||||
|
||||
I'm proceeding to save this design to thoughts/shared/designs/2026-03-19-stemwijzer-design.md and will spawn the planner to create a detailed implementation plan. Interrupt if you want changes to the design text above.
|
||||
@@ -0,0 +1,116 @@
|
||||
---
|
||||
date: 2026-03-21
|
||||
topic: "Reuse motions as a guided policy explorer"
|
||||
status: draft
|
||||
---
|
||||
|
||||
## Problem Statement
|
||||
|
||||
We want to repurpose existing "motions" data so it becomes a lightweight, discovery-driven way for users to explore policy positions and discover related content. This is not a full proposal system; it's a guided exploration and bookmarking flow that leverages our existing ingestion, summarization, embeddings, and session voting work.
|
||||
|
||||
**Why now:** We already ingest motions, generate layman explanations, compute embeddings, and store per-session votes. Reusing those building blocks gives high user value with modest effort.
|
||||
|
||||
## Constraints
|
||||
|
||||
**Non-negotiables and technical limits:**
|
||||
- Use the existing database schema where possible (motions table, embeddings table, user_sessions). Do not require a new external vector DB for MVP.
|
||||
- Keep the Streamlit UI model (app.py) and session-based votes intact for the initial rollout.
|
||||
- Avoid breaking migrations: rely on existing migrations and add new ones when necessary (no forced drops).
|
||||
- Respect current error-handling posture: network calls can fail; system must degrade gracefully.
|
||||
|
||||
## Chosen Approach
|
||||
|
||||
I'm choosing a "Guided Policy Explorer" approach because it reuses thehighest-value existing pieces (summaries, embeddings, session voting) and delivers a clear UX that fits the current codebase. This gives immediate product value with low risk.
|
||||
|
||||
**Core idea:** present curated short sessions and motion detail pages that combine the existing layman explanation, party-match results, and semantic "related motions" powered by stored embeddings.
|
||||
|
||||
Alternatives considered:
|
||||
- "Motion-as-Proposal platform": full lifecycle (draft → comment → vote). Rejected for MVP due to high complexity and data model changes.
|
||||
- "Motion Digest / Research Assistant": read-only pages and newsletters. Lower effort, but less interactive and reuses fewer of our current session features.
|
||||
|
||||
## Architecture
|
||||
|
||||
High-level view (existing pieces in bold):
|
||||
- Ingest: **api_client.py** + **scraper.py** gather motions and create motion records in the DB.
|
||||
- Persist: **database.py** stores motions, embeddings, and user_sessions.
|
||||
- Enrichment: **summarizer.py** + **ai_provider.py** generate layman explanations and embeddings.
|
||||
- Background jobs: **scheduler.py** runs ingest, summarization, and periodic clustering.
|
||||
- UI: **app.py** current Streamlit session flow — extend with "Explore" and "Motion detail" pages.
|
||||
- New: small **clusterer / similarity API** to compute and cache related-motion lists per motion.
|
||||
|
||||
## Key Components & Responsibilities
|
||||
|
||||
- Motion Ingest (existing): keep ingest as-is; add metadata flags (e.g., curated, candidate).
|
||||
- Motion Store (existing): motions table + embeddings table; add an **events/audit** table for user actions and important state transitions.
|
||||
- Summarizer / Embedding Worker (existing): scheduled job that ensures motions have layman_explanation and embeddings; add retry/backoff and logging.
|
||||
- Similarity service (new): computes nearest neighbors using stored vectors in-process for MVP and caches results in a small table. Swap to a vector index later if needed.
|
||||
- Session & Voting (existing): continue using user_sessions JSON blob for individual sessions; add optional event log entries for each vote.
|
||||
- UI (update): add "Explore" landing, motion detail view with layman text, party-match snapshot, related motions, and bookmark/flag actions. Reuse Streamlit components.
|
||||
- Admin tooling (new): migration scripts, a CLI to recompute embeddings/similarity, and an audit query helper.
|
||||
|
||||
## Data Flow
|
||||
|
||||
1. Ingest job (api_client/scraper) produces motion records and calls db.insert_motion.
|
||||
2. Summarizer worker picks up motions without layman_explanation or embeddings, calls ai_provider, and writes layman_explanation + embeddings.
|
||||
3. Clusterer/similarity job computes related-motion lists using stored embeddings and writes them to a cache table.
|
||||
4. UI "Explore" shows curated motion lists; "Motion detail" reads motion, layman_explanation, party-match snapshot, and cached related motions.
|
||||
5. User vote actions update user_sessions and also append an event to the audit table for traceability.
|
||||
6. Background analytics (optional) reuses user_events and embeddings for offline insights.
|
||||
|
||||
## Error Handling Strategy
|
||||
|
||||
- External calls: add retries with exponential backoff for AI provider and external APIs. Failures set a marker (e.g., summary_missing) and the system continues.
|
||||
- Missing embeddings: UI gracefully disables "related motions" and offers "compute on demand".
|
||||
- Idempotency: make insert_motion idempotent by URL/external id check at DB layer; use optimistic handling for duplicates.
|
||||
- Concurrency: avoid read-modify-write races by writing user events (append-only) and deriving session state from events when race-prone updates are detected.
|
||||
- Observability: replace prints with structured logging (module-level logger) and add basic metrics for worker errors, API failures, and queue lags.
|
||||
|
||||
## Testing Strategy
|
||||
|
||||
- Unit tests: DB helpers (insert_motion, store_embedding, similarity cache), summarizer functions (mock ai_provider), and session vote logic.
|
||||
- Migration tests: follow the existing pattern of applying migration SQL in a temp DB and asserting schema.
|
||||
- Integration tests: end-to-end ingest → summarize → embedding → similarity → UI-read path in CI (use monkeypatch for AI calls).
|
||||
- Load tests: simulate a few thousand embeddings search calls against the in-process search to validate performance assumptions for MVP.
|
||||
- Acceptance: confirm UX flows: Explore session, Motion detail, Vote -> party match, Related motions populated.
|
||||
|
||||
## High-level Plan & Estimates
|
||||
|
||||
Assumptions: one full-stack engineer (Python + Streamlit) and one part-time reviewer. All estimates are rough.
|
||||
|
||||
Milestone 0 — Validate & quick discovery (1 day)
|
||||
- Locate user's added markdown plan and extract exact requirements. (I'm assuming the file exists in thoughts/shared; if not, we validated by searching.)
|
||||
|
||||
Milestone 1 — MVP (8–12 engineer days)
|
||||
- Add similarity cache table and migration.
|
||||
- Summarizer: make embedding generation robust with retries and store vectors.
|
||||
- Clusterer job: compute and cache related motions.
|
||||
- UI: Explore landing, Motion detail page, related motion UI, bookmark/flag button.
|
||||
- Add event/audit table and write events on user votes and bookmarks.
|
||||
|
||||
Milestone 2 — Hardening & instrumentation (3–5 engineer days)
|
||||
- Replace prints with structured logging across touched modules.
|
||||
- Add migration tests and CI integration tests (mock AI).
|
||||
- Add health metrics & basic alerting for worker failures.
|
||||
|
||||
Milestone 3 — Polish & UX feedback (3–5 engineer days)
|
||||
- UX tweaks, performance tuning, compute on-demand fallback for embeddings, documentation, admin CLI.
|
||||
|
||||
Total MVP + polish: ~2–3 weeks of focused work.
|
||||
|
||||
## Risks & Mitigations
|
||||
|
||||
- Risk: Naive in-process embedding search will not scale. Mitigation: cache nearest neighbors per motion and plan a migration path to a vector index.
|
||||
- Risk: AI provider flakiness. Mitigation: retries, timeouts, and clear UI fallback. Tests must mock provider in CI.
|
||||
- Risk: Race conditions on session votes. Mitigation: append-only event log and derive authoritative session view from events when needed.
|
||||
- Risk: Schema drift and missing migrations. Mitigation: add migration tests and document required migrations in repo.
|
||||
|
||||
## Open Questions
|
||||
|
||||
- Which exact user journeys do we want first (single-session discover vs. persistent account/bookmarking)?
|
||||
- Do we want bookmarks persisted globally or per-session only? (Privacy implications.)
|
||||
- What's acceptable latency for "related motions" — precomputed nightly vs. near-real-time?
|
||||
- Any policy/legal ban on storing full body_text or on long-term retention of user votes?
|
||||
|
||||
---
|
||||
|
||||
I'm proceeding to create the design doc file at thoughts/shared/designs/2026-03-21-motions-guided-explorer-design.md and will spawn the implementation planner next. Interrupt if you want changes to the approach or scope now.
|
||||
Reference in New Issue
Block a user