feat(pipeline): implement parliamentary embedding pipeline MVP
- Add 4 migration files: mp_votes, mp_metadata, svd_vectors, fused_embeddings - Extend database.py with 5 new helper methods and table init - Add pipeline/ package: extract_mp_votes, fetch_mp_metadata, text_pipeline, svd_pipeline (with Procrustes alignment), fusion - Add full test suite (17 tests) covering all pipeline modules and migrations - Fix Procrustes alignment bug: scipy scale is a norm value, not a multiplier - Fix DuckDB date type handling in test assertions (datetime.date vs string) - Remove duckdb.py shim; tests now run against real duckdb + scipy via uv Ref: thoughts/shared/plans/2026-03-21-parliamentary-embedding-pipeline-plan.md
This commit is contained in:
@@ -0,0 +1,106 @@
|
||||
---
|
||||
date: 2026-03-19
|
||||
topic: "Stemwijzer AI & DB implementation plan"
|
||||
status: draft
|
||||
---
|
||||
|
||||
## Summary
|
||||
|
||||
Implementation plan derived from thoughts/shared/designs/2026-03-19-stemwijzer-design.md.
|
||||
Goal: add a provider abstraction for AI calls, minimal embeddings stored in DuckDB (JSON), and an ibis-based read DAL. Keep changes small, additive and well-tested.
|
||||
|
||||
|
||||
## High-level approach (chosen)
|
||||
|
||||
- Add **ai_provider**: adapter exposing get_embedding(text) and chat_completion(messages) with retries and ProviderError.
|
||||
- Add **embeddings** table (DuckDB) and store/search helpers in database.py (naive Python cosine scan).
|
||||
- Add **query_dal**: ibis-based read helpers for Streamlit (get_filtered_motions, calculate_party_matches).
|
||||
- Refactor summarizer to call ai_provider and optionally store embeddings.
|
||||
- Minimal housekeeping fixes: reset.py and SCRAPING_DELAY in scraper.py.
|
||||
|
||||
|
||||
## Micro-tasks (11 tasks)
|
||||
|
||||
All tasks are intentionally small (file-level changes + tests). Estimates assume one developer full-time; see Risk and Calendar section below.
|
||||
|
||||
Batch 1 (foundation, parallelizable)
|
||||
|
||||
1. Add tests fixtures for temporary DuckDB (tests/conftest.py) — 2h — low risk
|
||||
2. Add migration SQL to create embeddings table (migrations/2026-03-19-add-embeddings.sql) — 1h — low risk
|
||||
3. Add ai_provider adapter (src/ai_provider.py) + tests (tests/test_ai_provider.py) — 6h — medium risk
|
||||
4. Add scraper SCRAPING_DELAY default (src/scraper.py) + tests — 1h — low risk
|
||||
5. Fix reset script to run migrations (src/reset.py) + tests — 2h — low risk
|
||||
|
||||
Batch 2 (core modules)
|
||||
|
||||
6. Add store_embedding and search_similar to src/database.py + tests (tests/test_database_embeddings.py) — 8h — medium risk
|
||||
7. Add query_dal (src/query_dal.py) with ibis reads + tests (tests/test_query_dal.py) — 6h — medium risk
|
||||
8. Refactor summarizer to use ai_provider and optionally store embeddings (src/summarizer.py) + tests (tests/test_summarizer.py) — 6h — medium risk
|
||||
|
||||
Batch 3 (integration)
|
||||
|
||||
9. Add CLI semantic search helper (src/cli_search.py) + tests — 4h — low-medium risk
|
||||
10. Update app read paths to use query_dal (src/app.py) + tests — 3h — low risk
|
||||
|
||||
Batch 4 (docs/config)
|
||||
|
||||
11. Add .env.example entries for new env vars — 1h — low risk
|
||||
|
||||
|
||||
## PR order (recommended, small focused PRs)
|
||||
|
||||
1. PR A — tests/conftest (fixtures)
|
||||
2. PR B — migration SQL (embeddings table)
|
||||
3. PR C — ai_provider + tests
|
||||
4. PR D — database store/search helpers + tests
|
||||
5. PR E — query_dal + tests
|
||||
6. PR F — summarizer refactor + tests
|
||||
7. PR G — cli_search + tests
|
||||
8. PR H — app read changes + tests
|
||||
9. PR I — scraper/reset small fixes + tests
|
||||
10. PR J — .env.example
|
||||
|
||||
|
||||
## Estimates & schedule (one dev, full-time ~8h/day)
|
||||
|
||||
- Total estimated effort: ~50 hours (~6.25 days) + buffer → ~7 calendar days.
|
||||
- Conservative schedule: Batch 1 (2 days), Batch 2 (3 days), Batch 3 (1 day), Buffer/Review (1 day).
|
||||
|
||||
|
||||
## DB migration steps
|
||||
|
||||
- Add migrations/2026-03-19-add-embeddings.sql (additive).
|
||||
- Apply on staging first; backup DB, run migration, verify `SELECT count(*) FROM embeddings`.
|
||||
- No changes to motions table in first iteration.
|
||||
|
||||
|
||||
## Testing strategy
|
||||
|
||||
- Unit tests for ai_provider (mock HTTP responses). Use monkeypatch to avoid network.
|
||||
- DB tests use temporary DuckDB files (pytest fixtures) to verify storing and searching embeddings.
|
||||
- query_dal tests use ibis.duckdb.connect against a temporary DB file and parse JSON fields.
|
||||
- Summarizer tests mock ai_provider to assert DB writes (summary and optional embedding).
|
||||
|
||||
|
||||
## Error handling
|
||||
|
||||
- ai_provider: retry/backoff for transient errors; raise ProviderError for terminal failures.
|
||||
- Summarizer: non-fatal on AI failures — write fallback/empty summary, log, and surface message in UI when interactive.
|
||||
- DB functions: keep try/except patterns and ensure connections closed on error.
|
||||
|
||||
|
||||
## Risks & mitigations
|
||||
|
||||
- ai_provider changes: medium risk — mitigate with retries, clear ProviderError, and thorough unit tests.
|
||||
- Embedding search: medium (naive scan performance) — mitigate by keeping implementation simple and planning for ANN/FAISS later.
|
||||
- ibis usage: medium — mitigate with tests and keep query_dal narrow.
|
||||
|
||||
|
||||
## Next actions (what I'll do now)
|
||||
|
||||
- I wrote this implementation plan to thoughts/shared/plans/2026-03-19-stemwijzer-plan.md (draft).
|
||||
- I will NOT start applying code changes automatically. If you want, I can:
|
||||
- (A) Create the first PR patch (tests/conftest.py + migration) and open a draft for review, or
|
||||
- (B) Start implementing Task 1.1 (ai_provider) next.
|
||||
|
||||
Interrupt if you want changes to the plan or a different PR ordering. Otherwise tell me which task to start and I'll create the first patch.
|
||||
Reference in New Issue
Block a user