feat(mp-quiz): add MP quiz tab and DB helpers; add design and plan docs

thoughts/shared/designs/2026-03-24-welk-tweede-kamerlid-ben-jij-design.md

@@ -0,0 +1,113 @@
+---
+date: 2026-03-24
+topic: "Welk tweede kamerlid ben jij?"
+status: draft
+---
+
+## Problem Statement
+
+We need a new Streamlit tab in explorer.py titled **"Welk tweede kamerlid ben jij?"** that interactively narrows the list of 2026 MPs by asking the user a sequence of yes/no/abstain questions (motions). The goal: find the minimal set of motions (questions) that uniquely identify a single MP, or determine that no unique MP exists because two or more MPs have identical voting records.
+
+**Why:** This is a guided identification quiz that helps users discover which MP they most resemble by iteratively comparing their answers to historic MP votes.
+
+## Constraints
+
+- Work inside the existing Streamlit explorer (single-file UI: **explorer.py**).
+- Use existing data models/tables: **mp_votes**, **mp_metadata**, **motions** (DuckDB / MotionDatabase). No new external services.
+- Keep reads read-only: do not modify the DB from the UI flow.
+- YAGNI: minimal viable UX first (linear question flow, basic results table), extensible later.
+
+## Approach (chosen)
+
+I recommend a two-stage approach that balances simplicity and correctness:
+
+- **Stage A (Batch-match + ranking):** Ask the user a small curated set of motions (e.g., high-controversy / high-discriminative score). Collect answers into a map motion_id -> vote and compute per-MP agreement counts using a new read-only DB helper. Show ranked candidates and whether any are unique.
+- **Stage B (Minimal distinguishing set):** If multiple candidates tie (or more than one remain), compute a minimal discriminating set of additional motions by greedily selecting motions that best split the remaining candidate set and present them as follow-up questions until a unique MP or impossibility is reached.
+
+
+Alternatives considered (rejected):
+
+- Asking motions adaptively from the start using an information-gain search over the entire motion space. Rejected because it’s heavier to implement and harder to explain to users; we can implement a greedy information-gain variant later.
+- Building a full decision tree offline for all MPs. Rejected for now because the dataset and party churn make maintenance cumbersome.
+
+Effort estimate (rough):
+
+- Backend: add one DB method to MotionDatabase (match_mps_for_votes) + helper to compute split scores — 2–4 hours.
+- Frontend: add new Streamlit builder, UI state, and wiring into tabs — 2–4 hours.
+- Testing & polish: 2–3 hours.
+
+Risks & dependencies
+
+- **Data quality:** If mp_votes.party or mp_metadata are incomplete, matching may be imperfect. We rely on existing backfill scripts to improve party fields.
+- **Performance:** Joins over mp_votes can be large; we'll limit candidate motion set and use read-only DuckDB queries, with caching where appropriate.
+
+## Architecture
+
+High-level components (all in-process Streamlit app):
+
+- **Explorer UI (explorer.py)** — new tab builder **build_mp_quiz_tab**. Presents questions and displays results.
+- **MotionDatabase (database.py)** — new read-only method **match_mps_for_votes(user_votes, limit)** that returns per-MP agreement and overlap counts. Also a helper **choose_discriminating_motions(candidates, excluded_motion_ids, k=1)** that scores motions by how well they split candidate MPs.
+- **DuckDB (data)** — existing tables: motions, mp_votes, mp_metadata.
+
+All calls stay local — the Streamlit UI instantiates MotionDatabase(db_path) and calls the new read methods.
+
+## Components and Responsibilities
+
+- **build_mp_quiz_tab (explorer.py)**
+  - Render intro and instructions.
+  - Load an initial pool of candidate motions (curated by controversy or SVD components via existing load_motions_df).
+  - Present one question at a time, store answers in st.session_state (motion_id -> vote).
+  - After each answer (or on demand), call MotionDatabase.match_mps_for_votes to get ranked candidates.
+  - If multiple candidates remain, call the discriminating-motion helper to pick the next question.
+  - Show final result (unique MP or note that multiple MPs are indistinguishable).
+
+- **MotionDatabase.match_mps_for_votes (database.py)**
+  - Input: user_votes dict {motion_id: vote_str}
+  - Output: ordered list of {mp_name, party, matched, total, agreement_pct}
+  - Implementation: create an in-memory relation of user_votes, join with mp_votes where mp_name LIKE '%,%' and aggregate matched / overlap counts. Order by agreement_pct, matched desc.
+
+- **MotionDatabase.choose_discriminating_motions (database.py)**
+  - Input: remaining candidate mp_names, excluded_motion_ids
+  - Output: motion_id(s) ranked by split-score (e.g., entropy or max-min split)
+  - Implementation: for a small candidate set, compute how many MPs vote 'voor'/'tegen'/'onthouden' on each motion and pick motion with best split.
+
+Files to modify (concrete)
+
+- explorer.py
+  - Add function: build_mp_quiz_tab(...) near other build_*_tab functions (e.g., after build_svd_components_tab).
+  - Add new tab label to the tab_labels list and wire into the st.tabs and fallback radio branches. (See existing tab pattern at explorer.py around lines ~626-779.)
+
+- database.py
+  - Add methods: match_mps_for_votes and choose_discriminating_motions near calculate_party_matches / mp_votes helpers.
+
+## Data Flow
+
+1. UI loads candidate motion list via existing load_motions_df(db_path).
+2. User answers a question => stored in st.session_state['mp_quiz_votes'] mapping motion_id -> vote_token.
+3. UI calls MotionDatabase.match_mps_for_votes(user_votes) (read-only DuckDB). Returns sorted candidate MPs with matched/total/agreement_pct.
+4. If >1 candidate remains, UI calls MotionDatabase.choose_discriminating_motions(candidates, excluded) to pick next motion(s).
+5. Repeat until one candidate remains OR no motion splits candidates (tie by identical voting histories).
+
+## Error Handling
+
+- Validation: normalize UI votes to the canonical tokens used in mp_votes (lowercase Dutch tokens like 'voor','tegen','onthouden','afwezig').
+- Empty or missing data: if user_votes is empty or no overlaps exist, show helpful message and fall back to top-ranked MPs by similarity.
+- Division-by-zero: in match computations, treat zero-overlap MPs as excluded from ranking and surface a clear message.
+- Timeouts / heavy queries: restrict candidate set and use read-only DuckDB and caching (@st.cache_data) to avoid repeated heavy queries.
+
+## Testing Strategy
+
+- Unit tests for database methods (new tests/test_match_mps.py):
+  - small synthetic mp_votes fixture to assert matched/total/agreement_pct logic.
+  - tests for choose_discriminating_motions producing expected splits.
+- Integration test for explorer tab (tests/test_explorer_quiz.py): render the builder function in a headless mode and assert UI state updates and DB calls succeed (similar to existing tests/test_explorer_import.py).
+
+## Open Questions
+
+1. Do we want an initial curated motion set (top-10 controversial), or start fully adaptive? I'll implement a small curated seed and make adaptive/discovery optional.
+2. UX: Should we let users skip a question (abstain) and count abstain as a valid token? I assume yes and will treat abstain as a normal vote that matches mp_votes 'onthouden' or 'afwezig' values.
+3. Performance limits: how many motions should we allow the user to answer (arbitrary cap e.g., 20)? I suggest 20 to keep interactions snappy.
+
+## Next steps
+
+I'm proceeding to create the design doc file at thoughts/shared/designs/2026-03-24-welk-tweede-kamerlid-ben-jij-design.md and commit it. Interrupt if you want changes. After that I'll spawn the planner to create a detailed implementation plan based on this design.

thoughts/shared/plans/2026-03-24-welk-tweede-kamerlid-ben-jij-plan.md

@@ -0,0 +1,197 @@
+# "Welk tweede kamerlid ben jij?" Implementation Plan
+
+**Goal:** Add a Streamlit quiz tab that interactively asks the user motion (vote) questions and narrows the set of 2026 MPs to find the best-matching MP. Implement two DB helpers (matching + discriminating-motion selection), the UI builder and tab wiring, and tests. Minimal viable changes only — no UX bells & whistles.
+
+**Design:** thoughts/shared/designs/2026-03-24-welk-tweede-kamerlid-ben-jij-design.md
+
+---
+
+## Dependency Graph
+
+```
+Batch 1 (parallel): 1.1 [foundation - no deps], 1.2 (plan file) [none]
+Batch 2 (parallel): 2.1 [explorer UI - depends: 1.1]
+Batch 3 (parallel): 3.1 [integration tests - depends: 1.1,2.1]
+```
+
+---
+
+## Summary of implementation decisions (gap-filling)
+
+- MotionDatabase.match_mps_for_votes: implement as a read-only DuckDB-backed method on the existing MotionDatabase class (database.py). It accepts user_votes: Dict[int, str] where keys are motion ids and values are UI vote tokens. I will implement vote normalization inside the method (mapping UI tokens to canonical DB tokens) to avoid touching other modules. Rationale: keeps surface changes minimal and avoids creating new modules.
+
+- MotionDatabase.choose_discriminating_motions: implement in the same file. For a small candidate set (expected << 200 MPs), fetch mp_votes for candidate MPs across candidate motions (excluding already-answered motion ids). Score candidate motions by information-entropy of vote distribution among remaining candidates (higher entropy = better split). Ties broken by controversy_score then motion id.
+
+- Explorer UI changes: add build_mp_quiz_tab(db_path) to explorer.py and wire it into the tabs list and fallback radio. Use st.session_state['mp_quiz_votes'] to store answers as mapping str(motion_id)->UI token. Use @st.cache_data on any expensive DB-calls in the UI layer.
+
+- Vote token normalization: UI will present choices: "Voor", "Tegen", "Onthouden", "Afwezig / Geen stem". The DB stores lowercase tokens like 'voor', 'tegen', 'onthouden', 'afwezig'. match_mps_for_votes will normalize case and a small set of variants (e.g., 'Geen stem' -> 'afwezig', 'Abstain' -> 'onthouden') — explicit list included in tests.
+
+---
+
+## BATCH 1: Foundation (parallel - N implementers)
+All tasks in this batch have NO dependencies and can run simultaneously.
+
+### Task 1.1: Add DB helpers to MotionDatabase
+**File:** `database.py` (modify existing)
+**Test:** `tests/test_match_mps.py`
+**Depends:** none
+
+Description / Acceptance criteria:
+- Add two new public methods to MotionDatabase:
+  - match_mps_for_votes(user_votes: Dict[int, str], limit: int = 50) -> List[Dict]
+    - Returns an ordered list (desc by agreement_pct) of dicts with keys: mp_name, party, matched (int), overlap (int), agreement_pct (float 0-100).
+    - Behavior: for each mp present in mp_votes for any of the provided motions compute:
+      - overlap = number of motions where MP has a recorded vote AND the user provided a non-empty vote (i.e., not "Geen stem").
+      - matched = number of those overlaps where normalized(mp_vote) == normalized(user_vote).
+      - agreement_pct = matched / overlap * 100 rounded to 1 decimal. MPs with overlap==0 are excluded from the returned list.
+    - Ordering: agreement_pct desc, then matched desc, then mp_name asc.
+
+  - choose_discriminating_motions(candidates: List[str], excluded_motion_ids: List[int], k: int = 1) -> List[int]
+    - For the provided candidate mp_names, compute vote distributions per motion (voor/tegen/onthouden/afwezig) excluding motion ids in excluded_motion_ids.
+    - Score each motion by Shannon entropy of the distribution among the candidate MPs (treating 'afwezig' as a separate bucket). Higher entropy preferred.
+    - Return top-k motion ids as a list, tiebreakers: higher controversy_score (motions table) then lower motion id.
+
+Implementation notes & decisions:
+- Implement normalization inside these methods. Normalization mapping (DB vote -> canonical): map DB votes lowercased to one of {'voor','tegen','onthouden','afwezig'}. UI inputs (Voor/Tegen/Onthouden/Geen stem) normalized to these same tokens.
+- For performance, implement SQL queries that select mp_votes filtered by motion_id IN (...) and mp_name IN (candidates) and aggregate via GROUP BY mp_name and vote. For small candidate sets and a limited set of motion_ids this will be fast. If duckdb is not available, fall back to in-Python aggregates using the file-backed JSON format already present in MotionDatabase._init_database.
+- Add docstrings and basic parameter validation (raise ValueError for empty user_votes or empty candidates input). Tests will cover expected exceptions.
+
+Test outline (tests/test_match_mps.py):
+- Setup: create a temporary MotionDatabase using a temp db_path (MotionDatabase.reset_database() can be used if duckdb available; otherwise use file-backed mode). Insert a small set of motions and mp_votes via insert_motion / insert_mp_vote. Create at least 3 MPs with overlapping but distinct vote patterns across 4-6 motions.
+- Tests:
+  1) test_match_basic_counts: user_votes covering 3 motions returns expected matched/overlap/agreement_pct per MP.
+  2) test_match_excludes_zero_overlap: MPs with no recorded votes for provided motions are excluded.
+  3) test_choose_discriminating_motions_entropy_ranking: with a small candidate set, the chosen motion(s) split candidates as expected (assert returned motion id is one of known good splitters)
+  4) test_invalid_input: calling match_mps_for_votes with empty user_votes raises ValueError.
+
+Verify: `pytest -q tests/test_match_mps.py`
+
+Commit message: `feat(database): add match_mps_for_votes and choose_discriminating_motions`
+
+Estimated time: 3.0 - 4.5 hours
+
+---
+
+### Task 1.2: Add plan file (this document)
+**File:** `thoughts/shared/plans/2026-03-24-welk-tweede-kamerlid-ben-jij-plan.md` (this file)
+**Test:** none
+**Depends:** none
+
+Description: Add the implementation plan (this document) to the repo to provide step-by-step microtasks to implementers. No tests.
+
+Verify: visually review file in repo. No test run.
+
+Commit message: `docs(plans): add plan for 'Welk tweede kamerlid ben jij?'`
+
+Estimated time: 0.25 - 0.5 hours
+
+---
+
+## BATCH 2: Core UI (parallel - depends on Batch 1)
+All tasks in this batch assume the DB methods from Task 1.1 exist.
+
+### Task 2.1: Add Streamlit quiz tab & wiring
+**File:** `explorer.py` (modify existing)
+**Test:** `tests/test_explorer_quiz.py`
+**Depends:** 1.1
+
+Description / Acceptance criteria:
+- Add a function `build_mp_quiz_tab(db_path: str) -> None` placed near other build_*_tab functions (as described in the design, e.g., after build_svd_components_tab or near the top of the tab builders). The function must:
+  - Render a short intro/instructions.
+  - Load an initial pool of candidate motions using existing `load_motions_df(db_path)` and pick a seed set (top N by controversy_score). Decision: seed N = 8 (configurable constant in the function: SEED_MOTIONS = 8) — this is small and fast.
+  - Present questions one at a time: show motion title + layman_explanation (if available) and a radio with choices: "Voor", "Tegen", "Onthouden", "Geen stem" and a "Skip"/"Niet zeker" optional button mapped to "Geen stem". Choice stored to `st.session_state['mp_quiz_votes']` as mapping with keys str(motion_id) -> UI token.
+  - After each answer, call MotionDatabase.match_mps_for_votes(user_votes) to fetch ranked candidates and display a small DataFrame (top 10) with columns: MP name, party, matched, overlap, agreement_pct. Use st.dataframe.
+  - If more than 1 candidate remains with top agreement_pct tied, call MotionDatabase.choose_discriminating_motions(candidates, excluded_motion_ids) to pick the next question to ask and continue until one unique MP remains or choose_discriminating_motions returns an empty list (tie / indistinguishable). Cap total questions at 20 (SESS_CAP = 20).
+  - When unique MP is found (agreement_pct == 100 and overlap>0 and only one MP with highest agreement), show final MP summary (name, party) and their matching motions count.
+  - Use caching: wrap any repeated DB lookups (e.g., load_motions_df already cached) and mark heavy updates via @st.cache_data where appropriate.
+
+Implementation notes & decisions:
+- Keep all UI state local to st.session_state with keys prefixed `mp_quiz_` to avoid collisions.
+- Normalize UI tokens before sending to DB helper (but DB methods will also normalize; duplication is defensive).
+- Keep the UI function self-contained in explorer.py (do not create new modules for this minimal MVP).
+
+Test outline (tests/test_explorer_quiz.py):
+- Use monkeypatching to inject a MotionDatabase mock into explorer module or run in a test DB using MotionDatabase with temp db_path. The test must be import-safe (explorer.py imports many heavy libs), so follow pattern used by existing tests/test_explorer_import.py: import the module and assert `build_mp_quiz_tab` exists and is callable.
+- Functional assertions:
+  1) test_builder_exists: import explorer, assert callable(build_mp_quiz_tab)
+  2) test_ui_state_update_simulation: simulate st.session_state by creating a fake session dict (use monkeypatch to set st.session_state to a dict-like object) and calling build_mp_quiz_tab with a small temp DB where motions and mp_votes are prepared. Assert that after calling the builder with pre-filled votes the DataFrame block would display ranked candidates (test inspects returned structure if builder returns it, or else monkeypatch MotionDatabase.match_mps_for_votes to verify it was called with expected mapping).
+
+Verification: `pytest -q tests/test_explorer_quiz.py`
+
+Commit message: `feat(ui): add 'Welk tweede kamerlid ben jij?' tab and wiring in explorer.py`
+
+Estimated time: 2.0 - 4.0 hours
+
+---
+
+## BATCH 3: Integration & Tests (parallel - depends on Batches 1+2)
+
+### Task 3.1: Add integration test for quiz flow
+**File:** `tests/test_explorer_quiz_integration.py`
+**Test:** this file
+**Depends:** 1.1, 2.1
+
+Description / Acceptance criteria:
+- Create an end-to-end-ish headless test that:
+  - Sets up a temporary MotionDatabase instance (temp file path) and inserts a small controlled dataset: ~6 motions, 4 MPs with distinct votes.
+  - Calls build_mp_quiz_tab via explorer with monkeypatched st.session_state (or with a minimal wrapper) and simulates a sequence of user answers by pre-populating st.session_state['mp_quiz_votes'].
+  - Asserts that final candidate set matches expectations: either a unique MP (when answers match exactly one MP) or that the function properly identifies indistinguishable MPs (when two MPs have identical votes).
+
+Testing details & choices:
+- Avoid launching Streamlit server; tests only import explorer module and call the builder function in the same way other explorer tests do. Use monkeypatch to stub expensive functions (plotly, query_similar) where required.
+
+Verify: `pytest -q tests/test_explorer_quiz_integration.py`
+
+Commit message: `test(ui): add integration tests for mp quiz tab flow`
+
+Estimated time: 2.0 - 3.0 hours
+
+---
+
+## Verification & CI
+
+- Local verification commands (per task) use pytest. Example:
+  - `pytest -q tests/test_match_mps.py`
+  - `pytest -q tests/test_explorer_quiz.py`
+  - `pytest -q tests/test_explorer_quiz_integration.py`
+
+- CI expectations: run full test suite. The new tests should be lightweight and use temporary DBs / monkeypatching to avoid depending on large production DB.
+
+---
+
+## Commit & PR Strategy
+
+- Work in a feature branch `feat/mp-quiz-2026-03-24`.
+- Make small focused commits per task (messages suggested above). Each micro-task should be one commit.
+- PR organization:
+  - PR #1 (Batch 1): database.py changes + tests/test_match_mps.py — target only DB helpers and their unit tests. Keep this PR small so backend logic can be reviewed independently.
+  - PR #2 (Batch 2): explorer.py UI builder + tests/test_explorer_quiz.py — depends on PR #1; rebase after PR #1 merges or open as stacked PR (base=feat/mp-quiz-2026-03-24).
+  - PR #3 (Batch 3): integration+polish tests (tests/test_explorer_quiz_integration.py) and any small fixes discovered during integration testing.
+
+- Review checklist for each PR:
+  - Tests covering edge cases (zero-overlap MPs, empty inputs)
+  - DB queries use read_only DuckDB connections
+  - UI uses st.session_state and @st.cache_data appropriately
+  - No production DB writes, no schema changes
+
+---
+
+## Risks & Mitigations (short)
+
+- Performance: selecting motions across the entire motions table could be heavy. Mitigation: seed with top-N controversial motions and limit choose_discriminating_motions to motions that have mp_votes rows for the candidate MPs only.
+- Data quality: MPs with identical votes will remain indistinguishable — surface clearly to user. Tests include that scenario.
+
+---
+
+## Task checklist for implementers (copy/paste friendly)
+
+- [ ] Task 1.1: Modify database.py — implement match_mps_for_votes & choose_discriminating_motions. Add tests in tests/test_match_mps.py. (3.0–4.5h)
+- [ ] Task 1.2: Add this plan file. (0.25–0.5h)
+- [ ] Task 2.1: Modify explorer.py — add build_mp_quiz_tab and wire into tabs. Add tests in tests/test_explorer_quiz.py. (2.0–4.0h)
+- [ ] Task 3.1: Add integration test tests/test_explorer_quiz_integration.py to exercise quiz flow. (2.0–3.0h)
+
+---
+
+If you run into ambiguous input normalization details or DB edge-cases, follow the choices documented above (explicit normalization mapping, exclude zero-overlap MPs, use entropy scoring). If you encounter a blocker (e.g. missing mp_votes data in test fixtures), create small test fixtures using MotionDatabase.insert_motion and insert_mp_vote in the test setup.
+
+Good luck — keep PRs small and tests fast.