sgeboers
/
motief
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
27 Commits
1 Branch
0 Tags
6.0 MiB
 
 
 
Tag: Branch: Tree: eb73275f32
Branches Tags
${ item.name }
Create tag ${ searchTerm }
Create branch ${ searchTerm }
from 'eb73275f32'
${ noResults }
motief/thoughts/shared/plans/2026-03-24-welk-tweede-kame...

13 KiB
Raw Blame History

"Welk tweede kamerlid ben jij?" Implementation Plan

Goal: Add a Streamlit quiz tab that interactively asks the user motion (vote) questions and narrows the set of 2026 MPs to find the best-matching MP. Implement two DB helpers (matching + discriminating-motion selection), the UI builder and tab wiring, and tests. Minimal viable changes only — no UX bells & whistles.

Design: thoughts/shared/designs/2026-03-24-welk-tweede-kamerlid-ben-jij-design.md

Dependency Graph

Batch 1 (parallel): 1.1 [foundation - no deps], 1.2 (plan file) [none]
Batch 2 (parallel): 2.1 [explorer UI - depends: 1.1]
Batch 3 (parallel): 3.1 [integration tests - depends: 1.1,2.1]

Summary of implementation decisions (gap-filling)

  • MotionDatabase.match_mps_for_votes: implement as a read-only DuckDB-backed method on the existing MotionDatabase class (database.py). It accepts user_votes: Dict[int, str] where keys are motion ids and values are UI vote tokens. I will implement vote normalization inside the method (mapping UI tokens to canonical DB tokens) to avoid touching other modules. Rationale: keeps surface changes minimal and avoids creating new modules.

  • MotionDatabase.choose_discriminating_motions: implement in the same file. For a small candidate set (expected << 200 MPs), fetch mp_votes for candidate MPs across candidate motions (excluding already-answered motion ids). Score candidate motions by information-entropy of vote distribution among remaining candidates (higher entropy = better split). Ties broken by controversy_score then motion id.

  • Explorer UI changes: add build_mp_quiz_tab(db_path) to explorer.py and wire it into the tabs list and fallback radio. Use st.session_state['mp_quiz_votes'] to store answers as mapping str(motion_id)->UI token. Use @st.cache_data on any expensive DB-calls in the UI layer.

  • Vote token normalization: UI will present choices: "Voor", "Tegen", "Onthouden", "Afwezig / Geen stem". The DB stores lowercase tokens like 'voor', 'tegen', 'onthouden', 'afwezig'. match_mps_for_votes will normalize case and a small set of variants (e.g., 'Geen stem' -> 'afwezig', 'Abstain' -> 'onthouden') — explicit list included in tests.

BATCH 1: Foundation (parallel - N implementers)

All tasks in this batch have NO dependencies and can run simultaneously.

Task 1.1: Add DB helpers to MotionDatabase

File: database.py (modify existing) Test: tests/test_match_mps.py Depends: none

Description / Acceptance criteria:

  • Add two new public methods to MotionDatabase:

    • match_mps_for_votes(user_votes: Dict[int, str], limit: int = 50) -> List[Dict]

      • Returns an ordered list (desc by agreement_pct) of dicts with keys: mp_name, party, matched (int), overlap (int), agreement_pct (float 0-100).
      • Behavior: for each mp present in mp_votes for any of the provided motions compute:
        • overlap = number of motions where MP has a recorded vote AND the user provided a non-empty vote (i.e., not "Geen stem").
        • matched = number of those overlaps where normalized(mp_vote) == normalized(user_vote).
        • agreement_pct = matched / overlap * 100 rounded to 1 decimal. MPs with overlap==0 are excluded from the returned list.
      • Ordering: agreement_pct desc, then matched desc, then mp_name asc.

    • choose_discriminating_motions(candidates: List[str], excluded_motion_ids: List[int], k: int = 1) -> List[int]

      • For the provided candidate mp_names, compute vote distributions per motion (voor/tegen/onthouden/afwezig) excluding motion ids in excluded_motion_ids.
      • Score each motion by Shannon entropy of the distribution among the candidate MPs (treating 'afwezig' as a separate bucket). Higher entropy preferred.
      • Return top-k motion ids as a list, tiebreakers: higher controversy_score (motions table) then lower motion id.

Implementation notes & decisions:

  • Implement normalization inside these methods. Normalization mapping (DB vote -> canonical): map DB votes lowercased to one of {'voor','tegen','onthouden','afwezig'}. UI inputs (Voor/Tegen/Onthouden/Geen stem) normalized to these same tokens.
  • For performance, implement SQL queries that select mp_votes filtered by motion_id IN (...) and mp_name IN (candidates) and aggregate via GROUP BY mp_name and vote. For small candidate sets and a limited set of motion_ids this will be fast. If duckdb is not available, fall back to in-Python aggregates using the file-backed JSON format already present in MotionDatabase._init_database.
  • Add docstrings and basic parameter validation (raise ValueError for empty user_votes or empty candidates input). Tests will cover expected exceptions.

Test outline (tests/test_match_mps.py):

  • Setup: create a temporary MotionDatabase using a temp db_path (MotionDatabase.reset_database() can be used if duckdb available; otherwise use file-backed mode). Insert a small set of motions and mp_votes via insert_motion / insert_mp_vote. Create at least 3 MPs with overlapping but distinct vote patterns across 4-6 motions.
  • Tests:
    1. test_match_basic_counts: user_votes covering 3 motions returns expected matched/overlap/agreement_pct per MP.
    2. test_match_excludes_zero_overlap: MPs with no recorded votes for provided motions are excluded.
    3. test_choose_discriminating_motions_entropy_ranking: with a small candidate set, the chosen motion(s) split candidates as expected (assert returned motion id is one of known good splitters)
    4. test_invalid_input: calling match_mps_for_votes with empty user_votes raises ValueError.

Verify: pytest -q tests/test_match_mps.py

Commit message: feat(database): add match_mps_for_votes and choose_discriminating_motions

Estimated time: 3.0 - 4.5 hours

Task 1.2: Add plan file (this document)

File: thoughts/shared/plans/2026-03-24-welk-tweede-kamerlid-ben-jij-plan.md (this file) Test: none Depends: none

Description: Add the implementation plan (this document) to the repo to provide step-by-step microtasks to implementers. No tests.

Verify: visually review file in repo. No test run.

Commit message: docs(plans): add plan for 'Welk tweede kamerlid ben jij?'

Estimated time: 0.25 - 0.5 hours

BATCH 2: Core UI (parallel - depends on Batch 1)

All tasks in this batch assume the DB methods from Task 1.1 exist.

Task 2.1: Add Streamlit quiz tab & wiring

File: explorer.py (modify existing) Test: tests/test_explorer_quiz.py Depends: 1.1

Description / Acceptance criteria:

  • Add a function build_mp_quiz_tab(db_path: str) -> None placed near other build_*_tab functions (as described in the design, e.g., after build_svd_components_tab or near the top of the tab builders). The function must:
    • Render a short intro/instructions.
    • Load an initial pool of candidate motions using existing load_motions_df(db_path) and pick a seed set (top N by controversy_score). Decision: seed N = 8 (configurable constant in the function: SEED_MOTIONS = 8) — this is small and fast.
    • Present questions one at a time: show motion title + layman_explanation (if available) and a radio with choices: "Voor", "Tegen", "Onthouden", "Geen stem" and a "Skip"/"Niet zeker" optional button mapped to "Geen stem". Choice stored to st.session_state['mp_quiz_votes'] as mapping with keys str(motion_id) -> UI token.
    • After each answer, call MotionDatabase.match_mps_for_votes(user_votes) to fetch ranked candidates and display a small DataFrame (top 10) with columns: MP name, party, matched, overlap, agreement_pct. Use st.dataframe.
    • If more than 1 candidate remains with top agreement_pct tied, call MotionDatabase.choose_discriminating_motions(candidates, excluded_motion_ids) to pick the next question to ask and continue until one unique MP remains or choose_discriminating_motions returns an empty list (tie / indistinguishable). Cap total questions at 20 (SESS_CAP = 20).
    • When unique MP is found (agreement_pct == 100 and overlap>0 and only one MP with highest agreement), show final MP summary (name, party) and their matching motions count.
    • Use caching: wrap any repeated DB lookups (e.g., load_motions_df already cached) and mark heavy updates via @st.cache_data where appropriate.

Implementation notes & decisions:

  • Keep all UI state local to st.session_state with keys prefixed mp_quiz_ to avoid collisions.
  • Normalize UI tokens before sending to DB helper (but DB methods will also normalize; duplication is defensive).
  • Keep the UI function self-contained in explorer.py (do not create new modules for this minimal MVP).

Test outline (tests/test_explorer_quiz.py):

  • Use monkeypatching to inject a MotionDatabase mock into explorer module or run in a test DB using MotionDatabase with temp db_path. The test must be import-safe (explorer.py imports many heavy libs), so follow pattern used by existing tests/test_explorer_import.py: import the module and assert build_mp_quiz_tab exists and is callable.
  • Functional assertions:
    1. test_builder_exists: import explorer, assert callable(build_mp_quiz_tab)
    2. test_ui_state_update_simulation: simulate st.session_state by creating a fake session dict (use monkeypatch to set st.session_state to a dict-like object) and calling build_mp_quiz_tab with a small temp DB where motions and mp_votes are prepared. Assert that after calling the builder with pre-filled votes the DataFrame block would display ranked candidates (test inspects returned structure if builder returns it, or else monkeypatch MotionDatabase.match_mps_for_votes to verify it was called with expected mapping).

Verification: pytest -q tests/test_explorer_quiz.py

Commit message: feat(ui): add 'Welk tweede kamerlid ben jij?' tab and wiring in explorer.py

Estimated time: 2.0 - 4.0 hours

BATCH 3: Integration & Tests (parallel - depends on Batches 1+2)

Task 3.1: Add integration test for quiz flow

File: tests/test_explorer_quiz_integration.py Test: this file Depends: 1.1, 2.1

Description / Acceptance criteria:

  • Create an end-to-end-ish headless test that:
    • Sets up a temporary MotionDatabase instance (temp file path) and inserts a small controlled dataset: ~6 motions, 4 MPs with distinct votes.
    • Calls build_mp_quiz_tab via explorer with monkeypatched st.session_state (or with a minimal wrapper) and simulates a sequence of user answers by pre-populating st.session_state['mp_quiz_votes'].
    • Asserts that final candidate set matches expectations: either a unique MP (when answers match exactly one MP) or that the function properly identifies indistinguishable MPs (when two MPs have identical votes).

Testing details & choices:

  • Avoid launching Streamlit server; tests only import explorer module and call the builder function in the same way other explorer tests do. Use monkeypatch to stub expensive functions (plotly, query_similar) where required.

Verify: pytest -q tests/test_explorer_quiz_integration.py

Commit message: test(ui): add integration tests for mp quiz tab flow

Estimated time: 2.0 - 3.0 hours

Verification & CI

  • Local verification commands (per task) use pytest. Example:

    • pytest -q tests/test_match_mps.py
    • pytest -q tests/test_explorer_quiz.py
    • pytest -q tests/test_explorer_quiz_integration.py

  • CI expectations: run full test suite. The new tests should be lightweight and use temporary DBs / monkeypatching to avoid depending on large production DB.

Commit & PR Strategy

  • Work in a feature branch feat/mp-quiz-2026-03-24.

  • Make small focused commits per task (messages suggested above). Each micro-task should be one commit.

  • PR organization:

    • PR #1 (Batch 1): database.py changes + tests/test_match_mps.py — target only DB helpers and their unit tests. Keep this PR small so backend logic can be reviewed independently.
    • PR #2 (Batch 2): explorer.py UI builder + tests/test_explorer_quiz.py — depends on PR #1; rebase after PR #1 merges or open as stacked PR (base=feat/mp-quiz-2026-03-24).
    • PR #3 (Batch 3): integration+polish tests (tests/test_explorer_quiz_integration.py) and any small fixes discovered during integration testing.

  • Review checklist for each PR:

    • Tests covering edge cases (zero-overlap MPs, empty inputs)
    • DB queries use read_only DuckDB connections
    • UI uses st.session_state and @st.cache_data appropriately
    • No production DB writes, no schema changes

Risks & Mitigations (short)

  • Performance: selecting motions across the entire motions table could be heavy. Mitigation: seed with top-N controversial motions and limit choose_discriminating_motions to motions that have mp_votes rows for the candidate MPs only.
  • Data quality: MPs with identical votes will remain indistinguishable — surface clearly to user. Tests include that scenario.

Task checklist for implementers (copy/paste friendly)

  • Task 1.1: Modify database.py — implement match_mps_for_votes & choose_discriminating_motions. Add tests in tests/test_match_mps.py. (3.0–4.5h)
  • Task 1.2: Add this plan file. (0.25–0.5h)
  • Task 2.1: Modify explorer.py — add build_mp_quiz_tab and wire into tabs. Add tests in tests/test_explorer_quiz.py. (2.0–4.0h)
  • Task 3.1: Add integration test tests/test_explorer_quiz_integration.py to exercise quiz flow. (2.0–3.0h)

If you run into ambiguous input normalization details or DB edge-cases, follow the choices documented above (explicit normalization mapping, exclude zero-overlap MPs, use entropy scoring). If you encounter a blocker (e.g. missing mp_votes data in test fixtures), create small test fixtures using MotionDatabase.insert_motion and insert_mp_vote in the test setup.

Good luck — keep PRs small and tests fast.