diff --git a/docs/superpowers/plans/2026-03-28-compass-ui-improvements.md b/docs/superpowers/plans/2026-03-28-compass-ui-improvements.md new file mode 100644 index 0000000..db84715 --- /dev/null +++ b/docs/superpowers/plans/2026-03-28-compass-ui-improvements.md @@ -0,0 +1,643 @@ +# Compass UI Improvements Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Fix three independent UI issues in the political compass: (1) update stale axis 3/4/5 descriptions in SVD_THEMES, (2) fix broken Y-axis direction arrows, (3) add voting discipline section below compass. + +**Architecture:** All changes are confined to `explorer.py`. No new files. No schema changes. The discipline helper reads `mp_votes` read-only via DuckDB. Tests for the discipline function live in `tests/test_political_compass.py`. + +**Tech Stack:** Python, Streamlit, Plotly Express/Graph Objects, DuckDB, pytest (run via `uv run pytest`) + +--- + +## File Map + +| File | Change | +|------|--------| +| `explorer.py` | Update SVD_THEMES axes 3–5; fix Y-axis labels in two px.scatter calls and one update_layout; add `compute_party_discipline`; add discipline rendering in `build_compass_tab` | +| `tests/test_political_compass.py` | Add tests for `compute_party_discipline` | + +--- + +## Task 1: Update SVD_THEMES axes 3, 4, 5 + +**Files:** +- Modify: `explorer.py:1156–1204` + +These descriptions were written for an earlier dataset. The new text reflects stable multi-year patterns (not a single year's snapshot). The `flip` booleans are unchanged. + +**New text for axis 3** (flip=True — currently "Sociaal-economisch links versus marktliberaal en landelijk rechts"): + +The positive pole consistently shows SP, PvdD, GL-PvdA on social welfare motions; the negative pole shows VVD on market-oriented motions. But PVV also appears prominently positive (anti-establishment spending motions), meaning this is not a clean left-right economic axis — it's more accurately described as *state intervention versus market liberalism*, with populist anti-establishment motions on the same side as the socialist left. + +**New text for axis 4** (flip=True — currently "Christelijk-sociaal centrum versus populistisch-soevereinistisch"): + +NSC, SGP, CU and CDA consistently dominate the positive pole; VVD and GL-PvdA the negative. FVD scores near zero across years. The axis captures *religious-conservative institutionalism vs secular liberalism*, not populism vs mainstream. + +**New text for axis 5** (flip=False — currently "Christelijk-conservatief en ruraal sociaal versus seculier-progressief"): + +CDA, CU, SGP score positive; SP, PvdD, DENK score negative. D66 tends positive (not strongly negative), and NSC tends negative. The stable pattern is *established-institutional vs protest-populist* rather than a religious/secular split. + +- [ ] **Step 1: Replace axis 3 entry in `explorer.py`** + +In `explorer.py`, replace lines 1156–1171: + +```python + 3: { + "label": "Staatsingrijpen en publieke sector versus marktliberalisme", + "explanation": ( + "Deze as weerspiegelt de spanning tussen staatsingrijpen en marktliberalisme. " + "Aan de positieve kant staan SP-moties die bezuinigingen op zorg en gemeentefonds " + "willen terugdraaien, winstuitkeringen in de zorg verbieden en publieke controle " + "over fusies eisen. Ook PVV stemt positief — niet vanuit sociaal ideaal maar vanuit " + "anti-establishment populisme dat neigt naar overheidsinterventie voor de eigen achterban. " + "Aan de negatieve kant staan VVD-moties over marktwerking en deregulering, en NSC- en " + "BBB-moties met een agrarisch-marktgericht karakter. " + "Samengevat: de as scheidt voorstanders van staatsingrijpen (links én populistisch-rechts) " + "van marktliberalen en agrarisch-rechts." + ), + "positive_pole": "Staatsingrijpen: SP, PvdD, GL-PvdA, PVV (populistisch)", + "negative_pole": "Marktliberaal en agrarisch-rechts: VVD, NSC, BBB", + "flip": True, + }, +``` + +- [ ] **Step 2: Replace axis 4 entry in `explorer.py`** + +Replace lines 1172–1188: + +```python + 4: { + "label": "Christelijk-conservatief institutionalisme versus seculier-liberalisme", + "explanation": ( + "Deze as scheidt christelijk-conservatieve partijen die hechten aan traditionele " + "instituties en religieuze waarden (NSC, SGP, CU, CDA) van seculier-liberale partijen " + "die nadruk leggen op individuele rechten en internationale openheid (VVD, GL-PvdA). " + "CU-moties over vaderbetrokkenheid, huwelijksrecht en internationale samenwerking staan " + "aan de positieve kant; VVD-moties over marktregulering en D66-moties over internationale " + "verdragen aan de negatieve kant. FVD scoort dicht bij nul — het past noch in het " + "christelijk-conservatieve noch in het seculier-liberale kamp op deze as. " + "Dit is geen populisme-as maar een religieus-institutionele breuklijn." + ), + "positive_pole": "Christelijk-conservatief institutioneel: NSC, SGP, CU, CDA", + "negative_pole": "Seculier-liberaal: VVD, GL-PvdA, D66", + "flip": True, + }, +``` + +- [ ] **Step 3: Replace axis 5 entry in `explorer.py`** + +Replace lines 1189–1204: + +```python + 5: { + "label": "Gevestigd-institutioneel versus protest en populistisch", + "explanation": ( + "Deze as scheidt gevestigde centrumpartijen die vertrouwen op bestaande instituties " + "(CDA, CU, SGP, D66) van protest- en populistische partijen die dat vertrouwen " + "afwijzen (SP, PvdD, DENK, NSC). CDA-moties over vrijwilligers in schuldhulp, " + "maatschappelijke diensttijd en WW-hervorming staan aan de positieve kant. " + "SP- en PvdD-moties over meerouderschap, abortusrecht en buitenlandse beïnvloeding " + "staan aan de negatieve kant. NSC scoort negatief — ondanks zijn christelijk-conservatieve " + "karakter op andere assen is het hier een protest­partij die systeemkritiek uitdraagt. " + "D66 scoort licht positief, consistent met zijn institutionele en pro-EU profiel." + ), + "positive_pole": "Gevestigd-institutioneel: CDA, CU, SGP, D66", + "negative_pole": "Protest en populistisch: SP, PvdD, DENK, NSC", + "flip": False, + }, +``` + +- [ ] **Step 4: Run tests to confirm nothing broken** + +```bash +uv run pytest tests/test_political_compass.py -v +``` + +Expected: all 3 tests PASS (these tests don't touch SVD_THEMES). + +- [ ] **Step 5: Commit** + +```bash +git add explorer.py +git commit -m "fix: update SVD_THEMES axes 3-5 descriptions to reflect stable multi-year patterns" +``` + +--- + +## Task 2: Fix Y-axis direction indicators in compass and trajectories + +**Files:** +- Modify: `explorer.py:810–812` (partijen scatter labels) +- Modify: `explorer.py:830` (kamerleden scatter labels) +- Modify: `explorer.py:833–838` (compass update_layout — add helper call) +- Modify: `explorer.py:921–927` (trajectories update_layout) + +Plotly rotates Y-axis titles 90° counter-clockwise, so `↑` and `↓` in the title string point sideways. Fix: strip arrows from the axis title; add two `fig.add_annotation` calls to place `▲ Progressief` at the top and `▼ Conservatief` at the bottom of the chart area using `xref="paper", yref="paper"`. + +- [ ] **Step 1: Add `_add_y_direction_annotations` helper near top of `build_compass_tab`** + +Add this function just before `build_compass_tab` (after line 692, before line 694): + +```python +def _add_y_direction_annotations(fig: go.Figure) -> None: + """Add ▲ Progressief / ▼ Conservatief labels above and below the Y axis.""" + common = dict( + xref="paper", + yref="paper", + x=-0.07, + showarrow=False, + font=dict(size=11, color="#666666"), + ) + fig.add_annotation(**common, y=1.02, text="▲ Progressief", xanchor="center") + fig.add_annotation(**common, y=-0.06, text="▼ Conservatief", xanchor="center") +``` + +- [ ] **Step 2: Update labels in the "Partijen" scatter (line 810–814)** + +Change: +```python + labels={ + "x": "Links ← → Rechts", + "y": "Progressief ↑ / Conservatief ↓", + "n": "Kamerleden", + }, +``` + +To: +```python + labels={ + "x": "Links ← → Rechts", + "y": "Progressief / Conservatief", + "n": "Kamerleden", + }, +``` + +- [ ] **Step 3: Update labels in the "Kamerleden" scatter (line 830)** + +Change: +```python + labels={"x": "Links ← → Rechts", "y": "Progressief ↑ / Conservatief ↓"}, +``` + +To: +```python + labels={"x": "Links ← → Rechts", "y": "Progressief / Conservatief"}, +``` + +- [ ] **Step 4: Call the annotation helper after `fig.update_layout` in `build_compass_tab` (after line 838)** + +Change: +```python + fig.update_layout( + height=600, + legend_title_text="Partij", + xaxis={"range": [-1, 1]}, + yaxis={"range": [-0.6, 0.6]}, + ) + + with col1: + st.plotly_chart(fig, use_container_width=True) +``` + +To: +```python + fig.update_layout( + height=600, + legend_title_text="Partij", + xaxis={"range": [-1, 1]}, + yaxis={"range": [-0.6, 0.6]}, + ) + _add_y_direction_annotations(fig) + + with col1: + st.plotly_chart(fig, use_container_width=True) +``` + +- [ ] **Step 5: Fix trajectories Y-axis title (line 924) and add annotation** + +Change `build_trajectories_tab` update_layout block: +```python + fig.update_layout( + title="Partij trajectories", + xaxis_title="Links ← → Rechts", + yaxis_title="Progressief ↑ / Conservatief ↓", + height=600, + legend_title_text="Partij", + ) + st.plotly_chart(fig, use_container_width=True) +``` + +To: +```python + fig.update_layout( + title="Partij trajectories", + xaxis_title="Links ← → Rechts", + yaxis_title="Progressief / Conservatief", + height=600, + legend_title_text="Partij", + ) + _add_y_direction_annotations(fig) + st.plotly_chart(fig, use_container_width=True) +``` + +- [ ] **Step 6: Run tests** + +```bash +uv run pytest tests/test_political_compass.py -v +``` + +Expected: all 3 tests PASS. + +- [ ] **Step 7: Commit** + +```bash +git add explorer.py +git commit -m "fix: replace sideways Y-axis arrows with proper top/bottom annotations" +``` + +--- + +## Task 3: Add voting discipline section below compass + +**Files:** +- Modify: `explorer.py` — add `compute_party_discipline` function; add rendering block in `build_compass_tab` +- Modify: `tests/test_political_compass.py` — add two tests + +### 3a: Write the failing tests first + +- [ ] **Step 1: Add tests to `tests/test_political_compass.py`** + +Append at the end of `tests/test_political_compass.py`: + +```python +# --------------------------------------------------------------------------- +# Tests for compute_party_discipline +# --------------------------------------------------------------------------- + + +def _make_mp_votes_db(): + """Create an in-memory DuckDB with mp_votes fixture data. + + 6 motions, 2 parties (SP, VVD), each with 4 MPs. + SP is perfectly disciplined (all 4 vote the same each time). + VVD has 1 dissident on 2 of 6 motions → Rice index = (4+4+4+4+3+3)/6/4 ≈ 0.917. + Dates span 2023-01-01 to 2023-12-31. + """ + import duckdb + + conn = duckdb.connect(":memory:") + conn.execute(""" + CREATE TABLE mp_votes ( + id INTEGER, + motion_id VARCHAR, + mp_name VARCHAR, + party VARCHAR, + vote VARCHAR, + date DATE, + created_at TIMESTAMP + ) + """) + rows = [] + # motions 1-6, dates in 2023 + dates = [ + "2023-01-10", + "2023-03-15", + "2023-05-20", + "2023-07-25", + "2023-09-30", + "2023-11-05", + ] + sp_mps = ["Janssen, A.", "Pietersen, B.", "Willemsen, C.", "Hendriksen, D."] + vvd_mps = ["Adams, E.", "Bakker, F.", "Claassen, G.", "Dekker, H."] + for i, date in enumerate(dates, start=1): + m_id = f"M{i:03d}" + # SP: all vote 'voor' every motion (perfectly disciplined) + for mp in sp_mps: + rows.append((i * 10 + 1, m_id, mp, "SP", "voor", date, "2023-01-01")) + # VVD: motions 5 and 6 have one dissident (votes 'tegen' while others vote 'voor') + if i <= 4: + for mp in vvd_mps: + rows.append((i * 10 + 2, m_id, mp, "VVD", "voor", date, "2023-01-01")) + else: + for mp in vvd_mps[:3]: + rows.append((i * 10 + 2, m_id, mp, "VVD", "voor", date, "2023-01-01")) + rows.append((i * 10 + 3, m_id, vvd_mps[3], "VVD", "tegen", date, "2023-01-01")) + conn.executemany( + "INSERT INTO mp_votes VALUES (?, ?, ?, ?, ?, ?, ?)", rows + ) + return conn + + +def test_compute_party_discipline_basic(monkeypatch): + """compute_party_discipline returns correct Rice index for fixture data.""" + import duckdb as _duckdb + + fixture_conn = _make_mp_votes_db() + + monkeypatch.setattr( + _duckdb, "connect", lambda path, **kw: fixture_conn + ) + + # Import after monkeypatch so explorer can be imported without Streamlit crashing + import importlib + import sys + + # explorer imports streamlit — provide a minimal stub if not already stubbed + if "streamlit" not in sys.modules: + import types + st_stub = types.ModuleType("streamlit") + st_stub.cache_data = lambda **kw: (lambda f: f) + sys.modules["streamlit"] = st_stub + + # We need the function directly; import the module + import explorer as _explorer + importlib.reload(_explorer) + + df = _explorer.compute_party_discipline( + db_path="dummy", + start_date="2023-01-01", + end_date="2023-12-31", + ) + + assert not df.empty + assert set(df.columns) >= {"party", "n_motions", "discipline"} + + sp_row = df[df["party"] == "SP"].iloc[0] + vvd_row = df[df["party"] == "VVD"].iloc[0] + + assert sp_row["n_motions"] == 6 + assert sp_row["discipline"] == pytest.approx(1.0, abs=1e-6) + + assert vvd_row["n_motions"] == 6 + # 4 motions fully disciplined (4/4=1.0), 2 motions with one dissident (3/4=0.75) + expected_vvd = (4 * 1.0 + 2 * 0.75) / 6 + assert vvd_row["discipline"] == pytest.approx(expected_vvd, abs=1e-4) + + # All values in [0, 1] + assert (df["discipline"] >= 0).all() and (df["discipline"] <= 1).all() + + +def test_compute_party_discipline_empty_range(monkeypatch): + """Returns empty DataFrame when no motions fall in the date range.""" + import duckdb as _duckdb + + fixture_conn = _make_mp_votes_db() + monkeypatch.setattr(_duckdb, "connect", lambda path, **kw: fixture_conn) + + import importlib, sys + + if "streamlit" not in sys.modules: + import types + st_stub = types.ModuleType("streamlit") + st_stub.cache_data = lambda **kw: (lambda f: f) + sys.modules["streamlit"] = st_stub + + import explorer as _explorer + importlib.reload(_explorer) + + df = _explorer.compute_party_discipline( + db_path="dummy", + start_date="2000-01-01", + end_date="2000-12-31", + ) + + assert df.empty +``` + +- [ ] **Step 2: Run the failing tests** + +```bash +uv run pytest tests/test_political_compass.py::test_compute_party_discipline_basic tests/test_political_compass.py::test_compute_party_discipline_empty_range -v +``` + +Expected: FAIL with `AttributeError: module 'explorer' has no attribute 'compute_party_discipline'` + +### 3b: Implement `compute_party_discipline` + +- [ ] **Step 3: Add `compute_party_discipline` to `explorer.py`** + +Add this function after the `load_active_mps` function (find it, then place `compute_party_discipline` immediately after). The function must be a plain function (not decorated with `@st.cache_data`) so tests can monkeypatch `duckdb.connect`. + +```python +def compute_party_discipline( + db_path: str, + start_date: str, + end_date: str, +) -> pd.DataFrame: + """Compute per-party voting discipline (Rice index) for roll-call votes in a date range. + + Only individual MP vote rows are used (mp_name LIKE '%,%'). + Returns a DataFrame with columns [party, n_motions, discipline] sorted by discipline ascending. + Returns an empty DataFrame if fewer than 1 qualifying motion exists or on any DB error. + + Rice index per motion per party = fraction of party MPs voting with the party majority. + The per-party score is the average Rice index across all motions in the date range. + """ + try: + conn = duckdb.connect(db_path, read_only=True) + result = conn.execute( + """ + WITH individual_votes AS ( + -- Only individual MP rows (mp_name contains a comma, e.g. "Janssen, A.") + SELECT + motion_id, + party, + LOWER(vote) AS vote + FROM mp_votes + WHERE mp_name LIKE '%,%' + AND date >= CAST(? AS DATE) + AND date <= CAST(? AS DATE) + AND vote IN ('voor', 'tegen', 'afwezig', 'onthouden') + ), + vote_counts AS ( + -- Count each vote token per (motion, party) + SELECT + motion_id, + party, + vote, + COUNT(*) AS cnt + FROM individual_votes + GROUP BY motion_id, party, vote + ), + majority_vote AS ( + -- Determine the majority vote token per (motion, party) + SELECT + motion_id, + party, + FIRST(vote ORDER BY cnt DESC, vote ASC) AS maj_vote, + SUM(cnt) AS total_mp_votes + FROM vote_counts + GROUP BY motion_id, party + ), + rice_per_motion AS ( + -- Rice index: fraction voting with majority + SELECT + mv.motion_id, + mv.party, + SUM(CASE WHEN vc.vote = mv.maj_vote THEN vc.cnt ELSE 0 END) + * 1.0 / mv.total_mp_votes AS rice + FROM majority_vote mv + JOIN vote_counts vc + ON mv.motion_id = vc.motion_id AND mv.party = vc.party + GROUP BY mv.motion_id, mv.party, mv.total_mp_votes + ) + SELECT + party, + COUNT(DISTINCT motion_id) AS n_motions, + AVG(rice) AS discipline + FROM rice_per_motion + GROUP BY party + ORDER BY discipline ASC + """, + [start_date, end_date], + ).fetchdf() + conn.close() + return result + except Exception as exc: + logger.warning("compute_party_discipline failed: %s", exc) + return pd.DataFrame(columns=["party", "n_motions", "discipline"]) +``` + +- [ ] **Step 4: Run the tests to confirm they pass** + +```bash +uv run pytest tests/test_political_compass.py::test_compute_party_discipline_basic tests/test_political_compass.py::test_compute_party_discipline_empty_range -v +``` + +Expected: both PASS. + +- [ ] **Step 5: Run all compass tests** + +```bash +uv run pytest tests/test_political_compass.py -v +``` + +Expected: all 5 tests PASS. + +### 3c: Render discipline section in `build_compass_tab` + +- [ ] **Step 6: Add `_window_to_dates` helper just before `build_compass_tab`** + +Add this function just before `build_compass_tab` (around line 692, after the `_add_y_direction_annotations` helper added in Task 2): + +```python +def _window_to_dates(window_id: str) -> tuple[str, str]: + """Return (start_date, end_date) ISO strings for a given window_id. + + Annual windows like '2024' → ('2024-01-01', '2024-12-31'). + 'current_parliament' → ('2023-11-22', '2099-12-31') (2023 formation date, open end). + Unknown formats → ('2000-01-01', '2099-12-31') (effectively all time). + """ + if window_id == "current_parliament": + return ("2023-11-22", "2099-12-31") + if re.fullmatch(r"\d{4}", window_id): + return (f"{window_id}-01-01", f"{window_id}-12-31") + # Quarterly e.g. '2020-Q3' → 2020-07-01 to 2020-09-30 + m = re.fullmatch(r"(\d{4})-Q([1-4])", window_id) + if m: + year, q = int(m.group(1)), int(m.group(2)) + starts = {1: "01-01", 2: "04-01", 3: "07-01", 4: "10-01"} + ends = {1: "03-31", 2: "06-30", 3: "09-30", 4: "12-31"} + return (f"{year}-{starts[q]}", f"{year}-{ends[q]}") + return ("2000-01-01", "2099-12-31") +``` + +- [ ] **Step 7: Add discipline rendering after `st.plotly_chart` in `build_compass_tab`** + +The current end of `build_compass_tab` is (around line 840–841): + +```python + with col1: + st.plotly_chart(fig, use_container_width=True) +``` + +Add the discipline section immediately after (still inside the function, but outside the `with col1:` block): + +```python + # --- Voting discipline section --- + _MIN_MOTIONS_FOR_DISCIPLINE = 5 + start_date, end_date = _window_to_dates(window_idx) + disc_df = compute_party_discipline(db_path, start_date, end_date) + + st.subheader("Stemgedrag cohesie") + if disc_df.empty or disc_df["n_motions"].max() < _MIN_MOTIONS_FOR_DISCIPLINE: + st.caption( + "Te weinig hoofdelijke stemmingen in dit venster voor een cohesieanalyse." + ) + else: + # Filter to parties that appear in the compass + compass_parties = set(df_pos["party"].unique()) + disc_df = disc_df[disc_df["party"].isin(compass_parties)].copy() + + if disc_df.empty: + st.caption("Geen overlappende partijen tussen kompas en stemmingsdata.") + else: + disc_df["discipline_pct"] = (disc_df["discipline"] * 100).round(1) + disc_df["party_label"] = disc_df.apply( + lambda r: f"{r['party']} ({int(r['n_motions'])} moties)", axis=1 + ) + + bar_fig = px.bar( + disc_df.sort_values("discipline"), + x="discipline_pct", + y="party_label", + orientation="h", + color="discipline_pct", + color_continuous_scale="RdYlGn", + range_color=[80, 100], + labels={"discipline_pct": "Cohesie (%)", "party_label": "Partij"}, + title="Cohesie bij hoofdelijke stemmingen", + ) + bar_fig.update_layout( + height=max(300, len(disc_df) * 35 + 80), + showlegend=False, + coloraxis_showscale=False, + yaxis_title="", + ) + st.plotly_chart(bar_fig, use_container_width=True) + + # Extremes table + top3 = disc_df.nlargest(3, "discipline")[["party", "discipline_pct", "n_motions"]] + bot3 = disc_df.nsmallest(3, "discipline")[["party", "discipline_pct", "n_motions"]] + col_a, col_b = st.columns(2) + with col_a: + st.markdown("**Meest eensgezind**") + st.dataframe( + top3.rename(columns={"party": "Partij", "discipline_pct": "Cohesie (%)", "n_motions": "Moties"}), + hide_index=True, + use_container_width=True, + ) + with col_b: + st.markdown("**Meest verdeeld**") + st.dataframe( + bot3.rename(columns={"party": "Partij", "discipline_pct": "Cohesie (%)", "n_motions": "Moties"}), + hide_index=True, + use_container_width=True, + ) +``` + +- [ ] **Step 8: Run all tests** + +```bash +uv run pytest tests/test_political_compass.py -v +``` + +Expected: all 5 tests PASS. + +- [ ] **Step 9: Commit** + +```bash +git add explorer.py tests/test_political_compass.py +git commit -m "feat: add voting discipline section below political compass" +``` + +--- + +## Self-Review Checklist + +- [x] **Spec coverage:** Task 1 → SVD_THEMES axes 3–5. Task 2 → Y-axis arrows. Task 3 → discipline function + rendering. All three design requirements covered. +- [x] **Placeholder scan:** No TBD or TODO. All code blocks are complete. +- [x] **Type consistency:** `compute_party_discipline` returns `pd.DataFrame` with columns `["party", "n_motions", "discipline"]` — referenced consistently in tests and rendering code. `_window_to_dates` returns `tuple[str, str]` — used as `start_date, end_date` in rendering. +- [x] **Test isolation:** Tests monkeypatch `duckdb.connect` to return in-memory DB; tests add a minimal `streamlit` stub to avoid import errors. Both patterns match existing test style in the file. +- [x] **Edge case:** `_MIN_MOTIONS_FOR_DISCIPLINE = 5` guard ensures the section degrades gracefully for sparse windows. Empty DataFrame from `compute_party_discipline` is also handled.