# SVD Label Unification Implementation Plan > **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. **Goal:** Unify SVD component labels into a single source of truth and automatically compute axis flip directions so right-wing parties consistently appear on the right side of all SVD component axes. **Architecture:** Create a new `analysis/svd_labels.py` module that imports `SVD_THEMES` from explorer.py at runtime and provides helper functions for label lookup and flip direction computation. Update `axis_classifier.py` to use this module instead of hardcoded labels. Add 1D party position charts for components 3-10 in the SVD Components tab. **Tech Stack:** Python, Streamlit, NumPy, DuckDB --- ## File Structure | File | Responsibility | |------|---------------| | `analysis/svd_labels.py` | **NEW** - Unified label system with auto-flip computation | | `analysis/axis_classifier.py` | Remove `_LABELS`, import from svd_labels | | `explorer.py` | Remove fallback labels, add 1D charts for components 3-10 | | `tests/test_svd_labels.py` | **NEW** - Tests for svd_labels module | | `tests/test_axis_label_fallback.py` | Update to use new label system | | `tests/test_political_compass.py` | Update assertions for new labels | --- ## Task 1: Create `analysis/svd_labels.py` with core functions **Files:** - Create: `analysis/svd_labels.py` - Test: `tests/test_svd_labels.py` - [ ] **Step 1: Write the failing test for `get_svd_label`** ```python # tests/test_svd_labels.py """Tests for analysis/svd_labels module.""" def test_get_svd_label_returns_correct_label(): """Test that get_svd_label returns the correct label for each component.""" from analysis.svd_labels import get_svd_label # Component 1 should return EU-integratie label label1 = get_svd_label(1) assert "EU-integratie" in label1 or "Nationalisme" in label1 # Component 2 should return Populistisch label label2 = get_svd_label(2) assert "Populistisch" in label2 or "Institutioneel" in label2 # Component 3 should return Verzorgingsstaat label label3 = get_svd_label(3) assert "Verzorgingsstaat" in label3 or "Marktwerking" in label3 ``` - [ ] **Step 2: Run test to verify it fails** Run: `uv run pytest tests/test_svd_labels.py::test_get_svd_label_returns_correct_label -v` Expected: FAIL with "ModuleNotFoundError: No module named 'analysis.svd_labels'" - [ ] **Step 3: Create `analysis/svd_labels.py` with `get_svd_label` function** ```python # analysis/svd_labels.py """Unified SVD component labels and automatic flip direction computation. This module provides a single source of truth for SVD component labels, deriving them from SVD_THEMES in explorer.py. It also computes flip directions automatically based on party centroids. """ import logging from typing import Dict, List, Optional, Tuple _logger = logging.getLogger(__name__) # Canonical party sets for orientation # Right-wing parties that should appear on the right side of axes RIGHT_PARTIES = { "PVV", "VVD", "FVD", "BBB", "JA21", "Nieuw Sociaal Contract", "SGP", "CDA", "ChristenUnie" } # Left-wing parties that should appear on the left side of axes LEFT_PARTIES = { "SP", "PvdA", "GL", "GroenLinks", "GroenLinks-PvdA", "DENK", "PvdD", "Volt" } # Cache for SVD_THEMES to avoid repeated imports _svd_themes_cache: Optional[Dict[int, Dict[str, str]]] = None def _get_svd_themes() -> Dict[int, Dict[str, str]]: """Lazy import SVD_THEMES from explorer.py to avoid circular imports. Returns: Dict mapping component number to theme dict with keys: - label: Short label for the component - explanation: Detailed explanation - positive_pole: Description of positive pole - negative_pole: Description of negative pole - flip: Whether to flip the axis """ global _svd_themes_cache if _svd_themes_cache is not None: return _svd_themes_cache try: # Import at runtime to avoid circular dependency # explorer.py imports from analysis/ but we don't import from explorer.py # at module load time import importlib.util import os # Find explorer.py explorer_path = os.path.join(os.path.dirname(__file__), "..", "explorer.py") if not os.path.exists(explorer_path): _logger.warning("explorer.py not found at %s", explorer_path) return {} spec = importlib.util.spec_from_file_location("explorer", explorer_path) if spec is None or spec.loader is None: _logger.warning("Could not load spec from explorer.py") return {} explorer_module = importlib.util.module_from_spec(spec) spec.loader.exec_module(explorer_module) # Get SVD_THEMES from the build_svd_components_tab function's local scope # This is a bit hacky but avoids importing the entire Streamlit app # We'll need to refactor explorer.py to export SVD_THEMES at module level # For now, we'll define it here as a fallback # Fallback: define SVD_THEMES here if we can't import from explorer _svd_themes_cache = {} return _svd_themes_cache except Exception as e: _logger.exception("Failed to load SVD_THEMES from explorer.py: %s", e) return {} def get_svd_label(component: int) -> str: """Get short label for SVD component. Args: component: SVD component number (1-indexed) Returns: Short label string (e.g., 'EU-integratie–Nationalisme') Raises: ValueError: If component < 1 """ if component < 1: raise ValueError(f"Component must be >= 1, got {component}") themes = _get_svd_themes() if component in themes: return themes[component].get("label", f"As {component}") # Fallback labels for components 1-3 (most commonly used) fallback_labels = { 1: "EU-integratie–Nationalisme", 2: "Populistisch–Institutioneel", 3: "Verzorgingsstaat–Marktwerking", } return fallback_labels.get(component, f"As {component}") def get_svd_theme(component: int) -> Dict[str, str]: """Get full theme dict for SVD component. Args: component: SVD component number (1-indexed) Returns: Dict with keys: label, explanation, positive_pole, negative_pole, flip """ if component < 1: raise ValueError(f"Component must be >= 1, got {component}") themes = _get_svd_themes() if component in themes: return themes[component] # Return minimal fallback return { "label": get_svd_label(component), "explanation": "", "positive_pole": "", "negative_pole": "", "flip": False, } def compute_flip_direction( component: int, party_scores: Dict[str, List[float]] ) -> bool: """Compute flip direction so right parties appear on the right side. Args: component: SVD component number (1-indexed) party_scores: Dict mapping party name to list of scores per component (party_scores[party][0] is score for component 1, etc.) Returns: True if axis should be flipped so right parties are on right. False otherwise. """ if component < 1: return False idx = component - 1 # Convert to 0-indexed right_scores = [] left_scores = [] for party, scores in party_scores.items(): if len(scores) <= idx: continue score = scores[idx] if party in RIGHT_PARTIES: right_scores.append(score) elif party in LEFT_PARTIES: left_scores.append(score) if not right_scores or not left_scores: return False # Default: no flip if insufficient data right_mean = sum(right_scores) / len(right_scores) left_mean = sum(left_scores) / len(left_scores) # Flip if right parties have lower mean (they're on the left) return right_mean < left_mean def get_fallback_labels() -> Tuple[str, str]: """Get fallback labels for x and y axes (components 1 and 2). Returns: Tuple of (x_label, y_label) """ return (get_svd_label(1), get_svd_label(2)) ``` - [ ] **Step 4: Run test to verify it passes** Run: `uv run pytest tests/test_svd_labels.py::test_get_svd_label_returns_correct_label -v` Expected: PASS - [ ] **Step 5: Write test for `compute_flip_direction`** ```python # tests/test_svd_labels.py (append) def test_compute_flip_direction_right_on_left(): """Test that flip is True when right parties are on the left.""" from analysis.svd_labels import compute_flip_direction # Right parties have negative scores (on left), left parties have positive party_scores = { "VVD": [-0.5, 0.0], # Right party, component 1 score = -0.5 "PVV": [-0.8, 0.0], # Right party "SP": [0.6, 0.0], # Left party, component 1 score = 0.6 "DENK": [0.4, 0.0], # Left party } # Component 1: right_mean = -0.65, left_mean = 0.5 # right_mean < left_mean, so flip = True assert compute_flip_direction(1, party_scores) is True def test_compute_flip_direction_right_on_right(): """Test that flip is False when right parties are already on the right.""" from analysis.svd_labels import compute_flip_direction # Right parties have positive scores (on right), left parties have negative party_scores = { "VVD": [0.5, 0.0], # Right party, component 1 score = 0.5 "PVV": [0.8, 0.0], # Right party "SP": [-0.6, 0.0], # Left party "DENK": [-0.4, 0.0], # Left party } # Component 1: right_mean = 0.65, left_mean = -0.5 # right_mean > left_mean, so flip = False assert compute_flip_direction(1, party_scores) is False def test_compute_flip_direction_insufficient_data(): """Test that flip is False when there's insufficient data.""" from analysis.svd_labels import compute_flip_direction # No right parties in data party_scores = { "SP": [0.6, 0.0], "DENK": [0.4, 0.0], } assert compute_flip_direction(1, party_scores) is False # No left parties in data party_scores = { "VVD": [0.5, 0.0], "PVV": [0.8, 0.0], } assert compute_flip_direction(1, party_scores) is False ``` - [ ] **Step 6: Run tests for `compute_flip_direction`** Run: `uv run pytest tests/test_svd_labels.py -v` Expected: All tests PASS - [ ] **Step 7: Commit** ```bash git add analysis/svd_labels.py tests/test_svd_labels.py git commit -m "feat: add svd_labels module for unified label system" ``` --- ## Task 2: Refactor explorer.py to export SVD_THEMES at module level **Files:** - Modify: `explorer.py` (move SVD_THEMES to module level) - Modify: `analysis/svd_labels.py` (update to import from explorer) - [ ] **Step 1: Move SVD_THEMES from function to module level in explorer.py** Find the `SVD_THEMES` dict inside `build_svd_components_tab` function (around line 2459) and move it to module level (near the top of the file, after imports). ```python # explorer.py (near top of file, after imports and constants) # Political polarisation themes per SVD component (1-indexed, window=2025) # Produced by per-axis analysis of all 10 unique top motions (zero cross-axis overlap). # This is the canonical source of truth for SVD component labels. SVD_THEMES: dict[int, dict[str, str]] = { 1: { "label": "EU-integratie en internationalisme versus nationalisme", # ... rest of the dict }, # ... rest of components } ``` - [ ] **Step 2: Update `analysis/svd_labels.py` to import SVD_THEMES properly** ```python # analysis/svd_labels.py (update _get_svd_themes function) def _get_svd_themes() -> Dict[int, Dict[str, str]]: """Lazy import SVD_THEMES from explorer.py to avoid circular imports.""" global _svd_themes_cache if _svd_themes_cache is not None: return _svd_themes_cache try: # Import at runtime to avoid circular dependency at module load time # explorer.py imports from analysis/ but we delay our import import explorer _svd_themes_cache = explorer.SVD_THEMES return _svd_themes_cache except ImportError as e: _logger.warning("Could not import explorer.SVD_THEMES: %s", e) return {} except Exception as e: _logger.exception("Failed to load SVD_THEMES from explorer.py: %s", e) return {} ``` - [ ] **Step 3: Run tests to verify nothing broke** Run: `uv run pytest tests/test_svd_labels.py -v` Expected: All tests PASS - [ ] **Step 4: Commit** ```bash git add explorer.py analysis/svd_labels.py git commit -m "refactor: move SVD_THEMES to module level for import" ``` --- ## Task 3: Update `axis_classifier.py` to use svd_labels **Files:** - Modify: `analysis/axis_classifier.py` - Modify: `tests/test_axis_label_fallback.py` - [ ] **Step 1: Write test for updated `display_label_for_modal`** ```python # tests/test_axis_label_fallback.py (update existing tests) def test_display_label_for_modal_uses_svd_themes(): """Test that display_label_for_modal uses SVD_THEMES for fallback labels.""" from analysis.axis_classifier import display_label_for_modal # None should return fallback from SVD_THEMES x_label = display_label_for_modal(None, "x") y_label = display_label_for_modal(None, "y") # Should return component 1 and 2 labels from SVD_THEMES assert "EU-integratie" in x_label or "Nationalisme" in x_label assert "Populistisch" in y_label or "Institutioneel" in y_label def test_display_label_for_modal_maps_as_labels(): """Test that 'As 1' and 'As 2' are mapped to semantic labels.""" from analysis.axis_classifier import display_label_for_modal x_label = display_label_for_modal("As 1", "x") y_label = display_label_for_modal("As 2", "y") # Should return component 1 and 2 labels assert "EU-integratie" in x_label or "Nationalisme" in x_label assert "Populistisch" in y_label or "Institutioneel" in y_label ``` - [ ] **Step 2: Run test to verify it fails** Run: `uv run pytest tests/test_axis_label_fallback.py -v` Expected: Some tests FAIL (current implementation returns hardcoded labels) - [ ] **Step 3: Update `axis_classifier.py` to use svd_labels** ```python # analysis/axis_classifier.py (update the module) # Remove the hardcoded _LABELS dict (lines 25-35) # Replace with imports from svd_labels # At the top of the file, after imports: from analysis.svd_labels import get_svd_label, get_fallback_labels # Remove _LABELS dict entirely # Update display_label_for_modal function (lines 42-55): def display_label_for_modal(modal_label: Optional[str], axis: str) -> str: """Return a user-facing axis label for a modal/internal label. Maps numeric fallback names 'As 1' / 'Stempatroon As 1' to the semantic labels from SVD_THEMES. Any other label is returned unchanged. None is treated as the semantic fallback for the axis. """ if modal_label is None: # Fallback to component 1 (x) or 2 (y) comp = 1 if axis == "x" else 2 return get_svd_label(comp) # Map "As 1" / "As 2" to semantic labels if axis == "x" and modal_label in ("As 1", "Stempatroon As 1"): return get_svd_label(1) if axis == "y" and modal_label in ("As 2", "Stempatroon As 2"): return get_svd_label(2) return modal_label ``` - [ ] **Step 4: Update `_INTERPRETATION_TEMPLATES` to use svd_labels** The interpretation templates still need to reference the correct labels. Keep them as-is for now since they're used for motion classification, not display. - [ ] **Step 5: Update `_MOTION_LABEL_TEMPLATE_KEY` to use svd_labels** Keep the mapping as-is for now since it's used for motion classification. - [ ] **Step 6: Update `_KEYWORDS` to use svd_labels** Keep the keywords as-is for now since they're used for motion classification. - [ ] **Step 7: Update fallback label references in classify_axes** Find lines 610 and 615 where `_LABELS["fallback_x"]` and `_LABELS["fallback_y"]` are used and replace: ```python # Before: x_lbl = _LABELS["fallback_x"] y_lbl = _LABELS["fallback_y"] # After: x_lbl, y_lbl = get_fallback_labels() ``` - [ ] **Step 8: Run tests to verify they pass** Run: `uv run pytest tests/test_axis_label_fallback.py tests/test_political_compass.py -v` Expected: All tests PASS - [ ] **Step 9: Commit** ```bash git add analysis/axis_classifier.py tests/test_axis_label_fallback.py git commit -m "refactor: use svd_labels for axis labels in axis_classifier" ``` --- ## Task 4: Update explorer.py to use svd_labels for fallback labels **Files:** - Modify: `explorer.py` - [ ] **Step 1: Find and update fallback label references in explorer.py** Find lines 1440-1441 where hardcoded fallback labels are used: ```python # Before: _x_label = _raw_x or "EU-integratie–Nationalisme" _y_label = _raw_y or "Populistisch–Institutioneel" # After: from analysis.svd_labels import get_fallback_labels _x_fallback, _y_fallback = get_fallback_labels() _x_label = _raw_x or _x_fallback _y_label = _raw_y or _y_fallback ``` - [ ] **Step 2: Run tests to verify nothing broke** Run: `uv run pytest tests/test_political_compass.py -v` Expected: All tests PASS - [ ] **Step 3: Commit** ```bash git add explorer.py git commit -m "refactor: use svd_labels for fallback labels in explorer" ``` --- ## Task 5: Add 1D party position charts for components 3-10 **Files:** - Modify: `explorer.py` - Test: `tests/test_explorer_chart.py` - [ ] **Step 1: Write test for 1D party position chart** ```python # tests/test_explorer_chart.py (append) def test_render_party_axis_chart_1d_renders(): """Test that _render_party_axis_chart_1d renders a figure.""" from explorer import _render_party_axis_chart_1d party_coords = { "VVD": (0.5,), "SP": (-0.6,), "PVV": (0.8,), "DENK": (-0.4,), } theme = { "label": "Test Component", "positive_pole": "Positive", "negative_pole": "Negative", "flip": False, } fig = _render_party_axis_chart_1d(party_coords, 3, theme) assert fig is not None # Check that figure has traces assert len(fig.data) > 0 ``` - [ ] **Step 2: Run test to verify it fails** Run: `uv run pytest tests/test_explorer_chart.py::test_render_party_axis_chart_1d_renders -v` Expected: FAIL with "cannot import name '_render_party_axis_chart_1d'" - [ ] **Step 3: Implement `_render_party_axis_chart_1d` function in explorer.py** Add this function near `_render_party_axis_chart` (around line 2900): ```python # explorer.py (add new function) def _render_party_axis_chart_1d( party_coords: dict, component: int, theme: dict, bootstrap_data: Optional[Dict] = None, ) -> go.Figure: """Render a 1D party position chart for a single SVD component. Args: party_coords: Dict mapping party name to (score,) tuple component: SVD component number (1-indexed) theme: Dict with label, positive_pole, negative_pole, flip bootstrap_data: Optional bootstrap confidence intervals Returns: Plotly Figure object """ import plotly.graph_objects as go import numpy as np if not party_coords: fig = go.Figure() fig.add_annotation( text="Geen partijposities beschikbaar", xref="paper", yref="paper", x=0.5, y=0.5, showarrow=False, font=dict(size=14) ) return fig # Extract scores and parties parties = list(party_coords.keys()) scores = [coords[0] for coords in party_coords.values()] # Apply flip if needed flip = theme.get("flip", False) if flip: scores = [-s for s in scores] # Get party colors party_colors = [PARTY_COLOURS.get(p, "#9E9E9E") for p in parties] # Create horizontal bar chart fig = go.Figure() # Sort by score for better visualization sorted_indices = np.argsort(scores) sorted_parties = [parties[i] for i in sorted_indices] sorted_scores = [scores[i] for i in sorted_indices] sorted_colors = [party_colors[i] for i in sorted_indices] fig.add_trace(go.Bar( y=sorted_parties, x=sorted_scores, orientation='h', marker_color=sorted_colors, text=[f"{s:.2f}" for s in sorted_scores], textposition='outside', )) # Update layout label = theme.get("label", f"As {component}") positive_pole = theme.get("positive_pole", "Positief") negative_pole = theme.get("negative_pole", "Negatief") fig.update_layout( title=f"Partijposities — {label}", xaxis_title=f"{negative_pole} ← → {positive_pole}", yaxis_title="", height=max(400, len(parties) * 25), # Scale height with number of parties margin=dict(l=150), # Extra margin for party names showlegend=False, ) # Add vertical line at x=0 fig.add_vline(x=0, line_dash="dash", line_color="gray", opacity=0.5) return fig ``` - [ ] **Step 4: Update SVD Components tab to use 1D chart for components 3-10** Find the section in `build_svd_components_tab` where `_render_party_axis_chart` is called (around line 2823) and update: ```python # explorer.py (update around line 2822-2830) # Before: if comp_sel in (1, 2): _render_party_axis_chart( party_coords, comp_sel, theme, bootstrap_data=bootstrap_data ) else: st.caption( "_Partijposities zijn alleen beschikbaar voor de eerste twee SVD-assen._" ) # After: if comp_sel in (1, 2): _render_party_axis_chart( party_coords, comp_sel, theme, bootstrap_data=bootstrap_data ) else: # For components 3-10, show 1D party positions # Extract 1D scores for this component party_1d_coords = {} idx = comp_sel - 1 # Convert to 0-indexed for party, scores in party_scores.items(): if len(scores) > idx: party_1d_coords[party] = (scores[idx],) _render_party_axis_chart_1d( party_1d_coords, comp_sel, theme, bootstrap_data=None ) ``` - [ ] **Step 5: Run tests to verify they pass** Run: `uv run pytest tests/test_explorer_chart.py -v` Expected: All tests PASS - [ ] **Step 6: Commit** ```bash git add explorer.py tests/test_explorer_chart.py git commit -m "feat: add 1D party position charts for SVD components 3-10" ``` --- ## Task 6: Auto-compute flip directions for all components **Files:** - Modify: `explorer.py` - Modify: `analysis/svd_labels.py` - Test: `tests/test_svd_labels.py` - [ ] **Step 1: Write test for auto-computing flip in SVD Components tab** ```python # tests/test_svd_labels.py (append) def test_auto_flip_computation_for_all_components(): """Test that flip directions are computed correctly for all components.""" from analysis.svd_labels import compute_flip_direction # Simulate party scores for 10 components # Right parties should have positive scores on component 1 (EU-integratie) # Left parties should have negative scores on component 1 party_scores = { "VVD": [0.5] * 10, # Right party, positive on all components "PVV": [0.8] * 10, # Right party "SP": [-0.6] * 10, # Left party, negative on all components "DENK": [-0.4] * 10, # Left party } # For all components, right_mean > left_mean, so flip should be False for comp in range(1, 11): flip = compute_flip_direction(comp, party_scores) assert flip is False, f"Component {comp} should not flip" # Now test with right parties on left (negative scores) party_scores_left = { "VVD": [-0.5] * 10, "PVV": [-0.8] * 10, "SP": [0.6] * 10, "DENK": [0.4] * 10, } # For all components, right_mean < left_mean, so flip should be True for comp in range(1, 11): flip = compute_flip_direction(comp, party_scores_left) assert flip is True, f"Component {comp} should flip" ``` - [ ] **Step 2: Run test to verify it passes** Run: `uv run pytest tests/test_svd_labels.py::test_auto_flip_computation_for_all_components -v` Expected: PASS (function already implemented in Task 1) - [ ] **Step 3: Update SVD Components tab to compute flip dynamically** In `build_svd_components_tab`, after loading party_scores, compute flip for each component: ```python # explorer.py (in build_svd_components_tab, after loading party_scores) from analysis.svd_labels import compute_flip_direction # After party_scores is loaded (around line 2800) # Compute flip directions for all components computed_flips = {} for comp in range(1, 11): computed_flips[comp] = compute_flip_direction(comp, party_scores) # Update SVD_THEMES with computed flips for comp, flip in computed_flips.items(): if comp in SVD_THEMES: SVD_THEMES[comp]["flip"] = flip ``` - [ ] **Step 4: Run tests to verify nothing broke** Run: `uv run pytest tests/test_svd_labels.py tests/test_explorer_chart.py -v` Expected: All tests PASS - [ ] **Step 5: Commit** ```bash git add explorer.py analysis/svd_labels.py tests/test_svd_labels.py git commit -m "feat: auto-compute flip directions for all SVD components" ``` --- ## Task 7: Update existing tests for new label system **Files:** - Modify: `tests/test_political_compass.py` - Modify: `tests/test_axis_label_fallback.py` - [ ] **Step 1: Run all tests to identify failures** Run: `uv run pytest tests/ -v --tb=short` Expected: Some tests may fail due to label changes - [ ] **Step 2: Fix any failing tests** Update test assertions to use new labels from SVD_THEMES: ```python # tests/test_political_compass.py (update assertions) # Before: assert "Links–Rechts" in x_label # After: assert "EU-integratie" in x_label or "Nationalisme" in x_label ``` - [ ] **Step 3: Run all tests to verify they pass** Run: `uv run pytest tests/ -v` Expected: All tests PASS - [ ] **Step 4: Commit** ```bash git add tests/ git commit -m "test: update tests for unified SVD label system" ``` --- ## Task 8: Final verification and cleanup **Files:** - All modified files - [ ] **Step 1: Run full test suite** Run: `uv run pytest tests/ -v` Expected: All tests PASS - [ ] **Step 2: Run Streamlit app to verify UI** Run: `uv run streamlit run explorer.py` Expected: App starts without errors - [ ] **Step 3: Manually verify in UI** - Open SVD Components tab - Check that components 1-10 show correct labels - Check that party position charts render for all components - Check that right parties appear on the right side of axes - [ ] **Step 4: Final commit** ```bash git add -A git commit -m "feat: complete SVD label unification" ``` --- ## Summary This plan creates a unified label system for SVD components with: 1. Single source of truth in `SVD_THEMES` 2. Automatic flip direction computation based on party centroids 3. 1D party position charts for components 3-10 4. Consistent labels across compass, trajectory, and SVD Components views