27 KiB
SVD Label Unification Implementation Plan
For agentic workers: REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (
- [ ]) syntax for tracking.
Goal: Unify SVD component labels into a single source of truth and automatically compute axis flip directions so right-wing parties consistently appear on the right side of all SVD component axes.
Architecture: Create a new analysis/svd_labels.py module that imports SVD_THEMES from explorer.py at runtime and provides helper functions for label lookup and flip direction computation. Update axis_classifier.py to use this module instead of hardcoded labels. Add 1D party position charts for components 3-10 in the SVD Components tab.
Tech Stack: Python, Streamlit, NumPy, DuckDB
File Structure
| File | Responsibility |
|---|---|
analysis/svd_labels.py |
NEW - Unified label system with auto-flip computation |
analysis/axis_classifier.py |
Remove _LABELS, import from svd_labels |
explorer.py |
Remove fallback labels, add 1D charts for components 3-10 |
tests/test_svd_labels.py |
NEW - Tests for svd_labels module |
tests/test_axis_label_fallback.py |
Update to use new label system |
tests/test_political_compass.py |
Update assertions for new labels |
Task 1: Create analysis/svd_labels.py with core functions
Files:
-
Create:
analysis/svd_labels.py -
Test:
tests/test_svd_labels.py -
Step 1: Write the failing test for
get_svd_label
# tests/test_svd_labels.py
"""Tests for analysis/svd_labels module."""
def test_get_svd_label_returns_correct_label():
"""Test that get_svd_label returns the correct label for each component."""
from analysis.svd_labels import get_svd_label
# Component 1 should return EU-integratie label
label1 = get_svd_label(1)
assert "EU-integratie" in label1 or "Nationalisme" in label1
# Component 2 should return Populistisch label
label2 = get_svd_label(2)
assert "Populistisch" in label2 or "Institutioneel" in label2
# Component 3 should return Verzorgingsstaat label
label3 = get_svd_label(3)
assert "Verzorgingsstaat" in label3 or "Marktwerking" in label3
- Step 2: Run test to verify it fails
Run: uv run pytest tests/test_svd_labels.py::test_get_svd_label_returns_correct_label -v
Expected: FAIL with "ModuleNotFoundError: No module named 'analysis.svd_labels'"
- Step 3: Create
analysis/svd_labels.pywithget_svd_labelfunction
# analysis/svd_labels.py
"""Unified SVD component labels and automatic flip direction computation.
This module provides a single source of truth for SVD component labels,
deriving them from SVD_THEMES in explorer.py. It also computes flip
directions automatically based on party centroids.
"""
import logging
from typing import Dict, List, Optional, Tuple
_logger = logging.getLogger(__name__)
# Canonical party sets for orientation
# Right-wing parties that should appear on the right side of axes
RIGHT_PARTIES = {
"PVV", "VVD", "FVD", "BBB", "JA21",
"Nieuw Sociaal Contract", "SGP", "CDA", "ChristenUnie"
}
# Left-wing parties that should appear on the left side of axes
LEFT_PARTIES = {
"SP", "PvdA", "GL", "GroenLinks", "GroenLinks-PvdA",
"DENK", "PvdD", "Volt"
}
# Cache for SVD_THEMES to avoid repeated imports
_svd_themes_cache: Optional[Dict[int, Dict[str, str]]] = None
def _get_svd_themes() -> Dict[int, Dict[str, str]]:
"""Lazy import SVD_THEMES from explorer.py to avoid circular imports.
Returns:
Dict mapping component number to theme dict with keys:
- label: Short label for the component
- explanation: Detailed explanation
- positive_pole: Description of positive pole
- negative_pole: Description of negative pole
- flip: Whether to flip the axis
"""
global _svd_themes_cache
if _svd_themes_cache is not None:
return _svd_themes_cache
try:
# Import at runtime to avoid circular dependency
# explorer.py imports from analysis/ but we don't import from explorer.py
# at module load time
import importlib.util
import os
# Find explorer.py
explorer_path = os.path.join(os.path.dirname(__file__), "..", "explorer.py")
if not os.path.exists(explorer_path):
_logger.warning("explorer.py not found at %s", explorer_path)
return {}
spec = importlib.util.spec_from_file_location("explorer", explorer_path)
if spec is None or spec.loader is None:
_logger.warning("Could not load spec from explorer.py")
return {}
explorer_module = importlib.util.module_from_spec(spec)
spec.loader.exec_module(explorer_module)
# Get SVD_THEMES from the build_svd_components_tab function's local scope
# This is a bit hacky but avoids importing the entire Streamlit app
# We'll need to refactor explorer.py to export SVD_THEMES at module level
# For now, we'll define it here as a fallback
# Fallback: define SVD_THEMES here if we can't import from explorer
_svd_themes_cache = {}
return _svd_themes_cache
except Exception as e:
_logger.exception("Failed to load SVD_THEMES from explorer.py: %s", e)
return {}
def get_svd_label(component: int) -> str:
"""Get short label for SVD component.
Args:
component: SVD component number (1-indexed)
Returns:
Short label string (e.g., 'EU-integratie–Nationalisme')
Raises:
ValueError: If component < 1
"""
if component < 1:
raise ValueError(f"Component must be >= 1, got {component}")
themes = _get_svd_themes()
if component in themes:
return themes[component].get("label", f"As {component}")
# Fallback labels for components 1-3 (most commonly used)
fallback_labels = {
1: "EU-integratie–Nationalisme",
2: "Populistisch–Institutioneel",
3: "Verzorgingsstaat–Marktwerking",
}
return fallback_labels.get(component, f"As {component}")
def get_svd_theme(component: int) -> Dict[str, str]:
"""Get full theme dict for SVD component.
Args:
component: SVD component number (1-indexed)
Returns:
Dict with keys: label, explanation, positive_pole, negative_pole, flip
"""
if component < 1:
raise ValueError(f"Component must be >= 1, got {component}")
themes = _get_svd_themes()
if component in themes:
return themes[component]
# Return minimal fallback
return {
"label": get_svd_label(component),
"explanation": "",
"positive_pole": "",
"negative_pole": "",
"flip": False,
}
def compute_flip_direction(
component: int,
party_scores: Dict[str, List[float]]
) -> bool:
"""Compute flip direction so right parties appear on the right side.
Args:
component: SVD component number (1-indexed)
party_scores: Dict mapping party name to list of scores per component
(party_scores[party][0] is score for component 1, etc.)
Returns:
True if axis should be flipped so right parties are on right.
False otherwise.
"""
if component < 1:
return False
idx = component - 1 # Convert to 0-indexed
right_scores = []
left_scores = []
for party, scores in party_scores.items():
if len(scores) <= idx:
continue
score = scores[idx]
if party in RIGHT_PARTIES:
right_scores.append(score)
elif party in LEFT_PARTIES:
left_scores.append(score)
if not right_scores or not left_scores:
return False # Default: no flip if insufficient data
right_mean = sum(right_scores) / len(right_scores)
left_mean = sum(left_scores) / len(left_scores)
# Flip if right parties have lower mean (they're on the left)
return right_mean < left_mean
def get_fallback_labels() -> Tuple[str, str]:
"""Get fallback labels for x and y axes (components 1 and 2).
Returns:
Tuple of (x_label, y_label)
"""
return (get_svd_label(1), get_svd_label(2))
- Step 4: Run test to verify it passes
Run: uv run pytest tests/test_svd_labels.py::test_get_svd_label_returns_correct_label -v
Expected: PASS
- Step 5: Write test for
compute_flip_direction
# tests/test_svd_labels.py (append)
def test_compute_flip_direction_right_on_left():
"""Test that flip is True when right parties are on the left."""
from analysis.svd_labels import compute_flip_direction
# Right parties have negative scores (on left), left parties have positive
party_scores = {
"VVD": [-0.5, 0.0], # Right party, component 1 score = -0.5
"PVV": [-0.8, 0.0], # Right party
"SP": [0.6, 0.0], # Left party, component 1 score = 0.6
"DENK": [0.4, 0.0], # Left party
}
# Component 1: right_mean = -0.65, left_mean = 0.5
# right_mean < left_mean, so flip = True
assert compute_flip_direction(1, party_scores) is True
def test_compute_flip_direction_right_on_right():
"""Test that flip is False when right parties are already on the right."""
from analysis.svd_labels import compute_flip_direction
# Right parties have positive scores (on right), left parties have negative
party_scores = {
"VVD": [0.5, 0.0], # Right party, component 1 score = 0.5
"PVV": [0.8, 0.0], # Right party
"SP": [-0.6, 0.0], # Left party
"DENK": [-0.4, 0.0], # Left party
}
# Component 1: right_mean = 0.65, left_mean = -0.5
# right_mean > left_mean, so flip = False
assert compute_flip_direction(1, party_scores) is False
def test_compute_flip_direction_insufficient_data():
"""Test that flip is False when there's insufficient data."""
from analysis.svd_labels import compute_flip_direction
# No right parties in data
party_scores = {
"SP": [0.6, 0.0],
"DENK": [0.4, 0.0],
}
assert compute_flip_direction(1, party_scores) is False
# No left parties in data
party_scores = {
"VVD": [0.5, 0.0],
"PVV": [0.8, 0.0],
}
assert compute_flip_direction(1, party_scores) is False
- Step 6: Run tests for
compute_flip_direction
Run: uv run pytest tests/test_svd_labels.py -v
Expected: All tests PASS
- Step 7: Commit
git add analysis/svd_labels.py tests/test_svd_labels.py
git commit -m "feat: add svd_labels module for unified label system"
Task 2: Refactor explorer.py to export SVD_THEMES at module level
Files:
-
Modify:
explorer.py(move SVD_THEMES to module level) -
Modify:
analysis/svd_labels.py(update to import from explorer) -
Step 1: Move SVD_THEMES from function to module level in explorer.py
Find the SVD_THEMES dict inside build_svd_components_tab function (around line 2459) and move it to module level (near the top of the file, after imports).
# explorer.py (near top of file, after imports and constants)
# Political polarisation themes per SVD component (1-indexed, window=2025)
# Produced by per-axis analysis of all 10 unique top motions (zero cross-axis overlap).
# This is the canonical source of truth for SVD component labels.
SVD_THEMES: dict[int, dict[str, str]] = {
1: {
"label": "EU-integratie en internationalisme versus nationalisme",
# ... rest of the dict
},
# ... rest of components
}
- Step 2: Update
analysis/svd_labels.pyto import SVD_THEMES properly
# analysis/svd_labels.py (update _get_svd_themes function)
def _get_svd_themes() -> Dict[int, Dict[str, str]]:
"""Lazy import SVD_THEMES from explorer.py to avoid circular imports."""
global _svd_themes_cache
if _svd_themes_cache is not None:
return _svd_themes_cache
try:
# Import at runtime to avoid circular dependency at module load time
# explorer.py imports from analysis/ but we delay our import
import explorer
_svd_themes_cache = explorer.SVD_THEMES
return _svd_themes_cache
except ImportError as e:
_logger.warning("Could not import explorer.SVD_THEMES: %s", e)
return {}
except Exception as e:
_logger.exception("Failed to load SVD_THEMES from explorer.py: %s", e)
return {}
- Step 3: Run tests to verify nothing broke
Run: uv run pytest tests/test_svd_labels.py -v
Expected: All tests PASS
- Step 4: Commit
git add explorer.py analysis/svd_labels.py
git commit -m "refactor: move SVD_THEMES to module level for import"
Task 3: Update axis_classifier.py to use svd_labels
Files:
-
Modify:
analysis/axis_classifier.py -
Modify:
tests/test_axis_label_fallback.py -
Step 1: Write test for updated
display_label_for_modal
# tests/test_axis_label_fallback.py (update existing tests)
def test_display_label_for_modal_uses_svd_themes():
"""Test that display_label_for_modal uses SVD_THEMES for fallback labels."""
from analysis.axis_classifier import display_label_for_modal
# None should return fallback from SVD_THEMES
x_label = display_label_for_modal(None, "x")
y_label = display_label_for_modal(None, "y")
# Should return component 1 and 2 labels from SVD_THEMES
assert "EU-integratie" in x_label or "Nationalisme" in x_label
assert "Populistisch" in y_label or "Institutioneel" in y_label
def test_display_label_for_modal_maps_as_labels():
"""Test that 'As 1' and 'As 2' are mapped to semantic labels."""
from analysis.axis_classifier import display_label_for_modal
x_label = display_label_for_modal("As 1", "x")
y_label = display_label_for_modal("As 2", "y")
# Should return component 1 and 2 labels
assert "EU-integratie" in x_label or "Nationalisme" in x_label
assert "Populistisch" in y_label or "Institutioneel" in y_label
- Step 2: Run test to verify it fails
Run: uv run pytest tests/test_axis_label_fallback.py -v
Expected: Some tests FAIL (current implementation returns hardcoded labels)
- Step 3: Update
axis_classifier.pyto use svd_labels
# analysis/axis_classifier.py (update the module)
# Remove the hardcoded _LABELS dict (lines 25-35)
# Replace with imports from svd_labels
# At the top of the file, after imports:
from analysis.svd_labels import get_svd_label, get_fallback_labels
# Remove _LABELS dict entirely
# Update display_label_for_modal function (lines 42-55):
def display_label_for_modal(modal_label: Optional[str], axis: str) -> str:
"""Return a user-facing axis label for a modal/internal label.
Maps numeric fallback names 'As 1' / 'Stempatroon As 1' to the
semantic labels from SVD_THEMES. Any other label is returned unchanged.
None is treated as the semantic fallback for the axis.
"""
if modal_label is None:
# Fallback to component 1 (x) or 2 (y)
comp = 1 if axis == "x" else 2
return get_svd_label(comp)
# Map "As 1" / "As 2" to semantic labels
if axis == "x" and modal_label in ("As 1", "Stempatroon As 1"):
return get_svd_label(1)
if axis == "y" and modal_label in ("As 2", "Stempatroon As 2"):
return get_svd_label(2)
return modal_label
- Step 4: Update
_INTERPRETATION_TEMPLATESto use svd_labels
The interpretation templates still need to reference the correct labels. Keep them as-is for now since they're used for motion classification, not display.
- Step 5: Update
_MOTION_LABEL_TEMPLATE_KEYto use svd_labels
Keep the mapping as-is for now since it's used for motion classification.
- Step 6: Update
_KEYWORDSto use svd_labels
Keep the keywords as-is for now since they're used for motion classification.
- Step 7: Update fallback label references in classify_axes
Find lines 610 and 615 where _LABELS["fallback_x"] and _LABELS["fallback_y"] are used and replace:
# Before:
x_lbl = _LABELS["fallback_x"]
y_lbl = _LABELS["fallback_y"]
# After:
x_lbl, y_lbl = get_fallback_labels()
- Step 8: Run tests to verify they pass
Run: uv run pytest tests/test_axis_label_fallback.py tests/test_political_compass.py -v
Expected: All tests PASS
- Step 9: Commit
git add analysis/axis_classifier.py tests/test_axis_label_fallback.py
git commit -m "refactor: use svd_labels for axis labels in axis_classifier"
Task 4: Update explorer.py to use svd_labels for fallback labels
Files:
-
Modify:
explorer.py -
Step 1: Find and update fallback label references in explorer.py
Find lines 1440-1441 where hardcoded fallback labels are used:
# Before:
_x_label = _raw_x or "EU-integratie–Nationalisme"
_y_label = _raw_y or "Populistisch–Institutioneel"
# After:
from analysis.svd_labels import get_fallback_labels
_x_fallback, _y_fallback = get_fallback_labels()
_x_label = _raw_x or _x_fallback
_y_label = _raw_y or _y_fallback
- Step 2: Run tests to verify nothing broke
Run: uv run pytest tests/test_political_compass.py -v
Expected: All tests PASS
- Step 3: Commit
git add explorer.py
git commit -m "refactor: use svd_labels for fallback labels in explorer"
Task 5: Add 1D party position charts for components 3-10
Files:
-
Modify:
explorer.py -
Test:
tests/test_explorer_chart.py -
Step 1: Write test for 1D party position chart
# tests/test_explorer_chart.py (append)
def test_render_party_axis_chart_1d_renders():
"""Test that _render_party_axis_chart_1d renders a figure."""
from explorer import _render_party_axis_chart_1d
party_coords = {
"VVD": (0.5,),
"SP": (-0.6,),
"PVV": (0.8,),
"DENK": (-0.4,),
}
theme = {
"label": "Test Component",
"positive_pole": "Positive",
"negative_pole": "Negative",
"flip": False,
}
fig = _render_party_axis_chart_1d(party_coords, 3, theme)
assert fig is not None
# Check that figure has traces
assert len(fig.data) > 0
- Step 2: Run test to verify it fails
Run: uv run pytest tests/test_explorer_chart.py::test_render_party_axis_chart_1d_renders -v
Expected: FAIL with "cannot import name '_render_party_axis_chart_1d'"
- Step 3: Implement
_render_party_axis_chart_1dfunction in explorer.py
Add this function near _render_party_axis_chart (around line 2900):
# explorer.py (add new function)
def _render_party_axis_chart_1d(
party_coords: dict,
component: int,
theme: dict,
bootstrap_data: Optional[Dict] = None,
) -> go.Figure:
"""Render a 1D party position chart for a single SVD component.
Args:
party_coords: Dict mapping party name to (score,) tuple
component: SVD component number (1-indexed)
theme: Dict with label, positive_pole, negative_pole, flip
bootstrap_data: Optional bootstrap confidence intervals
Returns:
Plotly Figure object
"""
import plotly.graph_objects as go
import numpy as np
if not party_coords:
fig = go.Figure()
fig.add_annotation(
text="Geen partijposities beschikbaar",
xref="paper", yref="paper",
x=0.5, y=0.5, showarrow=False,
font=dict(size=14)
)
return fig
# Extract scores and parties
parties = list(party_coords.keys())
scores = [coords[0] for coords in party_coords.values()]
# Apply flip if needed
flip = theme.get("flip", False)
if flip:
scores = [-s for s in scores]
# Get party colors
party_colors = [PARTY_COLOURS.get(p, "#9E9E9E") for p in parties]
# Create horizontal bar chart
fig = go.Figure()
# Sort by score for better visualization
sorted_indices = np.argsort(scores)
sorted_parties = [parties[i] for i in sorted_indices]
sorted_scores = [scores[i] for i in sorted_indices]
sorted_colors = [party_colors[i] for i in sorted_indices]
fig.add_trace(go.Bar(
y=sorted_parties,
x=sorted_scores,
orientation='h',
marker_color=sorted_colors,
text=[f"{s:.2f}" for s in sorted_scores],
textposition='outside',
))
# Update layout
label = theme.get("label", f"As {component}")
positive_pole = theme.get("positive_pole", "Positief")
negative_pole = theme.get("negative_pole", "Negatief")
fig.update_layout(
title=f"Partijposities — {label}",
xaxis_title=f"{negative_pole} ← → {positive_pole}",
yaxis_title="",
height=max(400, len(parties) * 25), # Scale height with number of parties
margin=dict(l=150), # Extra margin for party names
showlegend=False,
)
# Add vertical line at x=0
fig.add_vline(x=0, line_dash="dash", line_color="gray", opacity=0.5)
return fig
- Step 4: Update SVD Components tab to use 1D chart for components 3-10
Find the section in build_svd_components_tab where _render_party_axis_chart is called (around line 2823) and update:
# explorer.py (update around line 2822-2830)
# Before:
if comp_sel in (1, 2):
_render_party_axis_chart(
party_coords, comp_sel, theme, bootstrap_data=bootstrap_data
)
else:
st.caption(
"_Partijposities zijn alleen beschikbaar voor de eerste twee SVD-assen._"
)
# After:
if comp_sel in (1, 2):
_render_party_axis_chart(
party_coords, comp_sel, theme, bootstrap_data=bootstrap_data
)
else:
# For components 3-10, show 1D party positions
# Extract 1D scores for this component
party_1d_coords = {}
idx = comp_sel - 1 # Convert to 0-indexed
for party, scores in party_scores.items():
if len(scores) > idx:
party_1d_coords[party] = (scores[idx],)
_render_party_axis_chart_1d(
party_1d_coords, comp_sel, theme, bootstrap_data=None
)
- Step 5: Run tests to verify they pass
Run: uv run pytest tests/test_explorer_chart.py -v
Expected: All tests PASS
- Step 6: Commit
git add explorer.py tests/test_explorer_chart.py
git commit -m "feat: add 1D party position charts for SVD components 3-10"
Task 6: Auto-compute flip directions for all components
Files:
-
Modify:
explorer.py -
Modify:
analysis/svd_labels.py -
Test:
tests/test_svd_labels.py -
Step 1: Write test for auto-computing flip in SVD Components tab
# tests/test_svd_labels.py (append)
def test_auto_flip_computation_for_all_components():
"""Test that flip directions are computed correctly for all components."""
from analysis.svd_labels import compute_flip_direction
# Simulate party scores for 10 components
# Right parties should have positive scores on component 1 (EU-integratie)
# Left parties should have negative scores on component 1
party_scores = {
"VVD": [0.5] * 10, # Right party, positive on all components
"PVV": [0.8] * 10, # Right party
"SP": [-0.6] * 10, # Left party, negative on all components
"DENK": [-0.4] * 10, # Left party
}
# For all components, right_mean > left_mean, so flip should be False
for comp in range(1, 11):
flip = compute_flip_direction(comp, party_scores)
assert flip is False, f"Component {comp} should not flip"
# Now test with right parties on left (negative scores)
party_scores_left = {
"VVD": [-0.5] * 10,
"PVV": [-0.8] * 10,
"SP": [0.6] * 10,
"DENK": [0.4] * 10,
}
# For all components, right_mean < left_mean, so flip should be True
for comp in range(1, 11):
flip = compute_flip_direction(comp, party_scores_left)
assert flip is True, f"Component {comp} should flip"
- Step 2: Run test to verify it passes
Run: uv run pytest tests/test_svd_labels.py::test_auto_flip_computation_for_all_components -v
Expected: PASS (function already implemented in Task 1)
- Step 3: Update SVD Components tab to compute flip dynamically
In build_svd_components_tab, after loading party_scores, compute flip for each component:
# explorer.py (in build_svd_components_tab, after loading party_scores)
from analysis.svd_labels import compute_flip_direction
# After party_scores is loaded (around line 2800)
# Compute flip directions for all components
computed_flips = {}
for comp in range(1, 11):
computed_flips[comp] = compute_flip_direction(comp, party_scores)
# Update SVD_THEMES with computed flips
for comp, flip in computed_flips.items():
if comp in SVD_THEMES:
SVD_THEMES[comp]["flip"] = flip
- Step 4: Run tests to verify nothing broke
Run: uv run pytest tests/test_svd_labels.py tests/test_explorer_chart.py -v
Expected: All tests PASS
- Step 5: Commit
git add explorer.py analysis/svd_labels.py tests/test_svd_labels.py
git commit -m "feat: auto-compute flip directions for all SVD components"
Task 7: Update existing tests for new label system
Files:
-
Modify:
tests/test_political_compass.py -
Modify:
tests/test_axis_label_fallback.py -
Step 1: Run all tests to identify failures
Run: uv run pytest tests/ -v --tb=short
Expected: Some tests may fail due to label changes
- Step 2: Fix any failing tests
Update test assertions to use new labels from SVD_THEMES:
# tests/test_political_compass.py (update assertions)
# Before:
assert "Links–Rechts" in x_label
# After:
assert "EU-integratie" in x_label or "Nationalisme" in x_label
- Step 3: Run all tests to verify they pass
Run: uv run pytest tests/ -v
Expected: All tests PASS
- Step 4: Commit
git add tests/
git commit -m "test: update tests for unified SVD label system"
Task 8: Final verification and cleanup
Files:
-
All modified files
-
Step 1: Run full test suite
Run: uv run pytest tests/ -v
Expected: All tests PASS
- Step 2: Run Streamlit app to verify UI
Run: uv run streamlit run explorer.py
Expected: App starts without errors
-
Step 3: Manually verify in UI
-
Open SVD Components tab
-
Check that components 1-10 show correct labels
-
Check that party position charts render for all components
-
Check that right parties appear on the right side of axes
-
Step 4: Final commit
git add -A
git commit -m "feat: complete SVD label unification"
Summary
This plan creates a unified label system for SVD components with:
- Single source of truth in
SVD_THEMES - Automatic flip direction computation based on party centroids
- 1D party position charts for components 3-10
- Consistent labels across compass, trajectory, and SVD Components views