parent
bed776f295
commit
a1c3e92fab
@ -0,0 +1,879 @@ |
|||||||
|
# SVD Label Unification Implementation Plan |
||||||
|
|
||||||
|
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. |
||||||
|
|
||||||
|
**Goal:** Unify SVD component labels into a single source of truth and automatically compute axis flip directions so right-wing parties consistently appear on the right side of all SVD component axes. |
||||||
|
|
||||||
|
**Architecture:** Create a new `analysis/svd_labels.py` module that imports `SVD_THEMES` from explorer.py at runtime and provides helper functions for label lookup and flip direction computation. Update `axis_classifier.py` to use this module instead of hardcoded labels. Add 1D party position charts for components 3-10 in the SVD Components tab. |
||||||
|
|
||||||
|
**Tech Stack:** Python, Streamlit, NumPy, DuckDB |
||||||
|
|
||||||
|
--- |
||||||
|
|
||||||
|
## File Structure |
||||||
|
|
||||||
|
| File | Responsibility | |
||||||
|
|------|---------------| |
||||||
|
| `analysis/svd_labels.py` | **NEW** - Unified label system with auto-flip computation | |
||||||
|
| `analysis/axis_classifier.py` | Remove `_LABELS`, import from svd_labels | |
||||||
|
| `explorer.py` | Remove fallback labels, add 1D charts for components 3-10 | |
||||||
|
| `tests/test_svd_labels.py` | **NEW** - Tests for svd_labels module | |
||||||
|
| `tests/test_axis_label_fallback.py` | Update to use new label system | |
||||||
|
| `tests/test_political_compass.py` | Update assertions for new labels | |
||||||
|
|
||||||
|
--- |
||||||
|
|
||||||
|
## Task 1: Create `analysis/svd_labels.py` with core functions |
||||||
|
|
||||||
|
**Files:** |
||||||
|
- Create: `analysis/svd_labels.py` |
||||||
|
- Test: `tests/test_svd_labels.py` |
||||||
|
|
||||||
|
- [ ] **Step 1: Write the failing test for `get_svd_label`** |
||||||
|
|
||||||
|
```python |
||||||
|
# tests/test_svd_labels.py |
||||||
|
"""Tests for analysis/svd_labels module.""" |
||||||
|
|
||||||
|
def test_get_svd_label_returns_correct_label(): |
||||||
|
"""Test that get_svd_label returns the correct label for each component.""" |
||||||
|
from analysis.svd_labels import get_svd_label |
||||||
|
|
||||||
|
# Component 1 should return EU-integratie label |
||||||
|
label1 = get_svd_label(1) |
||||||
|
assert "EU-integratie" in label1 or "Nationalisme" in label1 |
||||||
|
|
||||||
|
# Component 2 should return Populistisch label |
||||||
|
label2 = get_svd_label(2) |
||||||
|
assert "Populistisch" in label2 or "Institutioneel" in label2 |
||||||
|
|
||||||
|
# Component 3 should return Verzorgingsstaat label |
||||||
|
label3 = get_svd_label(3) |
||||||
|
assert "Verzorgingsstaat" in label3 or "Marktwerking" in label3 |
||||||
|
``` |
||||||
|
|
||||||
|
- [ ] **Step 2: Run test to verify it fails** |
||||||
|
|
||||||
|
Run: `uv run pytest tests/test_svd_labels.py::test_get_svd_label_returns_correct_label -v` |
||||||
|
Expected: FAIL with "ModuleNotFoundError: No module named 'analysis.svd_labels'" |
||||||
|
|
||||||
|
- [ ] **Step 3: Create `analysis/svd_labels.py` with `get_svd_label` function** |
||||||
|
|
||||||
|
```python |
||||||
|
# analysis/svd_labels.py |
||||||
|
"""Unified SVD component labels and automatic flip direction computation. |
||||||
|
|
||||||
|
This module provides a single source of truth for SVD component labels, |
||||||
|
deriving them from SVD_THEMES in explorer.py. It also computes flip |
||||||
|
directions automatically based on party centroids. |
||||||
|
""" |
||||||
|
|
||||||
|
import logging |
||||||
|
from typing import Dict, List, Optional, Tuple |
||||||
|
|
||||||
|
_logger = logging.getLogger(__name__) |
||||||
|
|
||||||
|
# Canonical party sets for orientation |
||||||
|
# Right-wing parties that should appear on the right side of axes |
||||||
|
RIGHT_PARTIES = { |
||||||
|
"PVV", "VVD", "FVD", "BBB", "JA21", |
||||||
|
"Nieuw Sociaal Contract", "SGP", "CDA", "ChristenUnie" |
||||||
|
} |
||||||
|
|
||||||
|
# Left-wing parties that should appear on the left side of axes |
||||||
|
LEFT_PARTIES = { |
||||||
|
"SP", "PvdA", "GL", "GroenLinks", "GroenLinks-PvdA", |
||||||
|
"DENK", "PvdD", "Volt" |
||||||
|
} |
||||||
|
|
||||||
|
# Cache for SVD_THEMES to avoid repeated imports |
||||||
|
_svd_themes_cache: Optional[Dict[int, Dict[str, str]]] = None |
||||||
|
|
||||||
|
|
||||||
|
def _get_svd_themes() -> Dict[int, Dict[str, str]]: |
||||||
|
"""Lazy import SVD_THEMES from explorer.py to avoid circular imports. |
||||||
|
|
||||||
|
Returns: |
||||||
|
Dict mapping component number to theme dict with keys: |
||||||
|
- label: Short label for the component |
||||||
|
- explanation: Detailed explanation |
||||||
|
- positive_pole: Description of positive pole |
||||||
|
- negative_pole: Description of negative pole |
||||||
|
- flip: Whether to flip the axis |
||||||
|
""" |
||||||
|
global _svd_themes_cache |
||||||
|
if _svd_themes_cache is not None: |
||||||
|
return _svd_themes_cache |
||||||
|
|
||||||
|
try: |
||||||
|
# Import at runtime to avoid circular dependency |
||||||
|
# explorer.py imports from analysis/ but we don't import from explorer.py |
||||||
|
# at module load time |
||||||
|
import importlib.util |
||||||
|
import os |
||||||
|
|
||||||
|
# Find explorer.py |
||||||
|
explorer_path = os.path.join(os.path.dirname(__file__), "..", "explorer.py") |
||||||
|
if not os.path.exists(explorer_path): |
||||||
|
_logger.warning("explorer.py not found at %s", explorer_path) |
||||||
|
return {} |
||||||
|
|
||||||
|
spec = importlib.util.spec_from_file_location("explorer", explorer_path) |
||||||
|
if spec is None or spec.loader is None: |
||||||
|
_logger.warning("Could not load spec from explorer.py") |
||||||
|
return {} |
||||||
|
|
||||||
|
explorer_module = importlib.util.module_from_spec(spec) |
||||||
|
spec.loader.exec_module(explorer_module) |
||||||
|
|
||||||
|
# Get SVD_THEMES from the build_svd_components_tab function's local scope |
||||||
|
# This is a bit hacky but avoids importing the entire Streamlit app |
||||||
|
# We'll need to refactor explorer.py to export SVD_THEMES at module level |
||||||
|
# For now, we'll define it here as a fallback |
||||||
|
|
||||||
|
# Fallback: define SVD_THEMES here if we can't import from explorer |
||||||
|
_svd_themes_cache = {} |
||||||
|
return _svd_themes_cache |
||||||
|
|
||||||
|
except Exception as e: |
||||||
|
_logger.exception("Failed to load SVD_THEMES from explorer.py: %s", e) |
||||||
|
return {} |
||||||
|
|
||||||
|
|
||||||
|
def get_svd_label(component: int) -> str: |
||||||
|
"""Get short label for SVD component. |
||||||
|
|
||||||
|
Args: |
||||||
|
component: SVD component number (1-indexed) |
||||||
|
|
||||||
|
Returns: |
||||||
|
Short label string (e.g., 'EU-integratie–Nationalisme') |
||||||
|
|
||||||
|
Raises: |
||||||
|
ValueError: If component < 1 |
||||||
|
""" |
||||||
|
if component < 1: |
||||||
|
raise ValueError(f"Component must be >= 1, got {component}") |
||||||
|
|
||||||
|
themes = _get_svd_themes() |
||||||
|
if component in themes: |
||||||
|
return themes[component].get("label", f"As {component}") |
||||||
|
|
||||||
|
# Fallback labels for components 1-3 (most commonly used) |
||||||
|
fallback_labels = { |
||||||
|
1: "EU-integratie–Nationalisme", |
||||||
|
2: "Populistisch–Institutioneel", |
||||||
|
3: "Verzorgingsstaat–Marktwerking", |
||||||
|
} |
||||||
|
return fallback_labels.get(component, f"As {component}") |
||||||
|
|
||||||
|
|
||||||
|
def get_svd_theme(component: int) -> Dict[str, str]: |
||||||
|
"""Get full theme dict for SVD component. |
||||||
|
|
||||||
|
Args: |
||||||
|
component: SVD component number (1-indexed) |
||||||
|
|
||||||
|
Returns: |
||||||
|
Dict with keys: label, explanation, positive_pole, negative_pole, flip |
||||||
|
""" |
||||||
|
if component < 1: |
||||||
|
raise ValueError(f"Component must be >= 1, got {component}") |
||||||
|
|
||||||
|
themes = _get_svd_themes() |
||||||
|
if component in themes: |
||||||
|
return themes[component] |
||||||
|
|
||||||
|
# Return minimal fallback |
||||||
|
return { |
||||||
|
"label": get_svd_label(component), |
||||||
|
"explanation": "", |
||||||
|
"positive_pole": "", |
||||||
|
"negative_pole": "", |
||||||
|
"flip": False, |
||||||
|
} |
||||||
|
|
||||||
|
|
||||||
|
def compute_flip_direction( |
||||||
|
component: int, |
||||||
|
party_scores: Dict[str, List[float]] |
||||||
|
) -> bool: |
||||||
|
"""Compute flip direction so right parties appear on the right side. |
||||||
|
|
||||||
|
Args: |
||||||
|
component: SVD component number (1-indexed) |
||||||
|
party_scores: Dict mapping party name to list of scores per component |
||||||
|
(party_scores[party][0] is score for component 1, etc.) |
||||||
|
|
||||||
|
Returns: |
||||||
|
True if axis should be flipped so right parties are on right. |
||||||
|
False otherwise. |
||||||
|
""" |
||||||
|
if component < 1: |
||||||
|
return False |
||||||
|
|
||||||
|
idx = component - 1 # Convert to 0-indexed |
||||||
|
|
||||||
|
right_scores = [] |
||||||
|
left_scores = [] |
||||||
|
|
||||||
|
for party, scores in party_scores.items(): |
||||||
|
if len(scores) <= idx: |
||||||
|
continue |
||||||
|
|
||||||
|
score = scores[idx] |
||||||
|
if party in RIGHT_PARTIES: |
||||||
|
right_scores.append(score) |
||||||
|
elif party in LEFT_PARTIES: |
||||||
|
left_scores.append(score) |
||||||
|
|
||||||
|
if not right_scores or not left_scores: |
||||||
|
return False # Default: no flip if insufficient data |
||||||
|
|
||||||
|
right_mean = sum(right_scores) / len(right_scores) |
||||||
|
left_mean = sum(left_scores) / len(left_scores) |
||||||
|
|
||||||
|
# Flip if right parties have lower mean (they're on the left) |
||||||
|
return right_mean < left_mean |
||||||
|
|
||||||
|
|
||||||
|
def get_fallback_labels() -> Tuple[str, str]: |
||||||
|
"""Get fallback labels for x and y axes (components 1 and 2). |
||||||
|
|
||||||
|
Returns: |
||||||
|
Tuple of (x_label, y_label) |
||||||
|
""" |
||||||
|
return (get_svd_label(1), get_svd_label(2)) |
||||||
|
``` |
||||||
|
|
||||||
|
- [ ] **Step 4: Run test to verify it passes** |
||||||
|
|
||||||
|
Run: `uv run pytest tests/test_svd_labels.py::test_get_svd_label_returns_correct_label -v` |
||||||
|
Expected: PASS |
||||||
|
|
||||||
|
- [ ] **Step 5: Write test for `compute_flip_direction`** |
||||||
|
|
||||||
|
```python |
||||||
|
# tests/test_svd_labels.py (append) |
||||||
|
|
||||||
|
def test_compute_flip_direction_right_on_left(): |
||||||
|
"""Test that flip is True when right parties are on the left.""" |
||||||
|
from analysis.svd_labels import compute_flip_direction |
||||||
|
|
||||||
|
# Right parties have negative scores (on left), left parties have positive |
||||||
|
party_scores = { |
||||||
|
"VVD": [-0.5, 0.0], # Right party, component 1 score = -0.5 |
||||||
|
"PVV": [-0.8, 0.0], # Right party |
||||||
|
"SP": [0.6, 0.0], # Left party, component 1 score = 0.6 |
||||||
|
"DENK": [0.4, 0.0], # Left party |
||||||
|
} |
||||||
|
|
||||||
|
# Component 1: right_mean = -0.65, left_mean = 0.5 |
||||||
|
# right_mean < left_mean, so flip = True |
||||||
|
assert compute_flip_direction(1, party_scores) is True |
||||||
|
|
||||||
|
|
||||||
|
def test_compute_flip_direction_right_on_right(): |
||||||
|
"""Test that flip is False when right parties are already on the right.""" |
||||||
|
from analysis.svd_labels import compute_flip_direction |
||||||
|
|
||||||
|
# Right parties have positive scores (on right), left parties have negative |
||||||
|
party_scores = { |
||||||
|
"VVD": [0.5, 0.0], # Right party, component 1 score = 0.5 |
||||||
|
"PVV": [0.8, 0.0], # Right party |
||||||
|
"SP": [-0.6, 0.0], # Left party |
||||||
|
"DENK": [-0.4, 0.0], # Left party |
||||||
|
} |
||||||
|
|
||||||
|
# Component 1: right_mean = 0.65, left_mean = -0.5 |
||||||
|
# right_mean > left_mean, so flip = False |
||||||
|
assert compute_flip_direction(1, party_scores) is False |
||||||
|
|
||||||
|
|
||||||
|
def test_compute_flip_direction_insufficient_data(): |
||||||
|
"""Test that flip is False when there's insufficient data.""" |
||||||
|
from analysis.svd_labels import compute_flip_direction |
||||||
|
|
||||||
|
# No right parties in data |
||||||
|
party_scores = { |
||||||
|
"SP": [0.6, 0.0], |
||||||
|
"DENK": [0.4, 0.0], |
||||||
|
} |
||||||
|
|
||||||
|
assert compute_flip_direction(1, party_scores) is False |
||||||
|
|
||||||
|
# No left parties in data |
||||||
|
party_scores = { |
||||||
|
"VVD": [0.5, 0.0], |
||||||
|
"PVV": [0.8, 0.0], |
||||||
|
} |
||||||
|
|
||||||
|
assert compute_flip_direction(1, party_scores) is False |
||||||
|
``` |
||||||
|
|
||||||
|
- [ ] **Step 6: Run tests for `compute_flip_direction`** |
||||||
|
|
||||||
|
Run: `uv run pytest tests/test_svd_labels.py -v` |
||||||
|
Expected: All tests PASS |
||||||
|
|
||||||
|
- [ ] **Step 7: Commit** |
||||||
|
|
||||||
|
```bash |
||||||
|
git add analysis/svd_labels.py tests/test_svd_labels.py |
||||||
|
git commit -m "feat: add svd_labels module for unified label system" |
||||||
|
``` |
||||||
|
|
||||||
|
--- |
||||||
|
|
||||||
|
## Task 2: Refactor explorer.py to export SVD_THEMES at module level |
||||||
|
|
||||||
|
**Files:** |
||||||
|
- Modify: `explorer.py` (move SVD_THEMES to module level) |
||||||
|
- Modify: `analysis/svd_labels.py` (update to import from explorer) |
||||||
|
|
||||||
|
- [ ] **Step 1: Move SVD_THEMES from function to module level in explorer.py** |
||||||
|
|
||||||
|
Find the `SVD_THEMES` dict inside `build_svd_components_tab` function (around line 2459) and move it to module level (near the top of the file, after imports). |
||||||
|
|
||||||
|
```python |
||||||
|
# explorer.py (near top of file, after imports and constants) |
||||||
|
|
||||||
|
# Political polarisation themes per SVD component (1-indexed, window=2025) |
||||||
|
# Produced by per-axis analysis of all 10 unique top motions (zero cross-axis overlap). |
||||||
|
# This is the canonical source of truth for SVD component labels. |
||||||
|
SVD_THEMES: dict[int, dict[str, str]] = { |
||||||
|
1: { |
||||||
|
"label": "EU-integratie en internationalisme versus nationalisme", |
||||||
|
# ... rest of the dict |
||||||
|
}, |
||||||
|
# ... rest of components |
||||||
|
} |
||||||
|
``` |
||||||
|
|
||||||
|
- [ ] **Step 2: Update `analysis/svd_labels.py` to import SVD_THEMES properly** |
||||||
|
|
||||||
|
```python |
||||||
|
# analysis/svd_labels.py (update _get_svd_themes function) |
||||||
|
|
||||||
|
def _get_svd_themes() -> Dict[int, Dict[str, str]]: |
||||||
|
"""Lazy import SVD_THEMES from explorer.py to avoid circular imports.""" |
||||||
|
global _svd_themes_cache |
||||||
|
if _svd_themes_cache is not None: |
||||||
|
return _svd_themes_cache |
||||||
|
|
||||||
|
try: |
||||||
|
# Import at runtime to avoid circular dependency at module load time |
||||||
|
# explorer.py imports from analysis/ but we delay our import |
||||||
|
import explorer |
||||||
|
_svd_themes_cache = explorer.SVD_THEMES |
||||||
|
return _svd_themes_cache |
||||||
|
except ImportError as e: |
||||||
|
_logger.warning("Could not import explorer.SVD_THEMES: %s", e) |
||||||
|
return {} |
||||||
|
except Exception as e: |
||||||
|
_logger.exception("Failed to load SVD_THEMES from explorer.py: %s", e) |
||||||
|
return {} |
||||||
|
``` |
||||||
|
|
||||||
|
- [ ] **Step 3: Run tests to verify nothing broke** |
||||||
|
|
||||||
|
Run: `uv run pytest tests/test_svd_labels.py -v` |
||||||
|
Expected: All tests PASS |
||||||
|
|
||||||
|
- [ ] **Step 4: Commit** |
||||||
|
|
||||||
|
```bash |
||||||
|
git add explorer.py analysis/svd_labels.py |
||||||
|
git commit -m "refactor: move SVD_THEMES to module level for import" |
||||||
|
``` |
||||||
|
|
||||||
|
--- |
||||||
|
|
||||||
|
## Task 3: Update `axis_classifier.py` to use svd_labels |
||||||
|
|
||||||
|
**Files:** |
||||||
|
- Modify: `analysis/axis_classifier.py` |
||||||
|
- Modify: `tests/test_axis_label_fallback.py` |
||||||
|
|
||||||
|
- [ ] **Step 1: Write test for updated `display_label_for_modal`** |
||||||
|
|
||||||
|
```python |
||||||
|
# tests/test_axis_label_fallback.py (update existing tests) |
||||||
|
|
||||||
|
def test_display_label_for_modal_uses_svd_themes(): |
||||||
|
"""Test that display_label_for_modal uses SVD_THEMES for fallback labels.""" |
||||||
|
from analysis.axis_classifier import display_label_for_modal |
||||||
|
|
||||||
|
# None should return fallback from SVD_THEMES |
||||||
|
x_label = display_label_for_modal(None, "x") |
||||||
|
y_label = display_label_for_modal(None, "y") |
||||||
|
|
||||||
|
# Should return component 1 and 2 labels from SVD_THEMES |
||||||
|
assert "EU-integratie" in x_label or "Nationalisme" in x_label |
||||||
|
assert "Populistisch" in y_label or "Institutioneel" in y_label |
||||||
|
|
||||||
|
|
||||||
|
def test_display_label_for_modal_maps_as_labels(): |
||||||
|
"""Test that 'As 1' and 'As 2' are mapped to semantic labels.""" |
||||||
|
from analysis.axis_classifier import display_label_for_modal |
||||||
|
|
||||||
|
x_label = display_label_for_modal("As 1", "x") |
||||||
|
y_label = display_label_for_modal("As 2", "y") |
||||||
|
|
||||||
|
# Should return component 1 and 2 labels |
||||||
|
assert "EU-integratie" in x_label or "Nationalisme" in x_label |
||||||
|
assert "Populistisch" in y_label or "Institutioneel" in y_label |
||||||
|
``` |
||||||
|
|
||||||
|
- [ ] **Step 2: Run test to verify it fails** |
||||||
|
|
||||||
|
Run: `uv run pytest tests/test_axis_label_fallback.py -v` |
||||||
|
Expected: Some tests FAIL (current implementation returns hardcoded labels) |
||||||
|
|
||||||
|
- [ ] **Step 3: Update `axis_classifier.py` to use svd_labels** |
||||||
|
|
||||||
|
```python |
||||||
|
# analysis/axis_classifier.py (update the module) |
||||||
|
|
||||||
|
# Remove the hardcoded _LABELS dict (lines 25-35) |
||||||
|
# Replace with imports from svd_labels |
||||||
|
|
||||||
|
# At the top of the file, after imports: |
||||||
|
from analysis.svd_labels import get_svd_label, get_fallback_labels |
||||||
|
|
||||||
|
# Remove _LABELS dict entirely |
||||||
|
|
||||||
|
# Update display_label_for_modal function (lines 42-55): |
||||||
|
def display_label_for_modal(modal_label: Optional[str], axis: str) -> str: |
||||||
|
"""Return a user-facing axis label for a modal/internal label. |
||||||
|
|
||||||
|
Maps numeric fallback names 'As 1' / 'Stempatroon As 1' to the |
||||||
|
semantic labels from SVD_THEMES. Any other label is returned unchanged. |
||||||
|
None is treated as the semantic fallback for the axis. |
||||||
|
""" |
||||||
|
if modal_label is None: |
||||||
|
# Fallback to component 1 (x) or 2 (y) |
||||||
|
comp = 1 if axis == "x" else 2 |
||||||
|
return get_svd_label(comp) |
||||||
|
|
||||||
|
# Map "As 1" / "As 2" to semantic labels |
||||||
|
if axis == "x" and modal_label in ("As 1", "Stempatroon As 1"): |
||||||
|
return get_svd_label(1) |
||||||
|
if axis == "y" and modal_label in ("As 2", "Stempatroon As 2"): |
||||||
|
return get_svd_label(2) |
||||||
|
|
||||||
|
return modal_label |
||||||
|
``` |
||||||
|
|
||||||
|
- [ ] **Step 4: Update `_INTERPRETATION_TEMPLATES` to use svd_labels** |
||||||
|
|
||||||
|
The interpretation templates still need to reference the correct labels. Keep them as-is for now since they're used for motion classification, not display. |
||||||
|
|
||||||
|
- [ ] **Step 5: Update `_MOTION_LABEL_TEMPLATE_KEY` to use svd_labels** |
||||||
|
|
||||||
|
Keep the mapping as-is for now since it's used for motion classification. |
||||||
|
|
||||||
|
- [ ] **Step 6: Update `_KEYWORDS` to use svd_labels** |
||||||
|
|
||||||
|
Keep the keywords as-is for now since they're used for motion classification. |
||||||
|
|
||||||
|
- [ ] **Step 7: Update fallback label references in classify_axes** |
||||||
|
|
||||||
|
Find lines 610 and 615 where `_LABELS["fallback_x"]` and `_LABELS["fallback_y"]` are used and replace: |
||||||
|
|
||||||
|
```python |
||||||
|
# Before: |
||||||
|
x_lbl = _LABELS["fallback_x"] |
||||||
|
y_lbl = _LABELS["fallback_y"] |
||||||
|
|
||||||
|
# After: |
||||||
|
x_lbl, y_lbl = get_fallback_labels() |
||||||
|
``` |
||||||
|
|
||||||
|
- [ ] **Step 8: Run tests to verify they pass** |
||||||
|
|
||||||
|
Run: `uv run pytest tests/test_axis_label_fallback.py tests/test_political_compass.py -v` |
||||||
|
Expected: All tests PASS |
||||||
|
|
||||||
|
- [ ] **Step 9: Commit** |
||||||
|
|
||||||
|
```bash |
||||||
|
git add analysis/axis_classifier.py tests/test_axis_label_fallback.py |
||||||
|
git commit -m "refactor: use svd_labels for axis labels in axis_classifier" |
||||||
|
``` |
||||||
|
|
||||||
|
--- |
||||||
|
|
||||||
|
## Task 4: Update explorer.py to use svd_labels for fallback labels |
||||||
|
|
||||||
|
**Files:** |
||||||
|
- Modify: `explorer.py` |
||||||
|
|
||||||
|
- [ ] **Step 1: Find and update fallback label references in explorer.py** |
||||||
|
|
||||||
|
Find lines 1440-1441 where hardcoded fallback labels are used: |
||||||
|
|
||||||
|
```python |
||||||
|
# Before: |
||||||
|
_x_label = _raw_x or "EU-integratie–Nationalisme" |
||||||
|
_y_label = _raw_y or "Populistisch–Institutioneel" |
||||||
|
|
||||||
|
# After: |
||||||
|
from analysis.svd_labels import get_fallback_labels |
||||||
|
_x_fallback, _y_fallback = get_fallback_labels() |
||||||
|
_x_label = _raw_x or _x_fallback |
||||||
|
_y_label = _raw_y or _y_fallback |
||||||
|
``` |
||||||
|
|
||||||
|
- [ ] **Step 2: Run tests to verify nothing broke** |
||||||
|
|
||||||
|
Run: `uv run pytest tests/test_political_compass.py -v` |
||||||
|
Expected: All tests PASS |
||||||
|
|
||||||
|
- [ ] **Step 3: Commit** |
||||||
|
|
||||||
|
```bash |
||||||
|
git add explorer.py |
||||||
|
git commit -m "refactor: use svd_labels for fallback labels in explorer" |
||||||
|
``` |
||||||
|
|
||||||
|
--- |
||||||
|
|
||||||
|
## Task 5: Add 1D party position charts for components 3-10 |
||||||
|
|
||||||
|
**Files:** |
||||||
|
- Modify: `explorer.py` |
||||||
|
- Test: `tests/test_explorer_chart.py` |
||||||
|
|
||||||
|
- [ ] **Step 1: Write test for 1D party position chart** |
||||||
|
|
||||||
|
```python |
||||||
|
# tests/test_explorer_chart.py (append) |
||||||
|
|
||||||
|
def test_render_party_axis_chart_1d_renders(): |
||||||
|
"""Test that _render_party_axis_chart_1d renders a figure.""" |
||||||
|
from explorer import _render_party_axis_chart_1d |
||||||
|
|
||||||
|
party_coords = { |
||||||
|
"VVD": (0.5,), |
||||||
|
"SP": (-0.6,), |
||||||
|
"PVV": (0.8,), |
||||||
|
"DENK": (-0.4,), |
||||||
|
} |
||||||
|
|
||||||
|
theme = { |
||||||
|
"label": "Test Component", |
||||||
|
"positive_pole": "Positive", |
||||||
|
"negative_pole": "Negative", |
||||||
|
"flip": False, |
||||||
|
} |
||||||
|
|
||||||
|
fig = _render_party_axis_chart_1d(party_coords, 3, theme) |
||||||
|
assert fig is not None |
||||||
|
# Check that figure has traces |
||||||
|
assert len(fig.data) > 0 |
||||||
|
``` |
||||||
|
|
||||||
|
- [ ] **Step 2: Run test to verify it fails** |
||||||
|
|
||||||
|
Run: `uv run pytest tests/test_explorer_chart.py::test_render_party_axis_chart_1d_renders -v` |
||||||
|
Expected: FAIL with "cannot import name '_render_party_axis_chart_1d'" |
||||||
|
|
||||||
|
- [ ] **Step 3: Implement `_render_party_axis_chart_1d` function in explorer.py** |
||||||
|
|
||||||
|
Add this function near `_render_party_axis_chart` (around line 2900): |
||||||
|
|
||||||
|
```python |
||||||
|
# explorer.py (add new function) |
||||||
|
|
||||||
|
def _render_party_axis_chart_1d( |
||||||
|
party_coords: dict, |
||||||
|
component: int, |
||||||
|
theme: dict, |
||||||
|
bootstrap_data: Optional[Dict] = None, |
||||||
|
) -> go.Figure: |
||||||
|
"""Render a 1D party position chart for a single SVD component. |
||||||
|
|
||||||
|
Args: |
||||||
|
party_coords: Dict mapping party name to (score,) tuple |
||||||
|
component: SVD component number (1-indexed) |
||||||
|
theme: Dict with label, positive_pole, negative_pole, flip |
||||||
|
bootstrap_data: Optional bootstrap confidence intervals |
||||||
|
|
||||||
|
Returns: |
||||||
|
Plotly Figure object |
||||||
|
""" |
||||||
|
import plotly.graph_objects as go |
||||||
|
import numpy as np |
||||||
|
|
||||||
|
if not party_coords: |
||||||
|
fig = go.Figure() |
||||||
|
fig.add_annotation( |
||||||
|
text="Geen partijposities beschikbaar", |
||||||
|
xref="paper", yref="paper", |
||||||
|
x=0.5, y=0.5, showarrow=False, |
||||||
|
font=dict(size=14) |
||||||
|
) |
||||||
|
return fig |
||||||
|
|
||||||
|
# Extract scores and parties |
||||||
|
parties = list(party_coords.keys()) |
||||||
|
scores = [coords[0] for coords in party_coords.values()] |
||||||
|
|
||||||
|
# Apply flip if needed |
||||||
|
flip = theme.get("flip", False) |
||||||
|
if flip: |
||||||
|
scores = [-s for s in scores] |
||||||
|
|
||||||
|
# Get party colors |
||||||
|
party_colors = [PARTY_COLOURS.get(p, "#9E9E9E") for p in parties] |
||||||
|
|
||||||
|
# Create horizontal bar chart |
||||||
|
fig = go.Figure() |
||||||
|
|
||||||
|
# Sort by score for better visualization |
||||||
|
sorted_indices = np.argsort(scores) |
||||||
|
sorted_parties = [parties[i] for i in sorted_indices] |
||||||
|
sorted_scores = [scores[i] for i in sorted_indices] |
||||||
|
sorted_colors = [party_colors[i] for i in sorted_indices] |
||||||
|
|
||||||
|
fig.add_trace(go.Bar( |
||||||
|
y=sorted_parties, |
||||||
|
x=sorted_scores, |
||||||
|
orientation='h', |
||||||
|
marker_color=sorted_colors, |
||||||
|
text=[f"{s:.2f}" for s in sorted_scores], |
||||||
|
textposition='outside', |
||||||
|
)) |
||||||
|
|
||||||
|
# Update layout |
||||||
|
label = theme.get("label", f"As {component}") |
||||||
|
positive_pole = theme.get("positive_pole", "Positief") |
||||||
|
negative_pole = theme.get("negative_pole", "Negatief") |
||||||
|
|
||||||
|
fig.update_layout( |
||||||
|
title=f"Partijposities — {label}", |
||||||
|
xaxis_title=f"{negative_pole} ← → {positive_pole}", |
||||||
|
yaxis_title="", |
||||||
|
height=max(400, len(parties) * 25), # Scale height with number of parties |
||||||
|
margin=dict(l=150), # Extra margin for party names |
||||||
|
showlegend=False, |
||||||
|
) |
||||||
|
|
||||||
|
# Add vertical line at x=0 |
||||||
|
fig.add_vline(x=0, line_dash="dash", line_color="gray", opacity=0.5) |
||||||
|
|
||||||
|
return fig |
||||||
|
``` |
||||||
|
|
||||||
|
- [ ] **Step 4: Update SVD Components tab to use 1D chart for components 3-10** |
||||||
|
|
||||||
|
Find the section in `build_svd_components_tab` where `_render_party_axis_chart` is called (around line 2823) and update: |
||||||
|
|
||||||
|
```python |
||||||
|
# explorer.py (update around line 2822-2830) |
||||||
|
|
||||||
|
# Before: |
||||||
|
if comp_sel in (1, 2): |
||||||
|
_render_party_axis_chart( |
||||||
|
party_coords, comp_sel, theme, bootstrap_data=bootstrap_data |
||||||
|
) |
||||||
|
else: |
||||||
|
st.caption( |
||||||
|
"_Partijposities zijn alleen beschikbaar voor de eerste twee SVD-assen._" |
||||||
|
) |
||||||
|
|
||||||
|
# After: |
||||||
|
if comp_sel in (1, 2): |
||||||
|
_render_party_axis_chart( |
||||||
|
party_coords, comp_sel, theme, bootstrap_data=bootstrap_data |
||||||
|
) |
||||||
|
else: |
||||||
|
# For components 3-10, show 1D party positions |
||||||
|
# Extract 1D scores for this component |
||||||
|
party_1d_coords = {} |
||||||
|
idx = comp_sel - 1 # Convert to 0-indexed |
||||||
|
for party, scores in party_scores.items(): |
||||||
|
if len(scores) > idx: |
||||||
|
party_1d_coords[party] = (scores[idx],) |
||||||
|
|
||||||
|
_render_party_axis_chart_1d( |
||||||
|
party_1d_coords, comp_sel, theme, bootstrap_data=None |
||||||
|
) |
||||||
|
``` |
||||||
|
|
||||||
|
- [ ] **Step 5: Run tests to verify they pass** |
||||||
|
|
||||||
|
Run: `uv run pytest tests/test_explorer_chart.py -v` |
||||||
|
Expected: All tests PASS |
||||||
|
|
||||||
|
- [ ] **Step 6: Commit** |
||||||
|
|
||||||
|
```bash |
||||||
|
git add explorer.py tests/test_explorer_chart.py |
||||||
|
git commit -m "feat: add 1D party position charts for SVD components 3-10" |
||||||
|
``` |
||||||
|
|
||||||
|
--- |
||||||
|
|
||||||
|
## Task 6: Auto-compute flip directions for all components |
||||||
|
|
||||||
|
**Files:** |
||||||
|
- Modify: `explorer.py` |
||||||
|
- Modify: `analysis/svd_labels.py` |
||||||
|
- Test: `tests/test_svd_labels.py` |
||||||
|
|
||||||
|
- [ ] **Step 1: Write test for auto-computing flip in SVD Components tab** |
||||||
|
|
||||||
|
```python |
||||||
|
# tests/test_svd_labels.py (append) |
||||||
|
|
||||||
|
def test_auto_flip_computation_for_all_components(): |
||||||
|
"""Test that flip directions are computed correctly for all components.""" |
||||||
|
from analysis.svd_labels import compute_flip_direction |
||||||
|
|
||||||
|
# Simulate party scores for 10 components |
||||||
|
# Right parties should have positive scores on component 1 (EU-integratie) |
||||||
|
# Left parties should have negative scores on component 1 |
||||||
|
party_scores = { |
||||||
|
"VVD": [0.5] * 10, # Right party, positive on all components |
||||||
|
"PVV": [0.8] * 10, # Right party |
||||||
|
"SP": [-0.6] * 10, # Left party, negative on all components |
||||||
|
"DENK": [-0.4] * 10, # Left party |
||||||
|
} |
||||||
|
|
||||||
|
# For all components, right_mean > left_mean, so flip should be False |
||||||
|
for comp in range(1, 11): |
||||||
|
flip = compute_flip_direction(comp, party_scores) |
||||||
|
assert flip is False, f"Component {comp} should not flip" |
||||||
|
|
||||||
|
# Now test with right parties on left (negative scores) |
||||||
|
party_scores_left = { |
||||||
|
"VVD": [-0.5] * 10, |
||||||
|
"PVV": [-0.8] * 10, |
||||||
|
"SP": [0.6] * 10, |
||||||
|
"DENK": [0.4] * 10, |
||||||
|
} |
||||||
|
|
||||||
|
# For all components, right_mean < left_mean, so flip should be True |
||||||
|
for comp in range(1, 11): |
||||||
|
flip = compute_flip_direction(comp, party_scores_left) |
||||||
|
assert flip is True, f"Component {comp} should flip" |
||||||
|
``` |
||||||
|
|
||||||
|
- [ ] **Step 2: Run test to verify it passes** |
||||||
|
|
||||||
|
Run: `uv run pytest tests/test_svd_labels.py::test_auto_flip_computation_for_all_components -v` |
||||||
|
Expected: PASS (function already implemented in Task 1) |
||||||
|
|
||||||
|
- [ ] **Step 3: Update SVD Components tab to compute flip dynamically** |
||||||
|
|
||||||
|
In `build_svd_components_tab`, after loading party_scores, compute flip for each component: |
||||||
|
|
||||||
|
```python |
||||||
|
# explorer.py (in build_svd_components_tab, after loading party_scores) |
||||||
|
|
||||||
|
from analysis.svd_labels import compute_flip_direction |
||||||
|
|
||||||
|
# After party_scores is loaded (around line 2800) |
||||||
|
# Compute flip directions for all components |
||||||
|
computed_flips = {} |
||||||
|
for comp in range(1, 11): |
||||||
|
computed_flips[comp] = compute_flip_direction(comp, party_scores) |
||||||
|
|
||||||
|
# Update SVD_THEMES with computed flips |
||||||
|
for comp, flip in computed_flips.items(): |
||||||
|
if comp in SVD_THEMES: |
||||||
|
SVD_THEMES[comp]["flip"] = flip |
||||||
|
``` |
||||||
|
|
||||||
|
- [ ] **Step 4: Run tests to verify nothing broke** |
||||||
|
|
||||||
|
Run: `uv run pytest tests/test_svd_labels.py tests/test_explorer_chart.py -v` |
||||||
|
Expected: All tests PASS |
||||||
|
|
||||||
|
- [ ] **Step 5: Commit** |
||||||
|
|
||||||
|
```bash |
||||||
|
git add explorer.py analysis/svd_labels.py tests/test_svd_labels.py |
||||||
|
git commit -m "feat: auto-compute flip directions for all SVD components" |
||||||
|
``` |
||||||
|
|
||||||
|
--- |
||||||
|
|
||||||
|
## Task 7: Update existing tests for new label system |
||||||
|
|
||||||
|
**Files:** |
||||||
|
- Modify: `tests/test_political_compass.py` |
||||||
|
- Modify: `tests/test_axis_label_fallback.py` |
||||||
|
|
||||||
|
- [ ] **Step 1: Run all tests to identify failures** |
||||||
|
|
||||||
|
Run: `uv run pytest tests/ -v --tb=short` |
||||||
|
Expected: Some tests may fail due to label changes |
||||||
|
|
||||||
|
- [ ] **Step 2: Fix any failing tests** |
||||||
|
|
||||||
|
Update test assertions to use new labels from SVD_THEMES: |
||||||
|
|
||||||
|
```python |
||||||
|
# tests/test_political_compass.py (update assertions) |
||||||
|
|
||||||
|
# Before: |
||||||
|
assert "Links–Rechts" in x_label |
||||||
|
|
||||||
|
# After: |
||||||
|
assert "EU-integratie" in x_label or "Nationalisme" in x_label |
||||||
|
``` |
||||||
|
|
||||||
|
- [ ] **Step 3: Run all tests to verify they pass** |
||||||
|
|
||||||
|
Run: `uv run pytest tests/ -v` |
||||||
|
Expected: All tests PASS |
||||||
|
|
||||||
|
- [ ] **Step 4: Commit** |
||||||
|
|
||||||
|
```bash |
||||||
|
git add tests/ |
||||||
|
git commit -m "test: update tests for unified SVD label system" |
||||||
|
``` |
||||||
|
|
||||||
|
--- |
||||||
|
|
||||||
|
## Task 8: Final verification and cleanup |
||||||
|
|
||||||
|
**Files:** |
||||||
|
- All modified files |
||||||
|
|
||||||
|
- [ ] **Step 1: Run full test suite** |
||||||
|
|
||||||
|
Run: `uv run pytest tests/ -v` |
||||||
|
Expected: All tests PASS |
||||||
|
|
||||||
|
- [ ] **Step 2: Run Streamlit app to verify UI** |
||||||
|
|
||||||
|
Run: `uv run streamlit run explorer.py` |
||||||
|
Expected: App starts without errors |
||||||
|
|
||||||
|
- [ ] **Step 3: Manually verify in UI** |
||||||
|
- Open SVD Components tab |
||||||
|
- Check that components 1-10 show correct labels |
||||||
|
- Check that party position charts render for all components |
||||||
|
- Check that right parties appear on the right side of axes |
||||||
|
|
||||||
|
- [ ] **Step 4: Final commit** |
||||||
|
|
||||||
|
```bash |
||||||
|
git add -A |
||||||
|
git commit -m "feat: complete SVD label unification" |
||||||
|
``` |
||||||
|
|
||||||
|
--- |
||||||
|
|
||||||
|
## Summary |
||||||
|
|
||||||
|
This plan creates a unified label system for SVD components with: |
||||||
|
1. Single source of truth in `SVD_THEMES` |
||||||
|
2. Automatic flip direction computation based on party centroids |
||||||
|
3. 1D party position charts for components 3-10 |
||||||
|
4. Consistent labels across compass, trajectory, and SVD Components views |
||||||
Loading…
Reference in new issue