docs: add improvement roadmap, research notes, and solution docs
- Add 2026-04-24 ROADMAP with 5 phases / 17 items - Add detailed implementation plans for P1-001 through P4-005 - Add research artifacts and solution docs from ledger merge - Add test for SVD component 1 compass alignment
This commit is contained in:
@@ -0,0 +1,125 @@
|
||||
---
|
||||
module: llm-classification
|
||||
tags: [polarization, nlp, prompt-design, democratic-norms]
|
||||
problem_type: classification-schema-design
|
||||
date: 2026-04-05
|
||||
reviewed_by:
|
||||
- correctness-reviewer
|
||||
- domain-expert (Dutch politics)
|
||||
- clarity-reviewer
|
||||
---
|
||||
|
||||
# LLM Motion Classification: Prompt Design Lessons
|
||||
|
||||
## Problem
|
||||
|
||||
Wanted to classify 28,000 Dutch parliamentary motions by "extremity" to measure polarization over time.
|
||||
|
||||
Initial prompt conflated multiple concepts:
|
||||
- Democratic norm erosion
|
||||
- Populist rhetoric style
|
||||
- Group targeting
|
||||
- Restrictiveness vs permissiveness
|
||||
|
||||
## Initial v1 Design (Flawed)
|
||||
|
||||
```python
|
||||
EXTREMITY_SCORE (1-5):
|
||||
- 1: Mainstream
|
||||
- 5: "Undermines checks & balances, threatens rule of law,
|
||||
discriminates groups, populist rhetoric"
|
||||
```
|
||||
|
||||
**Problems identified:**
|
||||
1. Populist rhetoric is style, not substance — shouldn't be in same score as democratic erosion
|
||||
2. "Extreme" undefined — compared to what baseline?
|
||||
3. Score 4/5 boundary unclear
|
||||
4. TARGETED_GROUP redundant with EXTREMITY_SCORE
|
||||
5. EU deviation always = score 5 (too broad)
|
||||
6. Missing Dutch-specific patterns (Nexit, referendum abolition)
|
||||
|
||||
## Refined v2 Design (Four Orthogonal Dimensions)
|
||||
|
||||
### 1. DEMOCRATIC_EROSION (0-4) — Substance only
|
||||
| Score | Label | Criteria |
|
||||
|-------|-------|----------|
|
||||
| 0 | None | No impact on democratic norms |
|
||||
| 1 | Minor | Small procedural deviations |
|
||||
| 2 | Moderate | Significant policy change, within constitutional framework |
|
||||
| 3 | Significant | Fundamental change to checks & balances |
|
||||
| 4 | Critical | Undermines rule of law, press freedom, systematic discrimination |
|
||||
|
||||
**Decision rules:**
|
||||
- Score 4 ONLY if: (a) direct attack on judiciary/press, OR (b) systematic discrimination in law, OR (c) call to violate international treaties
|
||||
- Score 3 if: (a) abolish referendum, OR (b) fundamentally question EU cooperation, OR (c) significantly expand executive powers
|
||||
|
||||
### 2. POPULIST_STYLE (0-1) — Style only
|
||||
Independent of democratic impact. A motion can be populist (1) but democratic (0).
|
||||
|
||||
**Indicators:**
|
||||
- "Het volk" vs "de elite/den Haag"
|
||||
- "Wij vs zij" framing
|
||||
- Call for "direct democracy" without checks
|
||||
- Emotionally charged language
|
||||
|
||||
### 3. GROUP_TARGETING (0-2) — Targeting only
|
||||
| Score | Label |
|
||||
|-------|-------|
|
||||
| 0 | Universal — general policy |
|
||||
| 1 | Indirect — general policy that disproportionately affects groups |
|
||||
| 2 | Direct — explicitly targets specific population group |
|
||||
|
||||
### 4. RESTRICTIVENESS (-1 to +1) — Direction only
|
||||
| Score | Label |
|
||||
|-------|-------|
|
||||
| -1 | Expansive |
|
||||
| 0 | Neutral |
|
||||
| +1 | Restrictive |
|
||||
|
||||
## Key Lessons Learned
|
||||
|
||||
### 1. Separate Style from Substance
|
||||
Populist rhetoric ≠ democratic erosion. A mainstream party using strong language isn't anti-democratic. Conflating them causes false positives.
|
||||
|
||||
### 2. Make Dimensions Orthogonal
|
||||
- DEMOCRATIC_EROSION × RESTRICTIVENESS: A policy can be erosive AND restrictive, or erosive AND permissive
|
||||
- POPULIST_STYLE × DEMOCRATIC_EROSION: Can have populist (1) with democratic (0), and vice versa
|
||||
- GROUP_TARGETING × RESTRICTIVENESS: Restrictive ≠ targeted (and vice versa)
|
||||
|
||||
### 3. Add Decision Rules for Boundaries
|
||||
Vague transitions ("significant" → "critical") cause inconsistency. Define specific triggers:
|
||||
```
|
||||
Score 4 ONLY when: (a) OR (b) OR (c)
|
||||
Score 3 when: (a) OR (b) OR (c)
|
||||
```
|
||||
|
||||
### 4. Gradate EU Deviation
|
||||
Not all EU deviation is equal:
|
||||
- Dutch implementation of EU policy → erosion 0-1
|
||||
- Nexit / leave EU → erosion 3-4
|
||||
- Violate EU rules → erosion 2-3
|
||||
|
||||
### 5. Include Domain-Specific Patterns
|
||||
Dutch context matters:
|
||||
- Referendum abolition = score 3
|
||||
- "Den Haag" / "establishment" attacks = check for populist style
|
||||
- Nexit = score 3-4 depending on framing
|
||||
|
||||
### 6. Define Reference Baselines
|
||||
"Abnormal" compared to what?
|
||||
- 2016 consensus
|
||||
- EU norms
|
||||
- Historical Dutch practice
|
||||
- International standards
|
||||
|
||||
## Testing Recommendations
|
||||
|
||||
1. **Calibration set**: 50 motions with expert annotations before production
|
||||
2. **Boundary cases**: Test score 3/4 transitions explicitly
|
||||
3. **Cross-rater reliability**: Multiple classifiers on same motions
|
||||
4. **Domain-specific test cases**: Migration, EU, constitutional reform
|
||||
|
||||
## Files
|
||||
|
||||
- `scripts/classify_motions.py` — Implementation with v2 prompt
|
||||
- `docs/research/motion-classification-prompt-v2.md` — Full prompt documentation
|
||||
@@ -0,0 +1,80 @@
|
||||
---
|
||||
title: "Policy Extremity vs Voting Extremity: Independent Phenomena"
|
||||
date: 2026-04-05
|
||||
module: analysis
|
||||
problem_type: research
|
||||
component: motion-analysis
|
||||
tags: [polarization, policy-extremity, voting-extremity, svd, embedding-norm]
|
||||
---
|
||||
|
||||
# Policy Extremity vs Voting Extremity: Independent Phenomena
|
||||
|
||||
## Key Finding
|
||||
|
||||
**Voting extremity** (how divided parliament is) and **policy extremity** (how far motions are from political center) are **independent phenomena** with opposite trends:
|
||||
|
||||
| Measure | 2016 | 2026 | Trend |
|
||||
|---------|------|------|-------|
|
||||
| **Voting Extremity** | 0.70 | 0.46 | More divided |
|
||||
| **Policy Extremity** | 9.0 | 4.2 | Less extreme |
|
||||
|
||||
**Correlation: r = -0.011** (essentially zero)
|
||||
|
||||
## Definitions
|
||||
|
||||
### Voting Extremity
|
||||
- **Formula**: margin / total votes
|
||||
- **Interpretation**: How divided parliament is
|
||||
- 1.0 = unanimous (all votes same direction)
|
||||
- 0.0 = perfectly split (50-50)
|
||||
- **Trend**: Increased (more close votes in recent years)
|
||||
|
||||
### Policy Extremity
|
||||
- **Formula**: L2 norm of SVD embedding vector
|
||||
- **Interpretation**: How "far out" a motion is in political semantic space
|
||||
- **Trend**: Decreased (motions closer to political center)
|
||||
|
||||
## Analysis
|
||||
|
||||
### Why Are They Independent?
|
||||
|
||||
1. **Voting extremity** captures **how parties divide** on issues
|
||||
2. **Policy extremity** captures **where motions sit** in policy space
|
||||
|
||||
A motion can be:
|
||||
- Near the center (low policy extremity) but divide parties 50-50 (high voting extremity)
|
||||
- Far from center (high policy extremity) but pass unanimously (low voting extremity)
|
||||
|
||||
### Historical Pattern
|
||||
|
||||
- **2016**: Coalition passed "extreme" motions (legislative proposals) with consensus
|
||||
- **2026**: More divided votes on "moderate" motions (procedural/administrative)
|
||||
|
||||
### Interpretation
|
||||
|
||||
The parliament has become **more divided in how it votes**, but the **policies being passed are actually less extreme** in semantic space.
|
||||
|
||||
This suggests:
|
||||
- The polarization is about **different issues** dividing parties
|
||||
- The "extremes" that pass are now closer to mainstream positions
|
||||
- What changed is **what divides parties**, not **how radical the policies are**
|
||||
|
||||
## Visualization
|
||||
|
||||
See `docs/research/voting_vs_policy_extremity.png`
|
||||
|
||||
## Methodology
|
||||
|
||||
```python
|
||||
# Voting extremity = margin / total
|
||||
voting_extremity = abs(votes_for - votes_against) / total_votes
|
||||
|
||||
# Policy extremity = L2 norm of SVD embedding
|
||||
policy_extremity = np.linalg.norm(embedding_vector)
|
||||
```
|
||||
|
||||
## Conclusion
|
||||
|
||||
These findings confirm that **voting extremity ≠ policy extremity**. They capture different aspects of parliamentary behavior and should be analyzed separately.
|
||||
|
||||
The increase in voting extremity reflects genuine polarization in parliamentary divisions. But the decrease in policy extremity suggests that the policies actually being passed are not more radical—they're just more contested.
|
||||
@@ -0,0 +1,146 @@
|
||||
---
|
||||
title: "Quantifying Political Extremity: Voting vs Policy"
|
||||
date: 2026-04-05
|
||||
module: analysis
|
||||
problem_type: research
|
||||
component: motion-analysis
|
||||
tags: [polarization, voting-extremity, policy-extremity, embedding-analysis, parliamentary-motion]
|
||||
---
|
||||
|
||||
# Quantifying Political Extremity: Voting vs Policy
|
||||
|
||||
## Context
|
||||
|
||||
Initial analysis of parliamentary motions sought to measure polarization by examining how "extreme" policies have become. The hypothesis was that extremes on both sides became more extreme. The analysis revealed this hypothesis was incorrect — and surfaced two independent phenomena.
|
||||
|
||||
## Guidance
|
||||
|
||||
### Key Finding: Two Independent Measures of Extremity
|
||||
|
||||
**Voting Extremity** and **Policy Extremity** are independent phenomena with different trends:
|
||||
|
||||
| Measure | 2016 | 2026 | Trend |
|
||||
|---------|------|------|-------|
|
||||
| **Voting Extremity** (margin/total) | 0.70 | 0.46 | Parliament votes more closely |
|
||||
| **Policy Extremity** (embedding distance from mainstream) | 5.65 | 4.17 | Policies are less extreme |
|
||||
|
||||
**Correlation: r ≈ 0** — these measures are statistically independent.
|
||||
|
||||
### What Each Measures
|
||||
|
||||
**Voting Extremity** = `abs(votes_for - votes_against) / total_votes`
|
||||
- 1.0 = unanimous (all votes same direction)
|
||||
- 0.0 = perfectly split (50-50)
|
||||
- Captures how divided parliament is when voting
|
||||
|
||||
**Policy Extremity** = `||embedding - mainstream_centroid||`
|
||||
- Euclidean distance in text embedding space (2560 dims)
|
||||
- Captures how far a motion is from the political center
|
||||
|
||||
### How to Measure Each
|
||||
|
||||
```python
|
||||
# Voting Extremity
|
||||
margin = abs(votes_for - votes_against)
|
||||
total = votes_for + votes_against
|
||||
voting_extremity = margin / total
|
||||
|
||||
# Policy Extremity (using text embeddings, not SVD)
|
||||
from embeddings table (qwen/qwen3-embedding-4b)
|
||||
policy_extremity = np.linalg.norm(motion_embedding - mainstream_centroid)
|
||||
```
|
||||
|
||||
### Why Use Text Embeddings (Not SVD)
|
||||
|
||||
SVD embeddings are fitted on **voting patterns**, capturing how parties vote together. They measure **voting extremity**, not **policy extremity**.
|
||||
|
||||
For policy content, use **raw text embeddings** (`embeddings` table, 2560 dimensions) which are computed from motion text only.
|
||||
|
||||
### Bipartisan Anchor Approach
|
||||
|
||||
Define the "mainstream" as the centroid of bipartisan motions (80%+ parties vote the same way):
|
||||
|
||||
```python
|
||||
# Find bipartisan motions
|
||||
bipartisan = [m for m in motions if majority_vote_pct >= 0.80]
|
||||
|
||||
# Compute mainstream centroid
|
||||
mainstream_centroid = mean([m.embedding for m in bipartisan])
|
||||
|
||||
# Measure policy extremity
|
||||
policy_extremity = ||motion.embedding - mainstream_centroid||
|
||||
```
|
||||
|
||||
## Why This Matters
|
||||
|
||||
The hypothesis "extremes became more extreme" was wrong because:
|
||||
|
||||
1. **Voting extremity increased** — parliament votes more divided now
|
||||
2. **Policy extremity decreased** — even extreme motions are closer to center
|
||||
|
||||
This means: what divides parties changed, not how radical the policies are.
|
||||
|
||||
## Quantifying Mainstream Shift
|
||||
|
||||
Using 2018 as baseline ("last normal year"):
|
||||
|
||||
| Period | Distance from 2018 | Interpretation |
|
||||
|--------|-------------------|----------------|
|
||||
| 2016-2018 | ~0.22 | Similar mainstream |
|
||||
| **2019** | **0.46** | Shift begins |
|
||||
| 2020-2026 | **~0.71** | New stable mainstream |
|
||||
|
||||
The mainstream shifted **0.71 units** after 2018 and has remained stable.
|
||||
|
||||
### Coalition Shift on Migration Policy
|
||||
|
||||
Parties that once opposed strict migration now vote for them:
|
||||
|
||||
| Party | 2016-2018 | 2025-2026 | Change |
|
||||
|-------|------------|------------|--------|
|
||||
| VVD | 100% voor | 78% voor | ↓ |
|
||||
| CDA | 100% voor | 81% voor | ↓ |
|
||||
| D66 | 100% voor | 60% voor | ↓↓ |
|
||||
| PVV | 20% voor | 56% voor | ↑↑ |
|
||||
| NSC | 0% (new) | 56% voor | new |
|
||||
| BBB | 0% (new) | 79% voor | new |
|
||||
|
||||
## When to Apply
|
||||
|
||||
- When analyzing parliamentary polarization trends
|
||||
- When comparing policy extremity across time periods
|
||||
- When studying coalition formation and party positioning
|
||||
- When testing hypotheses about political extremism
|
||||
|
||||
## Examples
|
||||
|
||||
### Correct Analysis
|
||||
```python
|
||||
# Compare voting extremity and policy extremity separately
|
||||
voting_ext = compute_voting_margin(motion)
|
||||
policy_ext = compute_embedding_distance(motion, mainstream_centroid)
|
||||
|
||||
# Plot both trends independently
|
||||
plot_trend(years, voting_ext, label="Voting Extremity")
|
||||
plot_trend(years, policy_ext, label="Policy Extremity")
|
||||
```
|
||||
|
||||
### Incorrect Analysis
|
||||
```python
|
||||
# DON'T use SVD scores to measure policy extremity
|
||||
svd_score = motion.svd_vector[0] # This measures voting pattern, not content!
|
||||
|
||||
# DO use text embeddings for policy content
|
||||
text_embedding = embeddings_table[motion.id]
|
||||
```
|
||||
|
||||
## Related Findings
|
||||
|
||||
- `svd-stability-vs-overtone-shift.md` — SVD axes measure voting structure, not semantics
|
||||
- `policy-extremity-vs-voting-extremity.md` — Initial documentation of the distinction
|
||||
|
||||
## Visualizations
|
||||
|
||||
- `docs/research/polarization_comprehensive.png` — Combined view of all metrics
|
||||
- `docs/research/mainstream_shift.png` — Mainstream shift over time
|
||||
- `docs/research/voting_vs_policy_extremity.png` — Independent trends
|
||||
Reference in New Issue
Block a user