You can not select more than 25 topics
Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
252 lines
11 KiB
252 lines
11 KiB
---
|
|
title: Visualize and Report Migration-Anti-Democratic Overlap Findings
|
|
type: feat
|
|
status: active
|
|
date: 2026-05-08
|
|
---
|
|
|
|
# Visualize and Report Migration-Anti-Democratic Overlap Findings
|
|
|
|
## Summary
|
|
|
|
Turn the Direction 3 analysis (migration as the dominant vehicle for anti-democratic rhetoric in right-wing motions) into a publishable artifact: committed script, visualizations, mechanism-coded extreme motions, and a findings report.
|
|
|
|
---
|
|
|
|
## Problem Frame
|
|
|
|
The Direction 3 analysis script `analysis/right_wing/direction3_migration_antidemocratic.py` has been written and run successfully against all 2,986 scored right-wing motions. It uncovered strong evidence that migration is the primary carrier of anti-democratic extremity (45% of ≥4.0-scored motions are migration-related). However, the findings currently exist only as console output. They need to be preserved, visualized, deepened with mechanism coding, and compiled into a shareable report.
|
|
|
|
---
|
|
|
|
## Requirements
|
|
|
|
- R1. The analysis script is committed to the feature branch.
|
|
- R2. Five focused charts visualize the key findings for immediate readability.
|
|
- R3. The 212 motions scoring ≥3.5 extremity are mechanism-coded via lightweight LLM batch.
|
|
- R4. A markdown report compiles numbers, charts, and mechanism coding into a narrative.
|
|
- R5. All artifacts are saved under `reports/right_wing_migration_antidemocratic/`.
|
|
|
|
---
|
|
|
|
## Scope Boundaries
|
|
|
|
- No new LLM pipeline infrastructure (reuses existing `ai_provider.chat_completion_json_parallel` pattern from `extremity_scorer.py`).
|
|
- No changes to the database schema.
|
|
- No Streamlit UI integration (this is a research artifact, not a UI feature).
|
|
- No anti-democratic scoring pipeline for the full 2,986 motions (that would be Direction 2, deferred).
|
|
|
|
### Deferred to Follow-Up Work
|
|
|
|
- Full anti-democratic scoring pipeline (Direction 2): separate plan if mechanism coding validates the concept.
|
|
- Streamlit tab to expose right-wing analysis tables: separate UI work.
|
|
|
|
---
|
|
|
|
## Context & Research
|
|
|
|
### Relevant Code and Patterns
|
|
|
|
- `analysis/right_wing/direction3_migration_antidemocratic.py` — the analysis script to commit
|
|
- `analysis/right_wing/extremity_scorer.py` — LLM batch pattern with `chat_completion_json_parallel`
|
|
- `analysis/right_wing/derive_categories.py` — two-phase LLM approach (derive taxonomy, then apply)
|
|
- `scripts/motion_drift.py` — chart generation pattern using matplotlib, saved to `reports/`
|
|
- `analysis/config.py` — `CANONICAL_RIGHT`, `CANONICAL_LEFT`, `PARTY_COLOURS`
|
|
|
|
### Institutional Learnings
|
|
|
|
- `docs/solutions/best-practices/svd-labels-voting-patterns-not-semantics.md` — SVD labels should reflect voting patterns, not semantic content. Same discipline applies here: mechanism coding should reflect procedural/institutional characteristics, not just topic keywords.
|
|
|
|
---
|
|
|
|
## Key Technical Decisions
|
|
|
|
- **Matplotlib for charts:** The repo already uses matplotlib in `scripts/motion_drift.py` and `explorer.py`. No new charting library.
|
|
- **Reuse existing LLM batch infrastructure:** The mechanism coder will call `ai_provider.chat_completion_json_parallel` with a JSON schema, same pattern as extremity/sentiment scoring. Batch size 10, ~22 batches for 212 motions.
|
|
- **Mechanism taxonomy (6 classes):** punitive, exclusionary, sovereignty-claiming, procedural-breaking, institutional-dismantling, none. Chosen to capture the observed framing evolution from "keep them out" to "dismantle the system that lets them in."
|
|
- **Reports directory:** `reports/right_wing_migration_antidemocratic/` mirrors the `reports/drift/` pattern.
|
|
|
|
---
|
|
|
|
## Open Questions
|
|
|
|
### Resolved During Planning
|
|
|
|
- **Q: Which visualization library?** Matplotlib — already used in the repo.
|
|
- **Q: How many mechanisms to code?** 6-class taxonomy, validated against the 5.0-scored motion sample.
|
|
|
|
### Deferred to Implementation
|
|
|
|
- **Q: Exact chart styling and color palette** — depends on what looks good when rendered; iterate visually.
|
|
- **Q: Whether to include migration-adjacent motions in the mechanism coding** — start with pure migration (404), expand to adjacent (55) if time permits.
|
|
|
|
---
|
|
|
|
## Implementation Units
|
|
|
|
- U1. **Commit Direction 3 Analysis Script**
|
|
|
|
**Goal:** Save the working analysis script to the feature branch.
|
|
|
|
**Requirements:** R1
|
|
|
|
**Dependencies:** None
|
|
|
|
**Files:**
|
|
- Create: `analysis/right_wing/direction3_migration_antidemocratic.py`
|
|
|
|
**Approach:**
|
|
- The script already exists and produces correct output. Stage and commit it with a conventional message.
|
|
|
|
**Test scenarios:**
|
|
- Test expectation: none — this is a standalone analysis script with no testable behavior changes.
|
|
|
|
**Verification:**
|
|
- Script is committed to `feat/right-wing-motion-analysis` and runs without errors.
|
|
|
|
---
|
|
|
|
- U2. **Generate Five Key Visualizations**
|
|
|
|
**Goal:** Produce charts that make the Direction 3 findings immediately readable.
|
|
|
|
**Requirements:** R2, R5
|
|
|
|
**Dependencies:** U1
|
|
|
|
**Files:**
|
|
- Create: `analysis/right_wing/visualize_direction3.py`
|
|
- Output: `reports/right_wing_migration_antidemocratic/*.png`
|
|
|
|
**Approach:**
|
|
- Write a script that queries the scored data and generates 5 charts:
|
|
1. **Stacked bar:** Category breakdown of ≥4.0 extremity motions (migration's 44.8% dominance)
|
|
2. **Line chart:** Migration motion volume + avg extremity by year (2018–2026)
|
|
3. **Horizontal bar:** Party avg extremity on migration (Wilders 3.79, Eerdmans 2.69, etc.)
|
|
4. **Grouped bar:** Migration sentiment by extremity bucket (the −0.717 drop)
|
|
5. **Treemap or stacked bar:** Migration + migration-adjacent by category (15.4% total footprint)
|
|
- Follow the matplotlib patterns from `scripts/motion_drift.py` and `explorer.py`.
|
|
- Save charts to `reports/right_wing_migration_antidemocratic/`.
|
|
|
|
**Patterns to follow:**
|
|
- `scripts/motion_drift.py` — chart saving with `plt.savefig()`, DPI settings
|
|
- `explorer.py` — color palette using `analysis/config.PARTY_COLOURS`
|
|
|
|
**Test scenarios:**
|
|
- Happy path: script runs and produces 5 PNG files in the reports directory.
|
|
- Edge case: empty result set for a chart component should render gracefully (e.g., skip treemap segment with zero count).
|
|
|
|
**Verification:**
|
|
- 5 PNG files exist in `reports/right_wing_migration_antidemocratic/` and render legible charts.
|
|
|
|
---
|
|
|
|
- U3. **Mechanism-Code the 212 Extreme Motions**
|
|
|
|
**Goal:** Classify the anti-democratic mechanism for each motion scoring ≥3.5 extremity.
|
|
|
|
**Requirements:** R3
|
|
|
|
**Dependencies:** U1
|
|
|
|
**Files:**
|
|
- Create: `analysis/right_wing/mechanism_coder.py`
|
|
- Create: `data/mechanism_codes.json` (or table: `right_wing_mechanisms`)
|
|
|
|
**Approach:**
|
|
- Query the 212 motion IDs with title, year, and category from `right_wing_motions` + `extremity_scores`.
|
|
- Use `ai_provider.chat_completion_json_parallel` with a JSON schema:
|
|
```json
|
|
{"mechanism": "punitive|exclusionary|sovereignty-claiming|procedural-breaking|institutional-dismantling|none", "confidence": 1-5, "rationale": "string"}
|
|
```
|
|
- Batch size 10, ~22 batches. Skip already-coded motions for resumability.
|
|
- Store results in a new table `right_wing_mechanisms` (motion_id, mechanism, confidence, rationale) with `CREATE TABLE IF NOT EXISTS` + `INSERT OR REPLACE`.
|
|
|
|
**Execution note:** Start with a small validation batch (10 motions) and spot-check the taxonomy before running the full 212.
|
|
|
|
**Patterns to follow:**
|
|
- `analysis/right_wing/extremity_scorer.py` — batch loop, `chat_completion_json_parallel`, resumability pattern
|
|
- `analysis/right_wing/derive_categories.py` — two-phase validation (small sample first, then apply)
|
|
|
|
**Test scenarios:**
|
|
- Happy path: all 212 motions are coded with a valid mechanism and confidence 1–5.
|
|
- Edge case: LLM returns invalid JSON → retry or log error, don't crash.
|
|
- Edge case: script interrupted mid-batch → rerunning skips already-coded motions.
|
|
- Integration: mechanism table can be joined back to `right_wing_motions` for reporting.
|
|
|
|
**Verification:**
|
|
- `right_wing_mechanisms` table has 212 rows with no NULL mechanisms.
|
|
- Spot-check 10 random rows: rationale is coherent and mechanism assignment is defensible.
|
|
|
|
---
|
|
|
|
- U4. **Compile Findings Report**
|
|
|
|
**Goal:** Write a markdown narrative pulling together numbers, charts, and mechanism coding.
|
|
|
|
**Requirements:** R4, R5
|
|
|
|
**Dependencies:** U2, U3
|
|
|
|
**Files:**
|
|
- Create: `reports/right_wing_migration_antidemocratic/findings_report.md`
|
|
|
|
**Approach:**
|
|
- Structure the report around the 5 analytical sections from Direction 3:
|
|
1. Overlap quantification (migration's 44.8% share of high extremity)
|
|
2. Party strategy (PVV/Wilders dominance, JA21 volume-vs-intensity tradeoff)
|
|
3. Framing shift (2018–2020 direct exclusion → 2023–2025 institutional dismantling)
|
|
4. Cross-category adjacency (55 migration-adjacent motions, veiligheid/justitie as primary spillover)
|
|
5. Sentiment divergence (migration as the only negative-sentiment category, −0.717 at high extremity)
|
|
- Embed the 5 charts using relative paths.
|
|
- Include a mechanism-coding summary table (distribution of the 6 mechanisms across categories and parties).
|
|
- Conclude with the bottom-line hypothesis confirmation.
|
|
|
|
**Patterns to follow:**
|
|
- `reports/drift/report.md` — existing report format in the repo
|
|
|
|
**Test scenarios:**
|
|
- Happy path: report renders correctly in a markdown viewer with all images loading.
|
|
- Edge case: report references charts that don't exist → verify all image paths are valid.
|
|
|
|
**Verification:**
|
|
- `reports/right_wing_migration_antidemocratic/findings_report.md` exists and is a complete, readable narrative.
|
|
- All chart references resolve to existing PNG files.
|
|
|
|
---
|
|
|
|
## System-Wide Impact
|
|
|
|
- **Interaction graph:** None — this is a research artifact pipeline. No callbacks, middleware, or UI changes.
|
|
- **Error propagation:** LLM batch failures in U3 should log errors per-batch, not crash the script.
|
|
- **State lifecycle risks:** The `right_wing_mechanisms` table is idempotent (INSERT OR REPLACE). Rerunning is safe.
|
|
- **API surface parity:** No public API changes.
|
|
- **Integration coverage:** U3's integration with the LLM provider is the only cross-layer concern; verify the batch loop handles rate limits gracefully.
|
|
- **Unchanged invariants:** Database schema, existing right-wing analysis tables, Streamlit UI, and agent_tools surface remain untouched.
|
|
|
|
---
|
|
|
|
## Risks & Dependencies
|
|
|
|
| Risk | Mitigation |
|
|
|------|------------|
|
|
| LLM API costs for 212 motions exceed budget | Validate on 10-motion sample first; abort if cost-per-motion is unexpectedly high. |
|
|
| Mechanism taxonomy doesn't map cleanly to the data | Start with 6 classes; if LLM struggles, collapse to 4 or add an "other" catch-all. |
|
|
| Charts look unprofessional or cluttered | Iterate on 1–2 charts first, get feedback, then apply style to all 5. |
|
|
| Report becomes too long or unfocused | Cap at ~2,000 words; use summary tables and embedded charts to carry the narrative. |
|
|
|
|
---
|
|
|
|
## Documentation / Operational Notes
|
|
|
|
- The report is intended for internal research consumption and potential external sharing. No operational rollout needed.
|
|
- If the mechanism coding validates the anti-democratic concept, it becomes the evidence base for a future Direction 2 plan (full anti-democratic scoring pipeline).
|
|
|
|
---
|
|
|
|
## Sources & References
|
|
|
|
- Origin analysis: `analysis/right_wing/direction3_migration_antidemocratic.py`
|
|
- Related plan: `docs/plans/2026-05-05-001-feat-right-wing-motion-analysis-plan.md`
|
|
- Chart patterns: `scripts/motion_drift.py`, `explorer.py`
|
|
- LLM batch patterns: `analysis/right_wing/extremity_scorer.py`, `analysis/right_wing/derive_categories.py`
|
|
|