motief/docs/plans/2026-05-08-001-feat-visualize-migration-antidemocratic-plan.md

title	type	status	date
Visualize and Report Migration-Anti-Democratic Overlap Findings	feat	active	2026-05-08

Visualize and Report Migration-Anti-Democratic Overlap Findings

Summary

Turn the Direction 3 analysis (migration as the dominant vehicle for anti-democratic rhetoric in right-wing motions) into a publishable artifact: committed script, visualizations, mechanism-coded extreme motions, and a findings report.

---

Problem Frame

The Direction 3 analysis script analysis/right_wing/direction3_migration_antidemocratic.py has been written and run successfully against all 2,986 scored right-wing motions. It uncovered strong evidence that migration is the primary carrier of anti-democratic extremity (45% of ≥4.0-scored motions are migration-related). However, the findings currently exist only as console output. They need to be preserved, visualized, deepened with mechanism coding, and compiled into a shareable report.

---

Requirements

• R1. The analysis script is committed to the feature branch.
• R2. Five focused charts visualize the key findings for immediate readability.
• R3. The 212 motions scoring ≥3.5 extremity are mechanism-coded via lightweight LLM batch.
• R4. A markdown report compiles numbers, charts, and mechanism coding into a narrative.
• R5. All artifacts are saved under reports/right_wing_migration_antidemocratic/.

---

Scope Boundaries

• No new LLM pipeline infrastructure (reuses existing ai_provider.chat_completion_json_parallel pattern from extremity_scorer.py).
• No changes to the database schema.
• No Streamlit UI integration (this is a research artifact, not a UI feature).
• No anti-democratic scoring pipeline for the full 2,986 motions (that would be Direction 2, deferred).

Deferred to Follow-Up Work

• Full anti-democratic scoring pipeline (Direction 2): separate plan if mechanism coding validates the concept.
• Streamlit tab to expose right-wing analysis tables: separate UI work.

---

Context & Research

Relevant Code and Patterns

• analysis/right_wing/direction3_migration_antidemocratic.py — the analysis script to commit
• analysis/right_wing/extremity_scorer.py — LLM batch pattern with chat_completion_json_parallel
• analysis/right_wing/derive_categories.py — two-phase LLM approach (derive taxonomy, then apply)
• scripts/motion_drift.py — chart generation pattern using matplotlib, saved to reports/
• analysis/config.py — CANONICAL_RIGHT, CANONICAL_LEFT, PARTY_COLOURS

Institutional Learnings

• docs/solutions/best-practices/svd-labels-voting-patterns-not-semantics.md — SVD labels should reflect voting patterns, not semantic content. Same discipline applies here: mechanism coding should reflect procedural/institutional characteristics, not just topic keywords.

---

Key Technical Decisions

• Matplotlib for charts: The repo already uses matplotlib in scripts/motion_drift.py and explorer.py. No new charting library.
• Reuse existing LLM batch infrastructure: The mechanism coder will call ai_provider.chat_completion_json_parallel with a JSON schema, same pattern as extremity/sentiment scoring. Batch size 10, ~22 batches for 212 motions.
• Mechanism taxonomy (6 classes): punitive, exclusionary, sovereignty-claiming, procedural-breaking, institutional-dismantling, none. Chosen to capture the observed framing evolution from "keep them out" to "dismantle the system that lets them in."
• Reports directory: reports/right_wing_migration_antidemocratic/ mirrors the reports/drift/ pattern.

---

Open Questions

Resolved During Planning

• Q: Which visualization library? Matplotlib — already used in the repo.
• Q: How many mechanisms to code? 6-class taxonomy, validated against the 5.0-scored motion sample.

Deferred to Implementation

• Q: Exact chart styling and color palette — depends on what looks good when rendered; iterate visually.
• Q: Whether to include migration-adjacent motions in the mechanism coding — start with pure migration (404), expand to adjacent (55) if time permits.

---

Implementation Units

• U1. Commit Direction 3 Analysis Script

Goal: Save the working analysis script to the feature branch.

Requirements: R1

Dependencies: None

Files:

• Create: analysis/right_wing/direction3_migration_antidemocratic.py

Approach:

• The script already exists and produces correct output. Stage and commit it with a conventional message.

Test scenarios:

• Test expectation: none — this is a standalone analysis script with no testable behavior changes.

Verification:

• Script is committed to feat/right-wing-motion-analysis and runs without errors.

---

• U2. Generate Five Key Visualizations

Goal: Produce charts that make the Direction 3 findings immediately readable.

Requirements: R2, R5

Dependencies: U1

Files:

• Create: analysis/right_wing/visualize_direction3.py
• Output: reports/right_wing_migration_antidemocratic/*.png

Approach:

• Write a script that queries the scored data and generates 5 charts:
  1. Stacked bar: Category breakdown of ≥4.0 extremity motions (migration's 44.8% dominance)
  2. Line chart: Migration motion volume + avg extremity by year (2018–2026)
  3. Horizontal bar: Party avg extremity on migration (Wilders 3.79, Eerdmans 2.69, etc.)
  4. Grouped bar: Migration sentiment by extremity bucket (the −0.717 drop)
  5. Treemap or stacked bar: Migration + migration-adjacent by category (15.4% total footprint)
• Follow the matplotlib patterns from scripts/motion_drift.py and explorer.py.
• Save charts to reports/right_wing_migration_antidemocratic/.

Patterns to follow:

• scripts/motion_drift.py — chart saving with plt.savefig(), DPI settings
• explorer.py — color palette using analysis/config.PARTY_COLOURS

Test scenarios:

• Happy path: script runs and produces 5 PNG files in the reports directory.
• Edge case: empty result set for a chart component should render gracefully (e.g., skip treemap segment with zero count).

Verification:

• 5 PNG files exist in reports/right_wing_migration_antidemocratic/ and render legible charts.

---

• U3. Mechanism-Code the 212 Extreme Motions

Goal: Classify the anti-democratic mechanism for each motion scoring ≥3.5 extremity.

Requirements: R3

Dependencies: U1

Files:

• Create: analysis/right_wing/mechanism_coder.py
• Create: data/mechanism_codes.json (or table: right_wing_mechanisms)

Approach:

• Query the 212 motion IDs with title, year, and category from right_wing_motions + extremity_scores.
• Use ai_provider.chat_completion_json_parallel with a JSON schema:

  {"mechanism": "punitive|exclusionary|sovereignty-claiming|procedural-breaking|institutional-dismantling|none", "confidence": 1-5, "rationale": "string"}
  

• Batch size 10, ~22 batches. Skip already-coded motions for resumability.
• Store results in a new table right_wing_mechanisms (motion_id, mechanism, confidence, rationale) with CREATE TABLE IF NOT EXISTS + INSERT OR REPLACE.

Execution note: Start with a small validation batch (10 motions) and spot-check the taxonomy before running the full 212.

Patterns to follow:

• analysis/right_wing/extremity_scorer.py — batch loop, chat_completion_json_parallel, resumability pattern
• analysis/right_wing/derive_categories.py — two-phase validation (small sample first, then apply)

Test scenarios:

• Happy path: all 212 motions are coded with a valid mechanism and confidence 1–5.
• Edge case: LLM returns invalid JSON → retry or log error, don't crash.
• Edge case: script interrupted mid-batch → rerunning skips already-coded motions.
• Integration: mechanism table can be joined back to right_wing_motions for reporting.

Verification:

• right_wing_mechanisms table has 212 rows with no NULL mechanisms.
• Spot-check 10 random rows: rationale is coherent and mechanism assignment is defensible.

---

• U4. Compile Findings Report

Goal: Write a markdown narrative pulling together numbers, charts, and mechanism coding.

Requirements: R4, R5

Dependencies: U2, U3

Files:

• Create: reports/right_wing_migration_antidemocratic/findings_report.md

Approach:

• Structure the report around the 5 analytical sections from Direction 3:
  1. Overlap quantification (migration's 44.8% share of high extremity)
  2. Party strategy (PVV/Wilders dominance, JA21 volume-vs-intensity tradeoff)
  3. Framing shift (2018–2020 direct exclusion → 2023–2025 institutional dismantling)
  4. Cross-category adjacency (55 migration-adjacent motions, veiligheid/justitie as primary spillover)
  5. Sentiment divergence (migration as the only negative-sentiment category, −0.717 at high extremity)
• Embed the 5 charts using relative paths.
• Include a mechanism-coding summary table (distribution of the 6 mechanisms across categories and parties).
• Conclude with the bottom-line hypothesis confirmation.

Patterns to follow:

• reports/drift/report.md — existing report format in the repo

Test scenarios:

• Happy path: report renders correctly in a markdown viewer with all images loading.
• Edge case: report references charts that don't exist → verify all image paths are valid.

Verification:

• reports/right_wing_migration_antidemocratic/findings_report.md exists and is a complete, readable narrative.
• All chart references resolve to existing PNG files.

---

System-Wide Impact

• Interaction graph: None — this is a research artifact pipeline. No callbacks, middleware, or UI changes.
• Error propagation: LLM batch failures in U3 should log errors per-batch, not crash the script.
• State lifecycle risks: The right_wing_mechanisms table is idempotent (INSERT OR REPLACE). Rerunning is safe.
• API surface parity: No public API changes.
• Integration coverage: U3's integration with the LLM provider is the only cross-layer concern; verify the batch loop handles rate limits gracefully.
• Unchanged invariants: Database schema, existing right-wing analysis tables, Streamlit UI, and agent_tools surface remain untouched.

---

Risks & Dependencies

Risk	Mitigation
LLM API costs for 212 motions exceed budget	Validate on 10-motion sample first; abort if cost-per-motion is unexpectedly high.
Mechanism taxonomy doesn't map cleanly to the data	Start with 6 classes; if LLM struggles, collapse to 4 or add an "other" catch-all.
Charts look unprofessional or cluttered	Iterate on 1–2 charts first, get feedback, then apply style to all 5.
Report becomes too long or unfocused	Cap at ~2,000 words; use summary tables and embedded charts to carry the narrative.

---

Documentation / Operational Notes

• The report is intended for internal research consumption and potential external sharing. No operational rollout needed.
• If the mechanism coding validates the anti-democratic concept, it becomes the evidence base for a future Direction 2 plan (full anti-democratic scoring pipeline).

---

Sources & References

• Origin analysis: analysis/right_wing/direction3_migration_antidemocratic.py
• Related plan: docs/plans/2026-05-05-001-feat-right-wing-motion-analysis-plan.md
• Chart patterns: scripts/motion_drift.py, explorer.py
• LLM batch patterns: analysis/right_wing/extremity_scorer.py, analysis/right_wing/derive_categories.py