21 Commits
Author SHA1 Message Date
sgeboers 3192a1a2bf fix(overton): correct 3 critical data errors and 3 high-severity caveats
CRIT-1: Stylistic extremity direction reversed — both dimensions declined
  (stijl 1.875→1.744, not increased +0.097). Holistic moderation, not divergence.
CRIT-2: Masking rate corrected from 36.1% to 9.7% (S≤2, M≥4 on full dataset).
  Original 36.8% was from 117-motion manual audit, not extrapolatable.
CRIT-3: Material impact values harmonized to motion-level means (2.79→2.45).
  Old values (2.78→2.43) used undisclosed mean-of-yearly-means aggregation.
HIGH-1: Migration domain provenance caveat — category column is NULL,
  analysis relies on title keyword matching (approximate boundaries).
HIGH-2: 2026-Q2 bounce caveat — n=44, bimodal distribution, sensitive to
  composition (many consensus defense motions among CS=1.0 items).
HIGH-3: Non-right-wing control group corrected — CS rose 58%→62% (+3.5pp),
  not 'flat at 49%'. Surge was disproportionate for right-wing content.

Also: fixed 6-party centrist definition (line 30) to 4-party,
removed 'did not shift rightward' phrasing, added Phase 4 synthesis
reminder to build_all_reports.py.
2026-06-14 22:45:12 +02:00
sgeboers 5706e86777 feat(overton): coherent narrative architecture — Quarto article, Explorer Overton tab, report cleanup
- U1: Remove stale findings_report.md and blog_post.html, add cross-reference
  headers to all 13 appendix reports, switch HTML report to canonical 4-party
  centrist definition
- U2: Create Quarto narrative spine (overton_window.qmd) with 9 sections and
  6 interactive Plotly charts. Includes 'About Stemwijzer' platform section.
- U3: Add Overton tab to Explorer (centrist support trend, right-wing motion
  browser, explore-further links). Add Overton context expander to Kompas tab
  and 2024 breakpoint annotation to Trajectories tab.
- U4: Create build_all_reports.py master regeneration script (3-phase,
  dependency-ordered, --skip-llm support)
- U5: Update README with Research section, create reports/overton_window/README.md
  reading guide, update STATUS.md with broader platform framing

Plan: docs/plans/2026-06-06-001-overton-coherent-narrative-plan.md
282 tests pass.
2026-06-07 22:02:04 +02:00
sgeboers a3154f72df refactor: extract shared helpers to common.py, fix bugs, add TDD tests
- Created analysis/right_wing/common.py with all shared helpers:
  Constants: CANONICAL_CENTRIST, COALITION, BREAK_YEAR, etc.
  Functions: _conn, cohens_d, build_party_name_map, parse_lead_submitter,
  motion_passed, quarter_sort_key, find_inflection_point

- Fixed bugs:
  1. ai_provider.py: requests.Timeout now caught alongside ConnectionError
  2. voting_margin.py: Removed walrus operator misuse, fixed Mann-Whitney test

- Updated 13 consuming files to import from common.py

- Added 35 TDD tests in tests/right_wing/test_common.py

- 282 tests pass (was 247)
2026-05-31 23:41:29 +02:00
sgeboers 364c312076 fix(right-wing): update DB with latest motions, fix DROP TABLE bug, score all missing 2D
- Fetched 276 new motions from Tweede Kamer API (2026-04-23 to 2026-05-31)
- Fixed classify_motions.py: DROP TABLE → CREATE TABLE IF NOT EXISTS
- Restored derived columns (centrist_support_strict, category, etc.) via migration
- Scored 180 missing motions in extremity_scores_2d (now 3,049 total, 0 missing)
- Re-ran temporal trajectory with updated data (inflection: 2024-Q2)
2026-05-31 22:21:35 +02:00
sgeboers 2d5b28fe1b feat(overton): coalition coding fix + regenerate breakpoint analysis 2026-05-31 20:11:03 +02:00
sgeboers d34d43a888 feat(overton): improvements and extensions — party differentiation, voting margin, SVD viz, mechanism validation, predictive model
U1: JA21 drives moderation effect (+0.203 CS shift, only party with volume+support gains)
U2: Coalition coding split at July 2024 — opposition effect confirmed (d=0.85 vs 0.87)
U3: Voting margin (ρ=0.812 with centrist support) is far superior to pass rate
U4: SVD trajectory confirms spatial divergence — centrists moved left (Δx=-0.30), right stationary
U5: Mechanism classification Cohen's κ=0.41 (moderate) — taxonomy needs revision
U6: Predictive model AUC-ROC=0.81 — submitter party and category are strongest predictors
2026-05-31 19:41:22 +02:00
sgeboers 7df961ba83 feat(overton): address 7 critical gaps in Overton window analysis
U1: Temporal trajectory — quarterly granularity reveals immediate
electoral jump at 2024-Q1 (+0.180), peak at 2024-Q4 (0.648), reversion
to 0.334 by 2026-Q1.

U2: 2D extremity temporal — single-score masks divergence. Material
impact decreased (-0.146) while stylistic increased (+0.097).
Wilcoxon p=0.002 confirms systematic divergence.

U3: Systematic mechanism classification — 150 motions. Consensus
framing confirmed (24% high-CS vs 8% low-CS, p=0.014). Post-2024
high-CS dominated by procedural (32%), consensus (24%), targeted
restriction (17%).

U4: Causal timing — shift is electorally driven (after Nov 2023 PVV
election, before Jul 2024 Schoof cabinet). Rules out coalition
dynamics, gradual learning, European contagion.

U5: Left-wing response — barely changed (21.3%→20.2%, -1.1pp).
Centrist shift (d=+1.89) is 18.3x larger than left hardening
(d=-0.75). Volt is only left party that softened (+12.9pp).

U6: Success correlation — significant trend (p<0.001) but success
premium only +3.2%, ceiling effect at 96%+ limits practical meaning.

U7: Synthesis update — integrated all findings, updated verdict
to note electoral-cycle effect and 2026-Q1 reversion.
2026-05-26 23:33:13 +02:00
sgeboers 80c68c0112 fix(right-wing): match store_scores column names and value order to DB schema 2026-05-25 00:01:31 +02:00
sgeboers bf37f84a8b feat(extremity): two-dimensional rescoring with subagent pipeline
- Project-local skill .opencode/skills/score-extremity/ for subagent dispatch
- Orchestrator extremity_rescore_2d.py with load_skill/sample/format/validate/store
- 16 TDD tests covering all orchestrator functions
- 117 motions scored by deepseek v4 flash subagents (12 parallel batches)
- Pearson r=0.45 between stylistic and material dimensions — separable
- Key finding: 36.8% of motions use restrained language for consequential policies
- 2d_extremity_correlation_report.md documents distribution, divergence patterns,
  and implications for the Overton acceptance-without-conversion narrative
2026-05-24 23:13:42 +02:00
sgeboers be007165b1 fix(right-wing): add resume support to extremity and sentiment scorers
Use CREATE TABLE IF NOT EXISTS and skip already-scored motions
to allow resuming interrupted batch runs.
2026-05-24 22:33:44 +02:00
sgeboers 711a410df3 chore: simplify Overton scripts, update README, add stemwijzer.db to gitignore
- Extracted EXTREMITY_BUCKET_ORDER constant and _extremity_bucket() helper (4 duplications removed)
- Merged two-pass query loop in compute_yearly_baseline into single pass
- Removed unused import (mticker), dead code (year_titles_map), 12 obvious comments
- Extracted _fmt_axis() helper in SVD drift script
- Updated README analysis/ description to include right-wing motion analysis
2026-05-24 22:19:21 +02:00
sgeboers 2a081ade25 fix(overton): strict centrist definition + left support analysis
- Reclassified centrist to {D66, CDA, CU, NSC} — removing VVD/BBB
  which are center-right coalition partners
- Added centrist_support_strict (0.251→0.507, d=+0.65), center_right_support,
  and left_support_mp columns via migration script
- Figure 1 now shows center-right (VVD/BBB) support as orange dashed line
- New Figure 3: bar chart of left-party support for right-wing motions
  (0.268→0.202, left opposition hardened)
- New report Section 6 covering left-wing support trends
- All analysis now uses strict centrist definition throughout
2026-05-09 00:45:38 +02:00
sgeboers e478235c84 fix(overton): correct SVD axis interpretation, drop pass rate, synthesis rewrite
- SVD axis 2 sign corrected: negative = nationalist (PVV -0.56, FVD -0.36), positive = kosmopolitisch (Volt +0.27). Centrists moved LEFT on both axes while right-wing moved further right culturally (+0.146 gap). 'Acceptance without conversion' named as unifying interpretation.
- U1: Figure 1 merged to single panel, pass rate removed, 5 centrist_support lines
- U2: Pass rate columns dropped from all breakpoint tables, PR narrative cut
- U3: Findings report rewritten: SVD section replaced, synthesis restructured into 3 tiers, extremity LLM bias qualified
- U4: Axis labels and sign convention added to svd_stability_report.md
- Added centrist_support_mp column (MP-weighted, correlates 0.998 with party-level)
2026-05-09 00:21:46 +02:00
sgeboers 76b499cdc0 feat(analysis): Overton window breakpoint analysis with opposition control and SVD drift
Quantify 2024 breakpoint in centrist support (d=+0.68 overall, d=+0.85 opposition-only),
domain decomposition, extremity-stratified pass rates, and manual LLM audit (75% agreement).
SVD center drift aborted due to axis instability (9/10 consecutive window pairs fail stability threshold).
2026-05-08 23:14:34 +02:00
sgeboers d170444bda feat(analysis): add migration anti-democratic overlap analysis 2026-05-08 22:56:38 +02:00
sgeboers fbf92c82cf feat(right-wing): dual-scoring extremity/sentiment + derived categories
Extremity Scorer (U4 enhanced):
- Now scores BOTH original motion text AND layman explanation separately
- Schema: text_score, text_explanation, layman_score, layman_explanation
- Text scores: 1→7, 2→33, 3→5, 4→5 (mild-to-moderate)
- Layman scores: 1→12, 2→20, 3→17, 4→1 (slightly milder)

Sentiment Analysis (U5 enhanced):
- Now scores BOTH original motion text AND layman explanation separately
- Schema: text_score, text_explanation, layman_score, layman_explanation
- Text sentiment avg: 0.294 (slightly positive)
- Layman sentiment avg: 0.416 (more positive - summaries tone down hostility)

Category Derivation (new):
- Two-phase LLM approach: derive taxonomy from sample, then apply to all
- Discovered 7 categories from 30-motion sample:
  veiligheid/justitie, corona/pandemie, economie/belasting, klimaat/milieu,
  defensie/buitenland, asiel/vreemdelingen, overig
- Applied to 50 motions with distribution shown in DB
- Adds category + category_explanation columns to right_wing_motions
2026-05-05 21:40:58 +02:00
sgeboers f94edc3d04 feat(right-wing): sentiment analysis pipeline for right-wing motions
Implements U5: sentiment_analysis.py uses LLM batch calls (fallback when no
local Dutch sentiment model is available) to score motion sentiment on [-1, 1]
scale.

Design:
- Prompt asks for sentiment from -1 (hostile/aggressive) to 1 (constructive)
- JSON schema enforces numeric score + Dutch explanation
- Batch size 10, max_workers 5 for parallel API calls
- Stores results in  table
- Updates  with avg_sentiment, sentiment_std,
  pct_strongly_negative per year

Sample validation (50 motions): good variance across [-0.9, 1.0] range.
2026-05-05 21:25:42 +02:00
sgeboers d2310edfc4 feat(right-wing): LLM-based policy extremity scoring
Implements U4: extremity_scorer.py uses ai_provider.chat_completion_json_parallel
with a JSON schema enforcing integer 1-5 + Dutch explanation.

Design:
- Batch size 10, max_workers 5 for parallel API calls
- Prompt asks for concrete policy + radicalism score in Dutch
- Stores results in  table (motion_id, score, explanation, error)
- Updates  with yearly averages
- Default sample=50 for validation; --sample -1 scores all motions

Sample validation (50 motions): scores distributed 1→2, 2→34, 3→7, 4→7,
yearly averages ~2.0-2.5 (mild-to-moderate radicalism).
2026-05-05 21:23:08 +02:00
sgeboers 1bc83c4384 feat(right-wing): temporal aggregation of right-wing motion trends
Implements U3: temporal_analysis.py computes yearly_summary from the
right_wing_motions table (U2 output).

Metrics per year:
- total_right_wing, pct_of_total, total_motions
- avg_right_support, avg_left_opposition, centrist_support
- avg_right_keyword_matches, extremity_index (U4 placeholder)
- yoy_right_wing_delta, yoy_pct_delta

Key finding: right-wing motions grew from ~4% (2018) to ~12% (2024-2025)
of all motions, with rising centrist support over time.
2026-05-05 21:20:12 +02:00
sgeboers d3dfb0ce2f feat(right-wing): hybrid motion classifier using keywords + votes
Implements U2: classify_motions.py loads keywords from U1 and classifies
motions as right-wing when:
- right_support >= 60% (CANONICAL_RIGHT parties voting 'voor')
- left_opposition >= 40% (CANONICAL_LEFT parties voting 'tegen')
- AND at least 1 right-wing keyword match in title/body_text

Outputs DuckDB table  with:
- motion_id, year, title, right_support, left_opposition, centrist_support
- right_keyword_matches, left_keyword_matches, classified flag

Classified 2986 of 28331 motions (10.5%) as right-wing.
2026-05-05 21:18:38 +02:00
sgeboers c6f8540671 feat(right-wing): derive right-wing keywords via differential TF-IDF
Implements U1: derive_keywords.py uses party voting patterns to classify
motions as right-wing vs left-wing, then computes differential TF-IDF on
cleaned motion titles to surface policy terms distinctive to right-wing
motions.

Key design choices:
- Vote threshold: 60% of parties in group must vote 'voor'
- Text cleaning strips motion prefixes aggressively (handles multi-word
  surnames, plural 'leden', t.v.v. parentheticals)
- Expanded Dutch stopword list filters procedural and generic noise
- Results written to analysis/right_wing/right_wing_keywords.json

Produces ~50 filtered terms including: asielzoekers, defensie, kernenergie,
boeren, vreemdelingenbeleid, stikstof, asielstop, strafrecht.
2026-05-05 21:14:11 +02:00