Commit Graph
6 Commits
Author SHA1 Message Date
sgeboers fbf92c82cf feat(right-wing): dual-scoring extremity/sentiment + derived categories
Extremity Scorer (U4 enhanced):
- Now scores BOTH original motion text AND layman explanation separately
- Schema: text_score, text_explanation, layman_score, layman_explanation
- Text scores: 1→7, 2→33, 3→5, 4→5 (mild-to-moderate)
- Layman scores: 1→12, 2→20, 3→17, 4→1 (slightly milder)

Sentiment Analysis (U5 enhanced):
- Now scores BOTH original motion text AND layman explanation separately
- Schema: text_score, text_explanation, layman_score, layman_explanation
- Text sentiment avg: 0.294 (slightly positive)
- Layman sentiment avg: 0.416 (more positive - summaries tone down hostility)

Category Derivation (new):
- Two-phase LLM approach: derive taxonomy from sample, then apply to all
- Discovered 7 categories from 30-motion sample:
  veiligheid/justitie, corona/pandemie, economie/belasting, klimaat/milieu,
  defensie/buitenland, asiel/vreemdelingen, overig
- Applied to 50 motions with distribution shown in DB
- Adds category + category_explanation columns to right_wing_motions
2026-05-05 21:40:58 +02:00
sgeboers f94edc3d04 feat(right-wing): sentiment analysis pipeline for right-wing motions
Implements U5: sentiment_analysis.py uses LLM batch calls (fallback when no
local Dutch sentiment model is available) to score motion sentiment on [-1, 1]
scale.

Design:
- Prompt asks for sentiment from -1 (hostile/aggressive) to 1 (constructive)
- JSON schema enforces numeric score + Dutch explanation
- Batch size 10, max_workers 5 for parallel API calls
- Stores results in  table
- Updates  with avg_sentiment, sentiment_std,
  pct_strongly_negative per year

Sample validation (50 motions): good variance across [-0.9, 1.0] range.
2026-05-05 21:25:42 +02:00
sgeboers d2310edfc4 feat(right-wing): LLM-based policy extremity scoring
Implements U4: extremity_scorer.py uses ai_provider.chat_completion_json_parallel
with a JSON schema enforcing integer 1-5 + Dutch explanation.

Design:
- Batch size 10, max_workers 5 for parallel API calls
- Prompt asks for concrete policy + radicalism score in Dutch
- Stores results in  table (motion_id, score, explanation, error)
- Updates  with yearly averages
- Default sample=50 for validation; --sample -1 scores all motions

Sample validation (50 motions): scores distributed 1→2, 2→34, 3→7, 4→7,
yearly averages ~2.0-2.5 (mild-to-moderate radicalism).
2026-05-05 21:23:08 +02:00
sgeboers 1bc83c4384 feat(right-wing): temporal aggregation of right-wing motion trends
Implements U3: temporal_analysis.py computes yearly_summary from the
right_wing_motions table (U2 output).

Metrics per year:
- total_right_wing, pct_of_total, total_motions
- avg_right_support, avg_left_opposition, centrist_support
- avg_right_keyword_matches, extremity_index (U4 placeholder)
- yoy_right_wing_delta, yoy_pct_delta

Key finding: right-wing motions grew from ~4% (2018) to ~12% (2024-2025)
of all motions, with rising centrist support over time.
2026-05-05 21:20:12 +02:00
sgeboers d3dfb0ce2f feat(right-wing): hybrid motion classifier using keywords + votes
Implements U2: classify_motions.py loads keywords from U1 and classifies
motions as right-wing when:
- right_support >= 60% (CANONICAL_RIGHT parties voting 'voor')
- left_opposition >= 40% (CANONICAL_LEFT parties voting 'tegen')
- AND at least 1 right-wing keyword match in title/body_text

Outputs DuckDB table  with:
- motion_id, year, title, right_support, left_opposition, centrist_support
- right_keyword_matches, left_keyword_matches, classified flag

Classified 2986 of 28331 motions (10.5%) as right-wing.
2026-05-05 21:18:38 +02:00
sgeboers c6f8540671 feat(right-wing): derive right-wing keywords via differential TF-IDF
Implements U1: derive_keywords.py uses party voting patterns to classify
motions as right-wing vs left-wing, then computes differential TF-IDF on
cleaned motion titles to surface policy terms distinctive to right-wing
motions.

Key design choices:
- Vote threshold: 60% of parties in group must vote 'voor'
- Text cleaning strips motion prefixes aggressively (handles multi-word
  surnames, plural 'leden', t.v.v. parentheticals)
- Expanded Dutch stopword list filters procedural and generic noise
- Results written to analysis/right_wing/right_wing_keywords.json

Produces ~50 filtered terms including: asielzoekers, defensie, kernenergie,
boeren, vreemdelingenbeleid, stikstof, asielstop, strafrecht.
2026-05-05 21:14:11 +02:00