Populated the right_wing_motions.category column (previously 100% NULL across 3,030 motions) via parallel subagent classification — 80 agents derived a 10-category taxonomy and classified all motions in minutes. Adds to the Overton QMD report: - Plotly dropdown filter on Chart 1 to toggle between policy categories - Chart 7: category delta bar chart (pre/post centrist support per domain) - Chart 8: quarterly domain trajectories for the 5 largest categories - Domain Decomposition narrative section Also fixes a Streamlit tab crash (m.text -> m.body_text) and adds TDD tests.main
parent
a5624f65bc
commit
19e8d5b8ba
@ -0,0 +1,167 @@ |
||||
--- |
||||
title: "feat: Category domain decomposition for Overton report" |
||||
type: feat |
||||
status: active |
||||
date: 2026-06-15 |
||||
--- |
||||
|
||||
# feat: Category domain decomposition for Overton report |
||||
|
||||
## Summary |
||||
|
||||
Add 3 new Plotly charts and a narrative section to `reports/overton_window/overton_window.qmd` that decompose the Overton shift by policy category (asiel/vreemdelingen, landbouw/natuur, energie/klimaat, etc.), making visible which domains drove the shift, which resisted it, and how each category's centrist support evolved over time. The yearly CS chart gains category filtering via Plotly dropdown menu. |
||||
|
||||
--- |
||||
|
||||
## Problem Frame |
||||
|
||||
The current Overton report treats all right-wing motions as a single aggregate. The synthesis mentions migration vs non-migration, but with 10 categories now populated across all 3,030 motions, we can show a richer picture: energie/klimaat had the second-largest CS surge (+0.107), landbouw/natuur actually declined (−0.063), and onderwijs/wetenschap barely moved (+0.053) despite being the highest-consensus domain. These stories are invisible in the current charts. |
||||
|
||||
--- |
||||
|
||||
## Requirements |
||||
|
||||
- R1. Add a horizontal bar chart showing pre/post centrist support delta per category, sorted by magnitude |
||||
- R2. Add category filtering to the yearly CS timeline (Chart 1) via Plotly dropdown menu, so users can view a single category or "All right-wing" |
||||
- R3. Add domain trajectories to the quarterly chart showing 4-5 key categories as separate lines |
||||
- R4. Add a "Domain Decomposition" narrative section with the category delta table and interpretive prose |
||||
- R5. All new charts must use the existing Plotly styling conventions (colors, template, height) |
||||
|
||||
--- |
||||
|
||||
## Scope Boundaries |
||||
|
||||
- No changes to the synthesis markdown or other analysis scripts |
||||
- No changes to the DB or data pipeline |
||||
- No interactive filtering beyond Plotly's built-in updatemenu (no Streamlit/JS) |
||||
|
||||
--- |
||||
|
||||
## Key Technical Decisions |
||||
|
||||
- **Plotly updatemenu for filtering:** Uses Plotly's built-in `updatemenus` with `buttons` to toggle between "All right-wing" (current line) and individual categories. No external JS or Dash needed. |
||||
- **Color scheme for categories:** Use a qualitative 10-color palette (Plotly `alphabet` or `set2`), not PARTY_COLOURS, to avoid confusion with party lines. |
||||
- **New chart cells inserted after existing Chart 1:** Category delta bar chart goes after the yearly CS chart (Chart 1) and its narrative. Domain trajectories go in the quarterly section (Chart 6). A new "Domain Decomposition" section links them. |
||||
|
||||
--- |
||||
|
||||
## Open Questions |
||||
|
||||
### Resolved During Planning |
||||
|
||||
- **Standalone HTML filtering:** Plotly updatemenu works in standalone HTML (confirmed via existing usage in the repo — the Overton report is a standalone HTML file with embedded Plotly). |
||||
|
||||
### Deferred to Implementation |
||||
|
||||
- **Whether to show all 10 categories or a curated subset:** The quarterly trajectories chart should show at most 5-6 lines to avoid visual clutter. The delta bar chart can show all 10. |
||||
- **Exact Plotly color assignment to each category:** Will match categories to a qualitative palette at implementation time. |
||||
|
||||
--- |
||||
|
||||
## Implementation Units |
||||
|
||||
- U1. **[Add category delta bar chart and dropdown-filtered yearly CS]** |
||||
|
||||
**Goal:** Two linked chart additions: (1) a category delta horizontal bar chart, (2) convert Chart 1's yearly CS line chart to support category filtering via Plotly updatemenu. |
||||
|
||||
**Requirements:** R1, R2, R5 |
||||
|
||||
**Dependencies:** None |
||||
|
||||
**Files:** |
||||
- Modify: `reports/overton_window/overton_window.qmd` |
||||
|
||||
**Approach:** |
||||
- Compute yearly CS per category via SQL: `SELECT year, category, AVG(centrist_support_strict) as cs, COUNT(*) as n FROM right_wing_motions WHERE classified = TRUE GROUP BY year, category` |
||||
- For the delta chart (new cell): compute pre/post CS per category, then use a horizontal bar (go.Bar with orientation='h'), sorted by delta descending. Color bars: green for positive delta, red for negative. |
||||
- For Chart 1 modification: extend the existing yearly SQL to GROUP BY year, category. In Python, pivot to get per-category columns. Create a go.Figure with all categories as traces plus an aggregate "All right-wing" trace. Add an updatemenu dropdown with buttons: "All right-wing" (restores all visible with only the aggregate line shown) and each category name (shows only that category's trace). |
||||
- Keep the existing pre/post mean lines and the break-year vertical line intact. The dropdown only controls which category trace is visible. |
||||
- Use a 10-color qualitative palette from plotly.express.colors.qualitative. |
||||
|
||||
**Patterns to follow:** |
||||
- Existing Chart 1 for SQL pattern, figure layout, annotation style |
||||
- Existing Chart 6 for quarterly trajectory styling |
||||
|
||||
**Test scenarios:** |
||||
- N/A — this is a Quarto document rendering change. Verify by rendering the QMD and checking that (a) the delta chart shows all 10 categories, (b) the dropdown in Chart 1 cycles through categories correctly, (c) pre/post mean lines remain visible in all views. |
||||
|
||||
**Verification:** |
||||
- `uv run quarto render reports/overton_window/overton_window.qmd` succeeds |
||||
- `overton_report.html` contains the new delta chart section and Chart 1 responds to dropdown interaction |
||||
|
||||
--- |
||||
|
||||
- U2. **[Add domain trajectories to quarterly chart]** |
||||
|
||||
**Goal:** Enhance the existing quarterly chart (Chart 6) by overlaying 4-5 key category lines alongside the aggregate. |
||||
|
||||
**Requirements:** R3, R5 |
||||
|
||||
**Dependencies:** U1 (the SQL for per-category yearly CS shares the same data approach; the quarterly chart needs its own SQL) |
||||
|
||||
**Files:** |
||||
- Modify: `reports/overton_window/overton_window.qmd` |
||||
|
||||
**Approach:** |
||||
- Select 5 categories to show as individual lines: asiel/vreemdelingen, energie/klimaat, buitenland/europa, landbouw/natuur, economie. These are the categories with largest deltas, largest volumes, or most story value. |
||||
- Compute quarterly CS per category: `SELECT EXTRACT(YEAR FROM m.date) AS y, CEIL(EXTRACT(MONTH FROM m.date) / 3.0) AS q, r.category, AVG(r.centrist_support_strict) AS cs, COUNT(*) AS n FROM right_wing_motions r JOIN motions m ON r.motion_id = m.id WHERE r.classified = TRUE AND m.date IS NOT NULL AND r.category IN (...) GROUP BY y, q, r.category` |
||||
- Add each category as a separate go.Scatter trace with distinct colors and dashed lines (to distinguish from the aggregate solid line). |
||||
- Keep the existing aggregate line in solid bold. Use the same inflection/peak annotations. |
||||
- Add a legend entry for each category. |
||||
|
||||
**Patterns to follow:** |
||||
- Existing Chart 6 for quarterly SQL, figure layout, inflection/peak annotations |
||||
|
||||
**Test scenarios:** |
||||
- N/A — verify by rendering and checking that 5 category traces appear with distinct colors and dashed styles alongside the aggregate. |
||||
|
||||
**Verification:** |
||||
- `uv run quarto render reports/overton_window/overton_window.qmd` succeeds |
||||
- The quarterly chart shows 5 category lines with a legend |
||||
|
||||
--- |
||||
|
||||
- U3. **[Add Domain Decomposition narrative section]** |
||||
|
||||
**Goal:** Add a heading-2 section "Domain Decomposition" between Indicator 1 and Indicator 2, with a markdown table of category deltas and 2-3 paragraphs of interpretive prose. |
||||
|
||||
**Requirements:** R4 |
||||
|
||||
**Dependencies:** U1 (the delta data is used; the table can be hardcoded from the pre-computed values or computed via inline SQL) |
||||
|
||||
**Files:** |
||||
- Modify: `reports/overton_window/overton_window.qmd` |
||||
|
||||
**Approach:** |
||||
- Insert the section after the existing Indicator 1 narrative text and before the "## Indicator 2: Spatial Divergence" heading. |
||||
- Include a markdown table with columns: Category, Pre-2024 CS, Post-2024 CS, Delta, Volume, Extremity gap (M−S). Use the known data from the DB queries. |
||||
- Write 3 paragraphs: |
||||
1. Overview: which categories drove the shift (migration, energy, foreign affairs) and which resisted (agriculture, healthcare, infrastructure). |
||||
2. The polarization paradox: landbouw/natuur and zorg/gezondheid as domains where centrist support declined despite content moderation. |
||||
3. The consensus domains: onderwijs/wetenschap and economie as stable high-CS categories where the window didn't need to shift. |
||||
- Reference the category delta chart (U1) and domain trajectories chart (U2) by their cell labels. |
||||
|
||||
**Patterns to follow:** |
||||
- Existing prose style in Indicator 1 and Indicator 2 sections |
||||
|
||||
**Test scenarios:** |
||||
- N/A — prose section. Review for factual accuracy against the DB data. |
||||
|
||||
**Verification:** |
||||
- Section renders in the HTML output with correct numbers and coherent prose |
||||
|
||||
--- |
||||
|
||||
## System-Wide Impact |
||||
|
||||
- **Interaction graph:** The QMD is the only file. No analysis scripts, DB schemas, or other artifacts are affected. |
||||
- **Unchanged invariants:** All existing charts, narrative sections, and data remain intact. New cells are added after existing ones, and the new section is inserted between existing sections. |
||||
|
||||
--- |
||||
|
||||
## Risks & Dependencies |
||||
|
||||
| Risk | Mitigation | |
||||
|------|------------| |
||||
| Plotly updatemenu may not fully render in Quarto HTML output | Test with a minimal prototype in an isolated QMD cell first; fallback is a faceted chart showing all categories as subplots | |
||||
| 5 category lines on the quarterly chart may be visually noisy | Use dashed lines for category traces and a solid bold line for aggregate. If too noisy, reduce to 3 categories | |
||||
@ -0,0 +1,98 @@ |
||||
--- |
||||
title: "Motion category classification via parallel subagent pipeline" |
||||
date: 2026-06-15 |
||||
category: best-practices |
||||
module: analysis/right_wing |
||||
problem_type: best_practice |
||||
component: development_workflow |
||||
severity: medium |
||||
applies_when: |
||||
- "classifying thousands of items into policy categories using LLMs" |
||||
- "sequential LLM batch pipelines time out or run too slowly" |
||||
- "a classification taxonomy can be derived from a sample rather than predefined" |
||||
- "items are independently classifiable with no cross-item state" |
||||
tags: |
||||
- motion-classification |
||||
- subagent-dispatch |
||||
- parallelism |
||||
- duckdb |
||||
- category-taxonomy |
||||
--- |
||||
|
||||
# Motion category classification via parallel subagent pipeline |
||||
|
||||
## Context |
||||
|
||||
The `right_wing_motions` table in `data/motions.db` had a `category` column that was 100% NULL across 3,030 classified motions — blocking downstream Overton analysis that splits centrist support by policy domain. The existing `derive_categories.py` script used OpenRouter's `chat_completion_json_parallel` to classify motions in sequential batches, but consistently timed out after 10 minutes without classifying anything at scale. A different approach was needed. |
||||
|
||||
## Guidance |
||||
|
||||
### 1. Derive taxonomy from a sample first |
||||
|
||||
Have a sub-agent read a random sample (e.g., 60 motions) and infer natural categories from the data. This produces categories grounded in the actual motion content rather than a preconceived list: |
||||
|
||||
- The sample ensures categories reflect real distribution (migration-heavy, stikstof-driven, etc.) |
||||
- The sub-agent returns a concise taxonomy with descriptions for each category |
||||
- Include a catch-all "overig" category for edge cases |
||||
|
||||
For this project the taxonomy yielded 10 categories: asiel/vreemdelingen, landbouw/natuur, veiligheid/justitie, zorg/gezondheid, economie, energie/klimaat, buitenland/europa, onderwijs/wetenschap, verkeer/infrastructuur, overig. |
||||
|
||||
### 2. Chunk data into independent batches |
||||
|
||||
Dump motions from the DB to JSON, then split into small chunks (~38 motions each) that fit comfortably within a single sub-agent's context window. Each chunk is a standalone JSON file containing motion_id, title, and body_text. |
||||
|
||||
### 3. Dispatch parallel classification sub-agents |
||||
|
||||
Spawn one sub-agent per chunk simultaneously (up to 80 in this case). Each receives: |
||||
- The chunk of motions to classify |
||||
- The taxonomy with category descriptions |
||||
- A strict JSON output format: `[{"motion_id": ..., "category": ..., "category_explanation": ...}]` |
||||
- An instruction to read both title and body_text before deciding on a category |
||||
|
||||
All 80 agents run in parallel, finishing in minutes rather than hours. |
||||
|
||||
### 4. Merge results and update the database |
||||
|
||||
Collect all result files. Validate each for correct structure (some may use non-standard key names). Then update the DB: |
||||
|
||||
```sql |
||||
UPDATE right_wing_motions |
||||
SET category = ?, category_explanation = ? |
||||
WHERE motion_id = ? |
||||
``` |
||||
|
||||
Verify by counting non-NULL rows. |
||||
|
||||
### 5. Integrate into downstream analysis |
||||
|
||||
Once the category column is populated, update analysis scripts and charts to use it. For the Overton QMD report this meant: |
||||
- A Plotly dropdown filter on the main centrist support chart to toggle between categories |
||||
- A category delta bar chart showing pre/post centrist support change per domain |
||||
- Quarterly domain trajectory charts for the 5 largest categories |
||||
|
||||
## Why This Matters |
||||
|
||||
- **Speed**: 80 parallel agents classified 3,030 motions in minutes vs. a sequential script that never finished at all |
||||
- **Simplicity**: No timeout handling, retry logic, or batch management needed — each agent is a fire-and-forget independent unit |
||||
- **Quality**: Classification is grounded in reasoning (reading title + full text), not keyword matching or vector similarity |
||||
- **Discoverability**: The derived taxonomy (10 categories) emerges naturally from the data rather than being imposed upfront |
||||
|
||||
## When to Apply |
||||
|
||||
- You have thousands of items needing per-item LLM processing |
||||
- Each item is independently classifiable |
||||
- The task fits in a sub-agent's context window when batched at ~30-50 items |
||||
- Parallel dispatch infrastructure is available (e.g., the `task` tool) |
||||
|
||||
## Examples |
||||
|
||||
The pipeline was applied to 3,030 Dutch right-wing motions. The taxonomy was derived from a 60-motion sample by a single sub-agent, then 80 parallel sub-agents classified ~38 motions each. Final distribution was: landbouw/natuur 487, economie 470, asiel/vreemdelingen 423, buitenland/europa 386, veiligheid/justitie 359, zorg/gezondheid 348, energie/klimaat 174, overig 159, verkeer/infrastructuur 138, onderwijs/wetenschap 86. |
||||
|
||||
Two chunks needed minor fixes (used `category_label` / `predicted_category` instead of `category`). A quick validation script caught these before the DB update. |
||||
|
||||
## Related |
||||
|
||||
- `docs/solutions/best-practices/large-scale-subagent-2d-extremity-scoring-2026-06-05.md` — parallel subagent pattern for numeric extremity scoring (same infrastructure, different task) |
||||
- `analysis/right_wing/derive_categories.py` — the original sequential script that timed out |
||||
- `docs/solutions/best-practices/domain-decomposition-overton-analysis.md` — why category-split analysis matters for Overton interpretation |
||||
- `docs/solutions/best-practices/overton-narrative-architecture-2026-06-06.md` — QMD report structure that consumed the categories |
||||
|
Before Width: | Height: | Size: 141 KiB After Width: | Height: | Size: 170 KiB |
|
Before Width: | Height: | Size: 140 KiB After Width: | Height: | Size: 152 KiB |
|
Before Width: | Height: | Size: 439 KiB After Width: | Height: | Size: 439 KiB |
|
Before Width: | Height: | Size: 225 KiB After Width: | Height: | Size: 278 KiB |
|
Before Width: | Height: | Size: 127 KiB After Width: | Height: | Size: 127 KiB |
|
Before Width: | Height: | Size: 320 KiB After Width: | Height: | Size: 374 KiB |
@ -0,0 +1,116 @@ |
||||
"""Validate category decomposition data for Overton report.""" |
||||
from __future__ import annotations |
||||
|
||||
import duckdb |
||||
import pytest |
||||
|
||||
DB_PATH = "data/motions.db" |
||||
|
||||
|
||||
@pytest.fixture(scope="module") |
||||
def con(): |
||||
c = duckdb.connect(DB_PATH) |
||||
yield c |
||||
c.close() |
||||
|
||||
|
||||
def test_category_distribution(con): |
||||
"""There are exactly 10 categories and all 3,030 motions are classified.""" |
||||
df = con.execute(""" |
||||
SELECT category, COUNT(*) as cnt |
||||
FROM right_wing_motions |
||||
WHERE classified = TRUE |
||||
GROUP BY category |
||||
ORDER BY cnt DESC |
||||
""").fetchdf() |
||||
assert len(df) == 10 |
||||
assert df["cnt"].sum() == 3030 |
||||
assert df[df["category"] == "overig"]["cnt"].values[0] >= 100 |
||||
|
||||
|
||||
def test_category_deltas(con): |
||||
"""Pre/post CS deltas (2024 split) — ALL categories gained, energie/klimaat leads.""" |
||||
df = con.execute(""" |
||||
SELECT |
||||
category, |
||||
AVG(CASE WHEN year < 2024 THEN centrist_support_strict END) as pre_cs, |
||||
AVG(CASE WHEN year >= 2024 THEN centrist_support_strict END) as post_cs, |
||||
COUNT(*) as n |
||||
FROM right_wing_motions |
||||
WHERE classified = TRUE AND year >= 2017 |
||||
GROUP BY category |
||||
""").fetchdf() |
||||
|
||||
assert len(df) == 10 |
||||
df = df.copy() |
||||
df["delta"] = df["post_cs"] - df["pre_cs"] |
||||
top = df.sort_values("delta", ascending=False) |
||||
assert top.iloc[0]["category"] == "energie/klimaat" |
||||
assert 0.35 < abs(top.iloc[0]["delta"]) < 0.45 |
||||
|
||||
bottom = df.sort_values("delta", ascending=True) |
||||
assert bottom.iloc[0]["category"] in ("veiligheid/justitie", "onderwijs/wetenschap") |
||||
|
||||
assert all(df["delta"] > 0) |
||||
|
||||
|
||||
def test_yearly_category_cs(con): |
||||
"""Yearly CS per category returns data for every year-category combo.""" |
||||
df = con.execute(""" |
||||
SELECT year, category, AVG(centrist_support_strict) as cs, COUNT(*) as n |
||||
FROM right_wing_motions |
||||
WHERE classified = TRUE AND year >= 2017 |
||||
GROUP BY year, category |
||||
ORDER BY year, category |
||||
""").fetchdf() |
||||
assert len(df) >= 50 |
||||
assert df["category"].nunique() == 10 |
||||
assert df["year"].nunique() >= 8 |
||||
|
||||
|
||||
def test_quarterly_category_data(con): |
||||
"""Quarterly CS per key categories returns expected shape.""" |
||||
key_cats = ["asiel/vreemdelingen", "energie/klimaat", "buitenland/europa", |
||||
"landbouw/natuur", "economie"] |
||||
df = con.execute(""" |
||||
SELECT |
||||
EXTRACT(YEAR FROM m.date) AS y, |
||||
CEIL(EXTRACT(MONTH FROM m.date) / 3.0) AS q, |
||||
r.category, |
||||
AVG(r.centrist_support_strict) AS cs, |
||||
COUNT(*) AS n |
||||
FROM right_wing_motions r |
||||
JOIN motions m ON r.motion_id = m.id |
||||
WHERE r.classified = TRUE AND m.date IS NOT NULL |
||||
AND r.category IN ({}) |
||||
GROUP BY y, q, r.category |
||||
ORDER BY y, q |
||||
""".format(",".join(f"'{c}'" for c in key_cats))).fetchdf() |
||||
|
||||
assert df["category"].nunique() <= 5 |
||||
for cat in key_cats: |
||||
assert cat in df["category"].values |
||||
|
||||
|
||||
def test_qmd_has_domain_decomposition_section(): |
||||
"""QMD should have a Domain Decomposition heading after implementation.""" |
||||
qmd = open("reports/overton_window/overton_window.qmd").read() |
||||
assert "## Domain Decomposition" in qmd |
||||
|
||||
|
||||
def test_qmd_has_category_delta_chart(): |
||||
"""QMD should have a category delta bar chart cell.""" |
||||
qmd = open("reports/overton_window/overton_window.qmd").read() |
||||
assert "chart-7-category-delta" in qmd |
||||
|
||||
|
||||
def test_qmd_has_category_filter_dropdown(): |
||||
"""Chart 1 should have updatemenu for category filtering.""" |
||||
qmd = open("reports/overton_window/overton_window.qmd").read() |
||||
assert "updatemenus" in qmd |
||||
|
||||
|
||||
def test_qmd_has_domain_trajectories(): |
||||
"""Quarterly chart should have category trajectories.""" |
||||
qmd = open("reports/overton_window/overton_window.qmd").read() |
||||
assert "chart-8-domain-trajectories" in qmd |
||||
Loading…
Reference in new issue