feat(overton): coherent narrative architecture — Quarto article, Explorer Overton tab, report cleanup
- U1: Remove stale findings_report.md and blog_post.html, add cross-reference headers to all 13 appendix reports, switch HTML report to canonical 4-party centrist definition - U2: Create Quarto narrative spine (overton_window.qmd) with 9 sections and 6 interactive Plotly charts. Includes 'About Stemwijzer' platform section. - U3: Add Overton tab to Explorer (centrist support trend, right-wing motion browser, explore-further links). Add Overton context expander to Kompas tab and 2024 breakpoint annotation to Trajectories tab. - U4: Create build_all_reports.py master regeneration script (3-phase, dependency-ordered, --skip-llm support) - U5: Update README with Research section, create reports/overton_window/README.md reading guide, update STATUS.md with broader platform framing Plan: docs/plans/2026-06-06-001-overton-coherent-narrative-plan.md 282 tests pass.
This commit is contained in:
+110
@@ -0,0 +1,110 @@
|
||||
---
|
||||
title: Large-scale subagent-based 2D extremity scoring
|
||||
date: 2026-06-05
|
||||
category: best-practices
|
||||
module: analysis/right_wing
|
||||
problem_type: best_practice
|
||||
component: development_workflow
|
||||
severity: medium
|
||||
applies_when:
|
||||
- "scaling LLM scoring from hundreds to tens of thousands of items"
|
||||
- "using subagent dispatch as a replacement for API-based batch scoring"
|
||||
- "parallel batch processing with stateful incremental storage"
|
||||
tags:
|
||||
- extremity-scoring
|
||||
- subagent-dispatch
|
||||
- parallelism
|
||||
- duckdb
|
||||
- llm-workflow
|
||||
---
|
||||
|
||||
# Large-scale subagent-based 2D extremity scoring
|
||||
|
||||
## Context
|
||||
|
||||
After scoring 117 right-wing motions with 2D extremity (stijl-extremiteit + materiele impact) using deepseek v4 flash subagents, we needed to scale to all 29,570 motions in the database. The existing OpenRouter-based batch pipeline (`chat_completion_json_parallel`) would be too expensive and slow at this scale. Subagent dispatch via the `task` tool was the alternative.
|
||||
|
||||
## Guidance
|
||||
|
||||
### 1. Batch file generation
|
||||
|
||||
Generate fixed-size batch files (20 motions each) containing filled prompt templates with all motion context upfront. This avoids repeated DB queries per subagent:
|
||||
|
||||
```python
|
||||
for i, chunk in enumerate(chunks):
|
||||
batch_content = ""
|
||||
for motion in chunk:
|
||||
batch_content += f"MOTION_ID: {motion['id']}\n{prompt_template.format(...)}\n\n"
|
||||
write(f"/tmp/all_batch_{i:04d}.txt", batch_content)
|
||||
```
|
||||
|
||||
Always write exact motion IDs in each batch file so results can be matched back without ambiguity.
|
||||
|
||||
### 2. Politically neutral prompt
|
||||
|
||||
When scoring motions across the full political spectrum (not just right-wing), adjust the material impact scale to be politically symmetric:
|
||||
|
||||
- Scale point 5 should describe "fundamentele herstructurering van rechten, instituties of economische systemen" — not only right-wing actions like "inperking van rechten"
|
||||
- Include examples from both left and right: high-impact left motions (nationalization, wealth taxes, climate mandates) and right motions (asylum cessation, EU exit) should both reach the top of the scale
|
||||
|
||||
The SKILL.md file is read at runtime via `load_skill()`, so prompt changes take effect immediately without code changes.
|
||||
|
||||
### 3. Subagent dispatch pattern
|
||||
|
||||
Dispatch subagents in parallel waves of 5-8, each handling 5 batch files (100 motions):
|
||||
|
||||
```
|
||||
For each wave of 5-8 subagents (in parallel):
|
||||
For each subagent (handling 5 batch files):
|
||||
task(score-extremity skill, "Score these motions: {batch_content}")
|
||||
Wait for all to complete
|
||||
Collect results from /tmp/all_result_*.json
|
||||
Validate and store to DB incrementally
|
||||
```
|
||||
|
||||
Key: store results to DB after each wave, not after all waves. /tmp files can be cleaned up by the system, and subagent timeouts can lose data.
|
||||
|
||||
### 4. Anti-scripting guard
|
||||
|
||||
Subagents sometimes write Python scripts to batch-score motions instead of scoring directly in their reasoning. Add explicit instructions:
|
||||
|
||||
```
|
||||
IMPORTANT: Do NOT write Python scripts to score these motions. Score them
|
||||
directly in your reasoning, returning the JSON array. Do not use code
|
||||
to automate this — your reasoning and judgment IS the scoring mechanism.
|
||||
```
|
||||
|
||||
### 5. Incremental storage
|
||||
|
||||
Use `INSERT OR REPLACE` for idempotent writes:
|
||||
|
||||
```sql
|
||||
INSERT OR REPLACE INTO extremity_scores_all
|
||||
(motion_id, stijl_extremiteit, stijl_toelichting, materiele_impact, materiele_toelichting)
|
||||
VALUES (?, ?, ?, ?, ?)
|
||||
```
|
||||
|
||||
This allows re-running waves without duplicate errors and makes the pipeline resumable.
|
||||
|
||||
### 6. Handling placeholder motions
|
||||
|
||||
Many motions in the database have only an outcome label ("Aangenomen." / "Verworpen.") with no text or layman explanation. These should be scored (1, 1) and the scoring subagent should detect and report this. Do not try to infer scores from metadata like controversy scores — this defeats the purpose of LLM-based scoring.
|
||||
|
||||
## Why This Matters
|
||||
|
||||
- **Cost**: Subagent-based scoring via deepseek v4 flash is ~$2-3 for 30K motions vs. $50-100+ via OpenRouter API at comparable scale
|
||||
- **Resumability**: Wave-by-wave DB storage means a timeout or crash loses at most one wave (~400-500 motions)
|
||||
- **Prompt agility**: SKILL.md changes propagate immediately to the next wave — no pipeline restart needed
|
||||
- **Independence**: Style and material impact dimensions maintain moderate correlation (r ≈ 0.43) even at scale, confirming they capture separable signals
|
||||
|
||||
## Examples
|
||||
|
||||
**Failed approach**: single monolithic subagent scoring all 30K motions. Times out, loses all progress.
|
||||
|
||||
**Working approach**: 1,184 batch files, ~80 waves of 5-8 subagents each, DB stored after each wave. 3-day pipeline, resumable, $3 total cost.
|
||||
|
||||
## Related
|
||||
|
||||
- `.opencode/skills/score-extremity/SKILL.md` — the scoring prompt and subagent workflow
|
||||
- `analysis/right_wing/extremity_score_all.py` — batch generation and orchestrator
|
||||
- `docs/solutions/best-practices/overton-extended-analysis-methodology-2026-05-26.md` — 2D scoring in Overton context
|
||||
@@ -0,0 +1,93 @@
|
||||
---
|
||||
title: Overton window analysis narrative architecture
|
||||
date: 2026-06-06
|
||||
category: best-practices
|
||||
module: analysis/right_wing
|
||||
problem_type: architecture_pattern
|
||||
component: development_workflow
|
||||
severity: medium
|
||||
applies_when:
|
||||
- "organizing multi-report analytical projects into a coherent narrative"
|
||||
- "connecting static reports to live dashboards"
|
||||
- "identifying gaps between parallel analytical tracks"
|
||||
tags:
|
||||
- overton-window
|
||||
- narrative-architecture
|
||||
- report-organization
|
||||
- dashboard-integration
|
||||
- quarto
|
||||
---
|
||||
|
||||
# Overton window analysis narrative architecture
|
||||
|
||||
## Context
|
||||
|
||||
The Overton window analysis produced 17 reports across `reports/overton_window/`, 3 live Streamlit Explorer dashboards, and a project-local scoring skill — but these pieces were built incrementally across sessions and never organized into a coherent narrative. The reports cross-reference each other inconsistently, overlap with dashboard data, and lack a clear reading order.
|
||||
|
||||
## Guidance
|
||||
|
||||
### 1. Three-tier narrative structure
|
||||
|
||||
Organize analytical outputs into three tiers, each with a different audience and purpose:
|
||||
|
||||
| Tier | Audience | Format | Content |
|
||||
|------|----------|--------|---------|
|
||||
| **Narrative spine** | Everyone | Quarto article (`.qmd`) | The coherent story: what happened, why, and what it means |
|
||||
| **Detailed appendices** | Researchers | Markdown reports in `reports/overton_window/` | Per-indicator deep dives with full methodology |
|
||||
| **Live exploration** | Power users | Streamlit Explorer tab | Interactive drill-down into the underlying data |
|
||||
|
||||
The narrative spine references appendices for detail. Appendices reference each other where analyses overlap. The live dashboard links back to the narrative via explanatory text.
|
||||
|
||||
### 2. Centrist definition must be consistent across all outputs
|
||||
|
||||
The strict 4-party definition (D66, CDA, CU, NSC) is the canonical one — it isolates the genuine center and produces cleaner signals. The 6-party definition (adding VVD, BBB) appeared in early iterations and survives in some reports. Every public-facing output must use the strict definition or explicitly note when the wide definition is used for comparison.
|
||||
|
||||
### 3. Live dashboards are part of the story
|
||||
|
||||
The Streamlit Explorer already shows the SVD compass (Tab A), party trajectories (Tab B), and component decomposition (Tab C) — all of which directly visualize Overton window dynamics. The gap is that:
|
||||
|
||||
- No tab explicitly labels itself as "Overton analysis"
|
||||
- No tab shows right-wing motion centrist support trends
|
||||
- No tab shows 2D extremity scoring results
|
||||
- The browser.py/search.py tabs exist but aren't wired
|
||||
|
||||
Adding a dedicated "Overton Window" tab or retrofitting the existing compass tab with an Overton context panel connects the static analysis to the live data surface.
|
||||
|
||||
### 4. Quarto bridges static reports and interactive dashboards
|
||||
|
||||
Static HTML (overton_report.html) is a dead-end artifact — it can't be updated without regeneration and can't be filtered or zoomed. Quarto `.qmd` files with embedded Plotly charts solve this:
|
||||
|
||||
- Interactive centrist support trend lines with hover tooltips
|
||||
- Filterable 2D extremity scatter plots
|
||||
- Linked views between SVD drift and centrist support
|
||||
- Self-contained HTML output with embedded data
|
||||
|
||||
The existing `plotly` dependency (6.6.0) works directly in Quarto's Jupyter engine.
|
||||
|
||||
### 5. Remove, don't accumulate
|
||||
|
||||
Not every report earned its place. Remove:
|
||||
- `findings_report.md` — fully superseded by synthesis
|
||||
- `blog_post.html` — replace with Quarto version
|
||||
- Duplicate analysis between breakpoint and synthesis — keep breakpoint as appendix only
|
||||
|
||||
### 6. Master build script for reproducibility
|
||||
|
||||
A single `analysis/right_wing/build_all_reports.py` that runs every analysis script in dependency order and verifies output existence. This guarantees that any future researcher can regenerate the entire Overton analysis from the same database state.
|
||||
|
||||
## Why This Matters
|
||||
|
||||
Without narrative architecture, a multi-session analytical project produces a fragmented artifact: individual reports are technically correct but nobody can follow the story from question to answer. The three-tier structure (narrative spine → appendices → live dashboard) maps to how different readers consume the work: skim the spine, drill into appendices for detail, explore the dashboard for their own questions.
|
||||
|
||||
## When to Apply
|
||||
|
||||
- Any analytical project that spans multiple sessions and produces more than 5 output files
|
||||
- When static reports overlap with live dashboards
|
||||
- When reports need to survive beyond the session that created them
|
||||
|
||||
## Related
|
||||
|
||||
- `reports/overton_window/overton_window_synthesis.md` — current master synthesis
|
||||
- `reports/overton_window/overton_report.html` — current static HTML deliverable
|
||||
- `.opencode/skills/score-extremity/SKILL.md` — 2D scoring methodology
|
||||
- `docs/solutions/best-practices/overton-window-shift-methodology-2026-05-24.md` — 7-step methodology
|
||||
Reference in New Issue
Block a user