chore: simplify Overton scripts, update README, add stemwijzer.db to gitignore

- Extracted EXTREMITY_BUCKET_ORDER constant and _extremity_bucket() helper (4 duplications removed) - Merged two-pass query loop in compute_yearly_baseline into single pass - Removed unused import (mticker), dead code (year_titles_map), 12 obvious comments - Extracted _fmt_axis() helper in SVD drift script - Updated README analysis/ description to include right-wing motion analysis
2026-05-24 22:19:21 +02:00
parent 7b5f97e177
commit 711a410df3
7 changed files with 693 additions and 85 deletions
@@ -0,0 +1,212 @@
+---
+title: "Parallel API batching for AI motion summarization with adaptive throughput"
+date: "2026-05-05"
+category: workflow-issues
+module: summarizer
+problem_type: workflow_issue
+component: service_object
+severity: medium
+applies_when:
+  - "Backfilling large numbers of AI-generated summaries via an API"
+  - "Rate limits or slow throughput bottlenecking batch processing"
+  - "Need to process 10,000+ items with an LLM API"
+tags:
+  - parallelization
+  - batching
+  - openrouter
+  - mistral
+  - throughput
+  - cost-optimization
+---
+
+# Parallel API Batching for AI Motion Summarization
+
+## Context
+
+Generating layman-friendly explanations for 29,000+ parliamentary motions via an LLM API. Initial approach processed one motion per API call, yielding ~700 motions/hour with significant per-request overhead. At this rate, the full backfill would take ~40 hours and cost ~$15-20. The budget tracking was also inaccurate — estimated $5.00 cap but actual API spend was only ~$1.78 when the cap was hit.
+
+## Guidance
+
+### 1. Application-level batching (not native API batching)
+
+OpenAI's `/chat/completions` endpoint does not support multiple independent conversations in one request. Instead, pack 10-20 motions into a single prompt and request structured JSON output:
+
+```python
+# Build one prompt with N motions
+prompt = f"""Je krijgt {len(motions)} moties.
+Schrijf voor ELKE motie 2-3 zinnen uitleg.
+Geef antwoord als JSON: {{"motion_id": "uitleg", ...}}
+
+{motions_block}
+"""
+
+# Request JSON mode
+payload = {
+    "model": model,
+    "messages": messages,
+    "response_format": {"type": "json_object"}
+}
+```
+
+This eliminates 90%+ of per-request HTTP overhead and context-window overhead.
+
+### 2. Parallel API requests with ThreadPoolExecutor
+
+When the API supports concurrent requests, use `ThreadPoolExecutor` to saturate the connection:
+
+```python
+from concurrent.futures import ThreadPoolExecutor
+
+def chat_completion_json_parallel(
+    message_batches, model=None, max_workers=3
+):
+    def _fetch_one(messages):
+        return chat_completion_json(messages, model=model)
+
+    with ThreadPoolExecutor(max_workers=max_workers) as executor:
+        futures = [executor.submit(_fetch_one, batch) for batch in message_batches]
+        return [f.result() for f in futures]
+```
+
+With 3 parallel workers × 15 motions per batch = 45 motions per chunk. This yielded **2.0x speedup** (0.5 → 1.0 motions/sec). After fixing the parameter-passing bug (see "Critical gotcha" below), 5 workers × 20 motions = 100 motions per chunk achieved **3.4x speedup** (1.1 → 3.7 motions/sec).
+
+### 3. Model selection for language quality
+
+Tested three models on the same problematic motions:
+
+| Model | Quality | Speed | Issues |
+|-------|---------|-------|--------|
+| `google/gemma-4-26b-a4b-it` | Good | Slow (~25s/batch) | Occasional English words |
+| `mistralai/mistral-small-2603` | **Excellent** | Medium | None observed |
+| `mistralai/mistral-small-3.2-24b-instruct` | Good | Medium | One blank output |
+
+**Recommendation**: `mistralai/mistral-small-2603` for Dutch-language tasks. Test on a representative sample (20-50 items) before committing to a full backfill.
+
+### 4. Post-processing pipeline
+
+Always add a post-processing step to catch model failures:
+
+```python
+def _post_process_summary(self, text: str) -> str:
+    # 1. Remove lines that are mostly non-Latin (Chinese, Arabic, etc.)
+    # 2. Replace known English words with Dutch equivalents
+    # 3. Fix common typos (e.g., "lageinkomen" → "laag inkomen")
+    # 4. Reject if >10 common English words remain
+    # 5. Remove metadata fragments like "(45-102)"
+    # 6. Normalize whitespace and punctuation
+```
+
+This caught: Arabic script hallucinations, English words like "filthy", Dutch typos like "formuliernoten", and metadata leaking from titles.
+
+### 5. Adaptive backfill script
+
+Create a backfill script that monitors throughput and adjusts parameters dynamically:
+
+```python
+# Start conservative
+api_batch_size = 15
+parallel_batches = 3
+chunk_size = api_batch_size * parallel_batches  # 45
+
+# After each chunk, measure time
+if chunk_time < avg_chunk_time * 0.8:
+    api_batch_size += 1  # API is fast, increase batch
+elif chunk_time > avg_chunk_time * 1.5:
+    api_batch_size -= 1  # API is slow, decrease batch
+
+    # On repeated failures, back off
+    if failures_in_row >= 2:
+        api_batch_size -= 2
+        delay_between_chunks += 1.0
+```
+
+### Critical gotcha: pass parallelism config all the way through
+
+When adding `parallel_batches` to the orchestration layer, make sure it actually reaches the API call. A common bug is adding the parameter to the outer script but leaving hardcoded defaults in the inner method:
+
+```python
+# summarizer.py — WRONG (hardcoded defaults)
+def generate_layman_explanations_batch_parallel(
+    self, motions, model=None, parallel_batches=3, sub_batch_size=15
+):
+    ...
+
+def update_motion_summaries(
+    self, ..., api_batch_size=15, parallel_batches=3  # still hardcoded!
+):
+    ...
+    summaries = self.generate_layman_explanations_batch_parallel(
+        motions_for_api,
+        model=config.QWEN_MODEL,
+        parallel_batches=3,        # <-- BUG: ignores parameter
+        sub_batch_size=15,         # <-- BUG: ignores parameter
+    )
+```
+
+**Fix**: pass the parameters through and compute `sub_batch_size` dynamically:
+
+```python
+# summarizer.py — CORRECT
+def update_motion_summaries(
+    self, ..., api_batch_size=15, parallel_batches=3
+):
+    ...
+    summaries = self.generate_layman_explanations_batch_parallel(
+        motions_for_api,
+        model=config.QWEN_MODEL,
+        parallel_batches=parallel_batches,
+        sub_batch_size=api_batch_size // parallel_batches,
+    )
+```
+
+**Impact of this bug**: With `parallel_batches=3` hardcoded, increasing backfill.py to 5 workers had zero effect. After fixing: speed jumped from **1.1 → 3.7 motions/sec** (3.4x).
+
+### 6. Accurate cost tracking
+
+Update cost estimates when switching models. The old estimate assumed Qwen at $0.60/M tokens, but Mistral Small is much cheaper:
+
+```python
+# Mistral Small 2603 pricing (OpenRouter)
+TOKEN_PRICE_PER_MILLION = 0.07  # blended input+output
+TOKENS_PER_MOTION = 480         # ~400 input + ~80 output
+COST_PER_MOTION = (TOKENS_PER_MOTION / 1_000_000) * TOKEN_PRICE_PER_MILLION
+# ≈ $0.000034 per motion
+```
+
+## Why This Matters
+
+- **Speed**: From ~700/hour (single) to ~13,320/hour (parallel batch, 5 workers) — **19x faster**
+- **Cost**: From ~$15-20 estimated to ~$1-2 actual for 30,000 motions
+- **Quality**: Model testing on edge cases prevents garbage-in-garbage-out at scale
+- **Reliability**: Post-processing catches ~5% of outputs that would degrade UX
+
+## When to Apply
+
+- Processing 1,000+ items through any LLM API
+- API charges per-request (not per-token) or has high latency per request
+- Output quality is critical and model hallucinations are unacceptable
+- Running overnight/background backfills where throughput matters more than latency
+
+## Examples
+
+### Before (single-motion API calls)
+```python
+for motion in motions:
+    summary = ai.chat_completion(build_prompt(motion))
+    # 50 motions = 50 API calls, ~250s total
+```
+
+### After (parallel batching)
+```python
+# Split into 3 batches of 15
+batches = [motions[i:i+15] for i in range(0, 50, 15)]
+message_batches = [build_batch_prompt(batch) for batch in batches]
+results = chat_completion_json_parallel(message_batches, max_workers=3)
+# 50 motions = 3 API calls, ~25s total
+```
+
+## Related
+
+- `ai_provider.py` — `chat_completion_json_parallel()` implementation
+- `summarizer.py` — `MotionSummarizer` with batch and parallel methods
+- `backfill.py` — Adaptive backfill script with dynamic parameter tuning