chore(ledgers): record fusion+similarity run summary and JSON details
This commit is contained in:
@@ -0,0 +1,177 @@
|
||||
---
|
||||
date: 2026-03-23
|
||||
topic: "Motion Content Enrichment via SyncFeed"
|
||||
status: validated
|
||||
---
|
||||
|
||||
# Motion Content Enrichment via SyncFeed
|
||||
|
||||
## Problem Statement
|
||||
|
||||
All 25,521 motions in the DB have NULL `body_text` and NULL `layman_explanation`. Their
|
||||
`title`/`description` are outcome strings ("Aangenomen.", "Verworpen.") because the bulk
|
||||
downloader used `skip_details=True`. The text embedding pipeline uses
|
||||
`COALESCE(layman_explanation, description, title)`, so all embeddings are effectively
|
||||
embeddings of "Aangenomen." — zero semantic signal.
|
||||
|
||||
Goal: populate real motion titles (Zaak.Onderwerp) and motion body text
|
||||
(officielebekendmakingen.nl HTML) for all motions, then re-run embeddings for the complete
|
||||
dataset.
|
||||
|
||||
## Constraints
|
||||
|
||||
- Do NOT modify `app.py` or `scheduler.py`
|
||||
- DuckDB only; open/close per method
|
||||
- Use Python logging, no print() in library code
|
||||
- `motions.id` primary key is an INTEGER autoincrement; `motions.url` contains
|
||||
`https://www.tweedekamer.nl/kamerstukken/stemmingsuitslagen/{besluit-uuid}` — the UUID
|
||||
is the Besluit.Id in the Tweede Kamer data model
|
||||
- `database.py` CREATE TABLE for motions is missing `body_text` and `externe_identifier`
|
||||
columns even though INSERT statements reference them — schema must be fixed
|
||||
|
||||
## Approach
|
||||
|
||||
Use the **SyncFeed API** (`https://gegevensmagazijn.tweedekamer.nl/SyncFeed/2.0/Feed`) to
|
||||
bulk-walk 4 entity types and build a complete local join index. This replaces the
|
||||
per-motion OData chain (3 API calls × 25,521 = 76,000+ calls) with ~2,000–4,000 paginated
|
||||
feed pages across all entity types.
|
||||
|
||||
Alternatives considered:
|
||||
- **OData per-motion** (`_get_motion_details`): 76k+ calls, estimated 10+ hours. Rejected.
|
||||
- **OData bulk $expand**: Works for titles (~100 pages) but getting ExterneIdentifier
|
||||
still requires per-Zaak calls. Partially useful but incomplete. Rejected in favour of
|
||||
SyncFeed which handles everything in one pass.
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
SyncFeed walk (4 feeds)
|
||||
├─ category=Besluit → {besluit_id: [zaak_ids]}
|
||||
├─ category=Zaak → {zaak_id: {onderwerp, soort}}
|
||||
├─ category=Document → {document_id: [zaak_ids]}
|
||||
└─ category=DocumentVersie → {document_id: externe_identifier}
|
||||
↓
|
||||
In-memory join:
|
||||
besluit_id → zaak_id → onderwerp (title)
|
||||
besluit_id → zaak_id → document_id → ext_id (ExterneIdentifier)
|
||||
↓
|
||||
DB update pass: UPDATE motions SET title=?, externe_identifier=? WHERE url LIKE ?
|
||||
↓
|
||||
Parallel HTML fetch (thread pool, 20 workers):
|
||||
GET zoek.officielebekendmakingen.nl/{ext_id}.html → extract text → UPDATE motions.body_text
|
||||
↓
|
||||
Pipeline re-run:
|
||||
clear embeddings → text pipeline → fusion (all windows) → similarity cache (all windows)
|
||||
```
|
||||
|
||||
## Components
|
||||
|
||||
### `scripts/sync_motion_content.py` (new)
|
||||
|
||||
Orchestrates the full enrichment:
|
||||
|
||||
1. **SyncFeed walker** — generic paginated Atom/XML reader that follows `<link rel="next">`
|
||||
until exhausted, yielding parsed entity dicts per page. Respects 429/rate-limit via
|
||||
exponential backoff.
|
||||
|
||||
2. **Entity parsers** — one per entity type:
|
||||
- `parse_besluit(xml)` → `{id, zaak_refs: [uuid, ...], verwijderd}`
|
||||
- `parse_zaak(xml)` → `{id, onderwerp, soort, verwijderd}`
|
||||
- `parse_document(xml)` → `{id, zaak_refs: [uuid, ...], verwijderd}`
|
||||
- `parse_documentversie(xml)` → `{id, document_id, externe_identifier, extensie, verwijderd}`
|
||||
|
||||
3. **Join builder** — after all 4 feeds are walked:
|
||||
- `build_title_map(besluit_index, zaak_index)` → `{besluit_id: onderwerp}`
|
||||
- `build_ext_id_map(besluit_index, zaak_index, doc_index, docversie_index)`
|
||||
→ `{besluit_id: externe_identifier}`
|
||||
- For motions with multiple Zaak, prefer Soort="Motie"; fall back to first
|
||||
|
||||
4. **DB updater** — open DuckDB, bulk UPDATE motions using the join maps. Extract
|
||||
`besluit_id` from `url` column via string split.
|
||||
|
||||
5. **Body text fetcher** — thread pool (20 workers), fetch HTML from
|
||||
`zoek.officielebekendmakingen.nl/{ext_id}.html`, strip HTML tags with regex (reuse
|
||||
existing `_fetch_body_text` logic), UPDATE `motions.body_text`.
|
||||
|
||||
6. **Progress reporting** — log counts: motions updated with title, motions with
|
||||
ExterneIdentifier found, body text fetched, failures.
|
||||
|
||||
### `database.py` schema fix
|
||||
|
||||
Add missing columns to `CREATE TABLE motions` DDL:
|
||||
- `body_text TEXT`
|
||||
- `externe_identifier TEXT`
|
||||
|
||||
Also add `ALTER TABLE IF NOT EXISTS` guard calls in `_init_database()` for existing DBs
|
||||
that don't have these columns yet.
|
||||
|
||||
### `pipeline/text_pipeline.py` change
|
||||
|
||||
Update `_select_text` SQL:
|
||||
```
|
||||
COALESCE(m.layman_explanation, m.body_text, m.description, m.title)
|
||||
```
|
||||
(adds `m.body_text` as second-priority fallback)
|
||||
|
||||
### `scripts/rerun_embeddings.py` (new or inline in sync script)
|
||||
|
||||
After enrichment:
|
||||
1. `DELETE FROM embeddings` — wipe all stale embeddings (they're all "Aangenomen.")
|
||||
2. Run `pipeline.text_pipeline.ensure_text_embeddings(db_path, model, batch_size)`
|
||||
3. Run `pipeline.fusion.fuse_for_window(window_id, db_path)` for all 20 windows
|
||||
4. Run `similarity.compute.compute_similarities(vector_type='fused', window_id=w)` for
|
||||
all 20 windows
|
||||
|
||||
## Data Flow
|
||||
|
||||
```
|
||||
motions.url
|
||||
→ extract besluit_uuid (split on '/')
|
||||
→ look up in title_map → UPDATE motions.title, motions.description
|
||||
→ look up in ext_id_map → UPDATE motions.externe_identifier
|
||||
→ fetch HTML → UPDATE motions.body_text
|
||||
|
||||
text_pipeline._select_text
|
||||
→ COALESCE(layman_explanation, body_text, description, title)
|
||||
→ now returns real motion text for ~60-80% of motions
|
||||
→ outcome string fallback for the rest
|
||||
|
||||
fused_embeddings
|
||||
→ [svd_vector || text_vector] (text now has semantic content)
|
||||
|
||||
similarity_cache
|
||||
→ re-computed for all 20 windows with meaningful vectors
|
||||
```
|
||||
|
||||
## Error Handling Strategy
|
||||
|
||||
- **SyncFeed**: exponential backoff on 429/5xx; log and skip individual malformed entries;
|
||||
checkpoint skiptoken to disk so walk can resume after crash
|
||||
- **Body text fetch**: catch all per-URL exceptions, log, continue; motions without body
|
||||
text fall back to Zaak.Onderwerp in COALESCE
|
||||
- **DB update**: use DuckDB transactions per batch of 1000; rollback on failure
|
||||
- **Missing Zaak/Document**: expected for procedural votes; log counts; these motions get
|
||||
title = NULL → COALESCE falls back to "Aangenomen." as before
|
||||
|
||||
## Testing Strategy
|
||||
|
||||
- Unit tests for each XML parser using hardcoded fixture XML strings
|
||||
- Unit test for `build_title_map` with a small synthetic index
|
||||
- Integration test: walk 1 page of Besluit SyncFeed live, assert > 0 entries returned
|
||||
- After full run: query `SELECT COUNT(*) FROM motions WHERE title NOT IN ('Aangenomen.',
|
||||
'Verworpen.', 'Gestaakt.')` — expect > 10,000
|
||||
- After embeddings: spot-check cosine similarity between two related motions (same topic)
|
||||
is higher than between unrelated motions
|
||||
|
||||
## Open Questions
|
||||
|
||||
- **Document–Zaak relationship**: The SyncFeed Document entity may reference multiple
|
||||
Zaak IDs. For motions with multiple linked documents, we prefer the one with
|
||||
Soort="Motie" on the Zaak. Edge cases may need manual inspection.
|
||||
- **SyncFeed total record count**: Unknown until walked. Estimate 2,000–6,000 pages total
|
||||
across 4 feeds. Could be more for Document/DocumentVersie.
|
||||
- **Rate limits**: SyncFeed documentation doesn't specify limits. Start at 1 req/s,
|
||||
increase if no 429s.
|
||||
- **Body text coverage**: Not all motions have an associated kamerstuk document.
|
||||
Procedural votes (e.g., "Rondgezonden en gepubliceerd") typically won't. Expect
|
||||
40–60% body text coverage.
|
||||
Reference in New Issue
Block a user