sgeboers/motief
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

feat(mp-quiz): add MP quiz tab and DB helpers; add design and plan docs

Browse Source
This commit is contained in:
sgeboers
2026-03-24 19:59:25 +01:00
parent b09e580f65
commit eb73275f32
17 changed files with 2879 additions and 191 deletions
Download Patch File Download Diff File Expand all files Collapse all files
thoughts/ledgers/CONTINUITY_stemwijzer.md
+47 -4
View File
@@ -1,9 +1,8 @@
# Session: stemwijzer — Parliamentary Embedding Pipeline
Updated: 2026-03-22T16:00:00Z
# Session: stemwijzer
Updated: 2026-03-23T09:00:00Z
## Goal
2D political compass + motion similarity search from parliamentary votes + motion text.
Full historical coverage 2016–2026, precomputed similarity cache, fused (SVD + text) embeddings.
2D political compass + motion similarity search from parliamentary votes + motion text. Full historical coverage 2016–2026, precomputed similarity cache, fused (SVD + text) embeddings.
## Constraints
- DuckDB only (`data/motions.db`); open/close `duckdb.connect(self.db_path)` per method
@@ -77,3 +76,47 @@ Full historical coverage 2016–2026, precomputed similarity cache, fused (SVD +
## Branch
`main`
## Progress
### Done
- [x] All items listed under "Completed This Session" above
### In Progress
- [ ] Rerun embeddings: started scripts/rerun_embeddings.py against `data/motions.db`
- Start time: 2026-03-23T01:42:00Z (approx)
- Current progress: embeddings stored = 950 / total motions = 28,172
- fused_embeddings = 0 (not started)
- similarity_cache = 0 (not started)
### Blocked
- Not fully blocked, but encountering provider failures and warnings that slow progress:
- Batch 951..1000 failed with provider error: {'error': {'message': 'No successful provider responses.', 'code': 404}} (recorded)
- Occasional connection pool warnings during earlier body fetch phase (logged)
- Provider failures are transient but may require retries or provider change if repeated
## Key Decisions
- **Retry strategy on provider failure**: On repeated provider failures, retry embedding batches with smaller batch_size (e.g. 50 -> 20) or switch provider. Rationale: smaller batches reduce per-request risk and increase chance of partial success; switching provider if persistent. (UNCONFIRMED)
## Next Steps
1. Continue the rerun_embeddings job until completion; monitor batches closely
2. If provider failures repeat, retry failed batches with smaller batch_size (50 -> 20) or switch provider (as above)
3. On completion, update ledger with final counts and list any failed motion IDs
4. If fused_embeddings / similarity_cache remain 0 after embeddings finished, run fusion and similarity recompute pipelines
## File Operations
### Read
- `data/motions.db`
- `scripts/rerun_embeddings.py` (invoked)
### Modified
- `thoughts/ledgers/CONTINUITY_stemwijzer.md` (this file)
## Critical Context
- Rerun started 2026-03-23T01:42Z; current embeddings stored = 950 of 28,172 total motions.
- Recent error: Batch 951..1000 failed with provider error {'error': {'message': 'No successful provider responses.', 'code': 404}} — these batch numbers and error payload should be retried.
- ETA: approx 1.5–2.5 hours remaining at current rate (UNCONFIRMED, depends on provider stability)
- Earlier stage produced occasional connection pool warnings while fetching motion bodies; these did not stop progress but may indicate transient network instability.
## Working Set
- Branch: `main`
- Key files: `data/motions.db`, `scripts/rerun_embeddings.py`, `thoughts/ledgers/CONTINUITY_stemwijzer.md`
thoughts/ledgers/audit_events.json
+549
View File
@@ -0,0 +1,549 @@
[
{
"id": "91d3a66a-5542-4325-8fc0-f0715b570e5b",
"actor_id": null,
"action": "embedding_failed",
"target_type": "motion",
"target_id": "99",
"metadata": {
"error": "RuntimeError(\"Simulated embedding failure for index 0: 'failing motion'\")"
},
"created_at": "2026-03-23T20:01:06.889693Z"
},
{
"id": "1e1bcee0-5f2a-4337-b57f-ca83ac93da7e",
"actor_id": null,
"action": "test_action",
"target_type": "unit",
"target_id": "u1",
"metadata": {
"k": 1
},
"created_at": "2026-03-23T20:01:07.327717Z"
},
{
"id": "51fec578-84ae-4d69-85df-b415a2b6c752",
"actor_id": null,
"action": "another_action",
"target_type": "motion",
"target_id": null,
"metadata": {},
"created_at": "2026-03-23T20:01:07.827925Z"
},
{
"id": "38c9ba12-5829-410c-ac38-b992a9b22652",
"actor_id": null,
"action": "embedding_failed",
"target_type": "motion",
"target_id": "99",
"metadata": {
"error": "RuntimeError(\"Simulated embedding failure for index 0: 'failing motion'\")"
},
"created_at": "2026-03-23T20:01:11.681524Z"
},
{
"id": "1e69741a-d0ff-41c9-b5e7-e4e2e8475836",
"actor_id": null,
"action": "test_action",
"target_type": "unit",
"target_id": "u1",
"metadata": {
"k": 1
},
"created_at": "2026-03-23T20:01:12.061577Z"
},
{
"id": "c2fa7d58-958b-4efb-b670-d044de0db357",
"actor_id": null,
"action": "another_action",
"target_type": "motion",
"target_id": null,
"metadata": {},
"created_at": "2026-03-23T20:01:12.104491Z"
},
{
"id": "4bd245c7-9e6c-41dc-bce0-f36d3b675ef8",
"actor_id": null,
"action": "embedding_failed",
"target_type": "motion",
"target_id": "99",
"metadata": {
"error": "RuntimeError(\"Simulated embedding failure for index 0: 'failing motion'\")"
},
"created_at": "2026-03-23T20:03:14.657886Z"
},
{
"id": "e885a2da-48f3-4130-b62f-413ae2670b9c",
"actor_id": null,
"action": "test_action",
"target_type": "unit",
"target_id": "u1",
"metadata": {
"k": 1
},
"created_at": "2026-03-23T20:03:14.997464Z"
},
{
"id": "29949f88-0739-4029-964a-d1a9be3d1030",
"actor_id": null,
"action": "another_action",
"target_type": "motion",
"target_id": null,
"metadata": {},
"created_at": "2026-03-23T20:03:15.057155Z"
},
{
"id": "b08b870b-1923-4cc7-8384-6a4fd5a5ae63",
"actor_id": null,
"action": "test_action",
"target_type": "unit",
"target_id": "u1",
"metadata": {
"k": 1
},
"created_at": "2026-03-23T20:08:18.540282Z"
},
{
"id": "acb1d1ef-1a2f-4256-b23c-dc7272e6cda8",
"actor_id": null,
"action": "another_action",
"target_type": "motion",
"target_id": null,
"metadata": {},
"created_at": "2026-03-23T20:08:18.992755Z"
},
{
"id": "a2e7f741-ce46-4533-a4a1-98d202ad5ba9",
"actor_id": null,
"action": "embedding_failed",
"target_type": "motion",
"target_id": "99",
"metadata": {
"error": "RuntimeError(\"Simulated embedding failure for index 0: 'failing motion'\")"
},
"created_at": "2026-03-23T20:16:44.733143Z"
},
{
"id": "208fd3d6-dcaa-408a-b7ef-054703083756",
"actor_id": null,
"action": "test_action",
"target_type": "unit",
"target_id": "u1",
"metadata": {
"k": 1
},
"created_at": "2026-03-23T20:16:45.983301Z"
},
{
"id": "9112851e-ba85-4498-8aa9-4f71aa91d6ec",
"actor_id": null,
"action": "another_action",
"target_type": "motion",
"target_id": null,
"metadata": {},
"created_at": "2026-03-23T20:16:46.060827Z"
},
{
"id": "2a5bffe4-c75b-46f2-baf1-164ac87953d6",
"actor_id": null,
"action": "embedding_failed",
"target_type": "motion",
"target_id": "99",
"metadata": {
"error": "RuntimeError(\"Simulated embedding failure for index 0: 'failing motion'\")"
},
"created_at": "2026-03-23T20:18:02.707023Z"
},
{
"id": "834a7419-6d1b-48e0-825c-08c3ea780c94",
"actor_id": null,
"action": "test_action",
"target_type": "unit",
"target_id": "u1",
"metadata": {
"k": 1
},
"created_at": "2026-03-23T20:18:04.158612Z"
},
{
"id": "9d27c575-c186-4cc6-b202-1e7d44600983",
"actor_id": null,
"action": "another_action",
"target_type": "motion",
"target_id": null,
"metadata": {},
"created_at": "2026-03-23T20:18:04.213958Z"
},
{
"id": "b03f2ce4-6ac0-41a5-8f32-36dc86db4048",
"actor_id": null,
"action": "embedding_failed",
"target_type": "motion",
"target_id": "99",
"metadata": {
"error": "RuntimeError(\"Simulated embedding failure for index 0: 'failing motion'\")"
},
"created_at": "2026-03-23T20:19:07.187178Z"
},
{
"id": "4589cfd1-16c9-4743-b2e9-15be42e121e7",
"actor_id": null,
"action": "test_action",
"target_type": "unit",
"target_id": "u1",
"metadata": {
"k": 1
},
"created_at": "2026-03-23T20:19:07.560463Z"
},
{
"id": "b2e8fa30-b3ca-42f1-9138-c188b2683723",
"actor_id": null,
"action": "another_action",
"target_type": "motion",
"target_id": null,
"metadata": {},
"created_at": "2026-03-23T20:19:07.631447Z"
},
{
"id": "492bd375-b002-446b-b424-4dc3cf40ea44",
"actor_id": null,
"action": "embedding_failed",
"target_type": "motion",
"target_id": "99",
"metadata": {
"error": "RuntimeError(\"Simulated embedding failure for index 0: 'failing motion'\")"
},
"created_at": "2026-03-23T20:20:12.734568Z"
},
{
"id": "e9349d9b-7962-489b-8a06-1753f1606048",
"actor_id": null,
"action": "test_action",
"target_type": "unit",
"target_id": "u1",
"metadata": {
"k": 1
},
"created_at": "2026-03-23T20:20:13.089162Z"
},
{
"id": "a8af51f1-2126-4b57-87e3-6913ada4643b",
"actor_id": null,
"action": "another_action",
"target_type": "motion",
"target_id": null,
"metadata": {},
"created_at": "2026-03-23T20:20:13.152945Z"
},
{
"id": "999cccc1-dcbe-48c7-80f0-229351780823",
"actor_id": null,
"action": "embedding_failed",
"target_type": "motion",
"target_id": "99",
"metadata": {
"error": "RuntimeError(\"Simulated embedding failure for index 0: 'failing motion'\")"
},
"created_at": "2026-03-23T20:20:26.037664Z"
},
{
"id": "03acdcb0-da76-4ac7-bc7d-ea5a314734b8",
"actor_id": null,
"action": "test_action",
"target_type": "unit",
"target_id": "u1",
"metadata": {
"k": 1
},
"created_at": "2026-03-23T20:20:27.339457Z"
},
{
"id": "964943f7-c431-484a-a2a3-070327287d90",
"actor_id": null,
"action": "another_action",
"target_type": "motion",
"target_id": null,
"metadata": {},
"created_at": "2026-03-23T20:20:27.382680Z"
},
{
"id": "4d4ffec0-83c9-48e6-8724-df04de5cf741",
"actor_id": null,
"action": "test_action",
"target_type": "unit",
"target_id": "u1",
"metadata": {
"k": 1
},
"created_at": "2026-03-23T20:25:42.534010Z"
},
{
"id": "56511987-38b1-4286-a8ac-9771c297051f",
"actor_id": null,
"action": "another_action",
"target_type": "motion",
"target_id": null,
"metadata": {},
"created_at": "2026-03-23T20:25:42.861164Z"
},
{
"id": "89e29b19-8926-47c8-8a52-922a65da4189",
"actor_id": null,
"action": "embedding_failed",
"target_type": "motion",
"target_id": "99",
"metadata": {
"error": "RuntimeError(\"Simulated embedding failure for index 0: 'failing motion'\")"
},
"created_at": "2026-03-23T20:25:44.542652Z"
},
{
"id": "a430eb8d-c36e-4ad1-b5c3-01c1fc5f3be5",
"actor_id": null,
"action": "embedding_failed",
"target_type": "motion",
"target_id": "99",
"metadata": {
"error": "RuntimeError(\"Simulated embedding failure for index 0: 'failing motion'\")"
},
"created_at": "2026-03-23T20:30:11.148013Z"
},
{
"id": "2b108662-6d4b-48b6-88ba-9f3111491217",
"actor_id": null,
"action": "test_action",
"target_type": "unit",
"target_id": "u1",
"metadata": {
"k": 1
},
"created_at": "2026-03-23T20:30:12.361529Z"
},
{
"id": "18aad6fc-567f-4d7d-8569-28357c6c301d",
"actor_id": null,
"action": "another_action",
"target_type": "motion",
"target_id": null,
"metadata": {},
"created_at": "2026-03-23T20:30:12.429871Z"
},
{
"id": "730467ab-2c1d-47ec-a29e-9d02803c8b1f",
"actor_id": null,
"action": "embedding_failed",
"target_type": "motion",
"target_id": "99",
"metadata": {
"error": "RuntimeError(\"Simulated embedding failure for index 0: 'failing motion'\")"
},
"created_at": "2026-03-23T20:31:09.659557Z"
},
{
"id": "e87170fe-c03b-42a6-bd96-d45a26717359",
"actor_id": null,
"action": "test_action",
"target_type": "unit",
"target_id": "u1",
"metadata": {
"k": 1
},
"created_at": "2026-03-23T20:31:10.991569Z"
},
{
"id": "cf5c8957-c893-478d-924b-ce77d9e53a41",
"actor_id": null,
"action": "another_action",
"target_type": "motion",
"target_id": null,
"metadata": {},
"created_at": "2026-03-23T20:31:11.039826Z"
},
{
"id": "d0aa2441-6fc8-4df9-bf37-621f8d9e5351",
"actor_id": null,
"action": "embedding_failed",
"target_type": "motion",
"target_id": "99",
"metadata": {
"error": "RuntimeError(\"Simulated embedding failure for index 0: 'failing motion'\")"
},
"created_at": "2026-03-23T21:37:36.823732Z"
},
{
"id": "e55be18c-0fb5-4576-aa51-24559148916e",
"actor_id": null,
"action": "test_action",
"target_type": "unit",
"target_id": "u1",
"metadata": {
"k": 1
},
"created_at": "2026-03-23T21:37:38.061523Z"
},
{
"id": "b8f7ee6c-4ce5-4415-97bc-e00e02b2c851",
"actor_id": null,
"action": "another_action",
"target_type": "motion",
"target_id": null,
"metadata": {},
"created_at": "2026-03-23T21:37:38.105562Z"
},
{
"id": "75e76c14-28e5-4f5b-a2d4-52c9dba5efc4",
"actor_id": null,
"action": "embedding_failed",
"target_type": "motion",
"target_id": "99",
"metadata": {
"error": "RuntimeError(\"Simulated embedding failure for index 0: 'failing motion'\")"
},
"created_at": "2026-03-23T21:53:11.948658Z"
},
{
"id": "0e515c39-ef8a-4abf-87fb-1c96e669e7f5",
"actor_id": null,
"action": "test_action",
"target_type": "unit",
"target_id": "u1",
"metadata": {
"k": 1
},
"created_at": "2026-03-23T21:53:13.215965Z"
},
{
"id": "836ba937-d26f-4869-9661-36d5744f649a",
"actor_id": null,
"action": "another_action",
"target_type": "motion",
"target_id": null,
"metadata": {},
"created_at": "2026-03-23T21:53:13.259045Z"
},
{
"id": "95606fed-c023-4fb1-98a4-5c5707e95056",
"actor_id": null,
"action": "embedding_failed",
"target_type": "motion",
"target_id": "99",
"metadata": {
"error": "RuntimeError(\"Simulated embedding failure for index 0: 'failing motion'\")"
},
"created_at": "2026-03-23T21:54:11.170757Z"
},
{
"id": "e188e203-932c-4f79-bed9-af1ec1336102",
"actor_id": null,
"action": "test_action",
"target_type": "unit",
"target_id": "u1",
"metadata": {
"k": 1
},
"created_at": "2026-03-23T21:54:12.444776Z"
},
{
"id": "591eb30e-a91f-448b-b9a0-9a3d75a35790",
"actor_id": null,
"action": "another_action",
"target_type": "motion",
"target_id": null,
"metadata": {},
"created_at": "2026-03-23T21:54:12.491990Z"
},
{
"id": "75d70cad-41f4-49f8-ae8a-08b1bafb02b4",
"actor_id": null,
"action": "embedding_failed",
"target_type": "motion",
"target_id": "99",
"metadata": {
"error": "RuntimeError(\"Simulated embedding failure for index 0: 'failing motion'\")"
},
"created_at": "2026-03-23T21:58:08.120685Z"
},
{
"id": "a0cc28c1-28f6-409e-ad5c-8a974ff28e00",
"actor_id": null,
"action": "test_action",
"target_type": "unit",
"target_id": "u1",
"metadata": {
"k": 1
},
"created_at": "2026-03-23T21:58:08.847188Z"
},
{
"id": "84e5699d-64d9-4179-99e2-f79cb9b27b33",
"actor_id": null,
"action": "another_action",
"target_type": "motion",
"target_id": null,
"metadata": {},
"created_at": "2026-03-23T21:58:08.888009Z"
},
{
"id": "f0e47b0a-4791-4475-9036-9e87a5e00be8",
"actor_id": null,
"action": "embedding_failed",
"target_type": "motion",
"target_id": "99",
"metadata": {
"error": "RuntimeError(\"Simulated embedding failure for index 0: 'failing motion'\")"
},
"created_at": "2026-03-23T22:31:18.698163Z"
},
{
"id": "50006ac9-d6b5-4198-9cca-c937dee60eab",
"actor_id": null,
"action": "test_action",
"target_type": "unit",
"target_id": "u1",
"metadata": {
"k": 1
},
"created_at": "2026-03-23T22:31:19.938465Z"
},
{
"id": "d44d09e5-fa20-40ea-8829-dfa044addf57",
"actor_id": null,
"action": "another_action",
"target_type": "motion",
"target_id": null,
"metadata": {},
"created_at": "2026-03-23T22:31:19.997516Z"
},
{
"id": "c1876f8b-79bd-4fd0-9d0f-68eb8dd3321d",
"actor_id": null,
"action": "embedding_failed",
"target_type": "motion",
"target_id": "99",
"metadata": {
"error": "RuntimeError(\"Simulated embedding failure for index 0: 'failing motion'\")"
},
"created_at": "2026-03-23T22:52:46.531868Z"
},
{
"id": "39fb4426-e348-455c-b4f7-0e77c6a72dc7",
"actor_id": null,
"action": "test_action",
"target_type": "unit",
"target_id": "u1",
"metadata": {
"k": 1
},
"created_at": "2026-03-23T22:52:47.772599Z"
},
{
"id": "fc04bbb8-1378-460c-9914-c923cd1a45f8",
"actor_id": null,
"action": "another_action",
"target_type": "motion",
"target_id": null,
"metadata": {},
"created_at": "2026-03-23T22:52:47.836920Z"
}
]
thoughts/ledgers/qa_similarity_20260323T194335Z.json
+29
View File
@@ -0,0 +1,29 @@
{
"timestamp": "2026-03-23T19:43:35.098568Z",
"sample_size": 2,
"top_k": 3,
"results": [
{
"motion_id": 1,
"top_k": 3,
"suspicious": 1
},
{
"motion_id": 2,
"top_k": 3,
"suspicious": 1
}
],
"motions": {
"1": {
"motion_id": 1,
"top_k": 3,
"suspicious": 1
},
"2": {
"motion_id": 2,
"top_k": 3,
"suspicious": 1
}
}
}