---
title: ARCHITECTURE
---

## Overview

- Small Python project that collects, stores and presents Dutch parliamentary motions (Tweede Kamer). It ingests votes via OData API, stores motions in a DuckDB file, generates short human-readable summaries using an LLM client, and exposes a Streamlit UI for users to vote and view matching results.

## Tech stack

- Language: Python (single-project repository)
- Data: DuckDB (file: data/motions.db)
- Web / UI: Streamlit (app.py, pages/)
- HTTP: requests (ai_provider.py, api_client.py)
- LLM: QWEN (via OpenRouter) / OpenAI-compatible client (ai_provider.py). Prefer QWEN via OpenRouter where possible.
- Analysis: scipy (SVD), scikit-learn (clustering), umap-learn (dimensionality reduction)
- Visualization: Plotly
- Packaging: pyproject.toml

## Top-level layout (annotated)

./
- app.py               — Streamlit UI entrypoint (Home.py routing)
- Home.py              — Thin wrapper with minimal logic
- database.py          — MotionDatabase: DuckDB schema, insert/query/update, party-match calculations
- api_client.py        — TweedeKamerAPI: fetch OData voting records and group into motions
- summarizer.py        — MotionSummarizer: LLM integration to generate layman_explanation
- config.py            — Config dataclass: central configuration (DATABASE_PATH, API/AI settings, constants)
- ai_provider.py       — Lightweight HTTP wrapper around OpenRouter/OpenAI-style backends
- explorer.py          — Explorer page logic, tab routing, SVD visualization
- explorer_helpers.py  — Pure functions for chart builders, coordinate computation
- data/                — data/motions.db (DuckDB file, ~18GB)
- pyproject.toml       — project metadata / dependencies
- .env                 — environment variables (not printed here)

## Directory structure

- `pages/`             — Streamlit pages: 1_Stemwijzer.py, 2_Explorer.py
- `pipeline/`          — Data ingestion pipelines: run_pipeline.py, svd_pipeline.py, text_pipeline.py
- `analysis/`          — SVD, clustering, trajectory, visualization modules
- `similarity/`        — Embedding-based similarity computation
- `scripts/`           — Utility scripts for data processing
- `tests/`             — Test suite using pytest
- `migrations/`        — SQL migration files

## Core components

- Streamlit UI (app.py + pages/)
  - Presents the voting UI, reads filtered motions from database, creates sessions, writes user votes
  - Explorer page (explorer.py) provides SVD visualization and party trajectory analysis
- Storage (database.py)
  - MotionDatabase encapsulates DuckDB schema creation and CRUD for motions and user sessions
  - Exposes a module-level instance `db = MotionDatabase()` used across the codebase
  - Key responsibilities: insert_motion, get_filtered_motions, create_session, update_user_vote, calculate_party_matches
- Ingestion (api_client.py + pipeline/)
  - api_client.py fetches votes via Tweede Kamer OData API and groups records into motions
  - pipeline/ orchestrates the full ingestion and analysis workflow
- Summarization (summarizer.py)
  - Wraps an OpenRouter/OpenAI-compatible client (QWEN via OpenRouter recommended) to produce short layman explanations and persists them to DB
  - Reads motions without layman_explanation and updates rows
- Analysis (analysis/)
  - SVD decomposition of voting patterns
  - UMAP for visualization
  - Clustering for motion grouping
  - Trajectory computation for party movement over time

## Data flow (high level)

1.  Ingestion
    - Pipeline triggers TweedeKamerAPI.get_motions(...)
    - Each produced motion dict is passed to MotionDatabase.insert_motion()
    - insert_motion writes to DuckDB (data/motions.db)
2.  Enrichment
    - summarizer.update_motion_summaries() reads motions lacking layman_explanation, calls the LLM client and writes summary text back to the DB
3.  Analysis
    - pipeline/svd_pipeline.py computes SVD embeddings from vote matrix
    - Results stored in svd_vectors table for visualization
4.  Presentation / Interaction
    - app.py (Streamlit) queries motions via db.get_filtered_motions() and displays them
    - Users vote; app.py writes votes into the database via db.update_user_vote()
    - app.py calls db.calculate_party_matches() to compute match percentages for parties

## External integrations & dependencies

- Tweede Kamer OData API (api_client.py)
- HTTP (requests)
- DuckDB (database file at data/motions.db)
- Streamlit for UI
- OpenRouter/OpenAI-compatible LLM client (ai_provider.py) — configured with environment variables in config.py

## Configuration

- config.py: central Config dataclass. Observed keys / env variables referenced across the codebase include:
  - config.DATABASE_PATH (default "data/motions.db")
  - OPENROUTER_API_KEY / other OPENROUTER_* variables used by ai_provider.py
  - QWEN_MODEL (or other model identifier) referenced in summarizer.py
  - API timeout / batch size constants
- .env file present at repo root (do not commit secrets)
- Packaging metadata: pyproject.toml

## Build, run & development notes

- Install dependencies via the project's Python packaging (pyproject.toml)
- Use `uv add` and `uv run` to manage the dependencies in this directory and run scripts
- Streamlit app: run `uv run streamlit run app.py` from project root to start the UI (app.py is the intended web entrypoint)
- Never use pip directly!
- Run tests: `uv run pytest tests/`

## Tests

- Test suite in `tests/` using pytest
- Run with `uv run pytest tests/`

## Notes / caveats

- Project is synchronous (no async/await patterns detected)
- Many modules rely on module-level singletons (e.g., `db = MotionDatabase()`, `summarizer = MotionSummarizer()`)
- Error handling frequently catches broad Exception and prints to stdout (see database.py, api_client.py)
- Logging is not centralized (print statements used)

## Where to look first (for contributors)

- app.py + pages/     — follow the UI flow and see how votes & sessions are used
- database.py          — core data model and calculations
- explorer.py          — SVD visualization and party analysis
- api_client.py        — OData ingestion logic
- summarizer.py        — LLM usage and environment variables
- pipeline/            — how ingestion and analysis is orchestrated