|
|
# System Overview
|
|
|
|
|
|
## Project: Stemwijzer (Dutch Political Voting Compass)
|
|
|
|
|
|
**Purpose**: A web application that maps the Dutch Tweede Kamer (House of Representatives) based on real parliamentary votes, helping citizens discover which political party aligns best with their views.
|
|
|
|
|
|
## Architecture Summary
|
|
|
|
|
|
### Data Flow
|
|
|
```
|
|
|
TweedeKamer OData API
|
|
|
↓
|
|
|
API Client (api_client.py)
|
|
|
↓
|
|
|
DuckDB Database (database.py)
|
|
|
↓
|
|
|
Pipeline Processing (pipeline/)
|
|
|
├── fetch_mp_metadata # MP party + tenure
|
|
|
├── extract_mp_votes # voting_results → mp_votes
|
|
|
├── svd_pipeline # SVD on vote matrix + Procrustes
|
|
|
├── text_pipeline # AI embeddings via OpenRouter
|
|
|
└── fusion # Combine SVD + text vectors
|
|
|
↓
|
|
|
Streamlit Web App (Home.py, pages/)
|
|
|
├── Home.py # Landing page
|
|
|
├── 1_Stemwijzer.py # Voting quiz
|
|
|
└── 2_Explorer.py # Political compass explorer
|
|
|
```
|
|
|
|
|
|
### Key Components
|
|
|
|
|
|
| Component | Purpose | File(s) |
|
|
|
|-----------|---------|---------|
|
|
|
| **Database** | Motion storage, MP votes, embeddings | `database.py` |
|
|
|
| **API Client** | TweedeKamer OData API integration | `api_client.py` |
|
|
|
| **AI Provider** | OpenRouter API for embeddings/summaries | `ai_provider.py` |
|
|
|
| **Pipeline** | Orchestrated data processing | `pipeline/run_pipeline.py` |
|
|
|
| **Analysis** | SVD, clustering, trajectory computation | `analysis/*.py` |
|
|
|
| **Explorer Helpers** | Pure functions, chart builders | `explorer_helpers.py` |
|
|
|
| **Web App** | Streamlit UI | `Home.py`, `pages/*.py` |
|
|
|
|
|
|
### Tech Stack
|
|
|
|
|
|
- **Language**: Python 3.13+
|
|
|
- **Web Framework**: Streamlit (multi-page app)
|
|
|
- **Database**: DuckDB with ibis ORM (DuckDB-native implementation)
|
|
|
- **ML/Analytics**: scipy (SVD, Procrustes), scikit-learn (KMeans, cosine_similarity), umap-learn (optional)
|
|
|
- **AI/LLM**: OpenRouter-compatible API (QWEN embeddings + chat)
|
|
|
- **Visualization**: Plotly (interactive charts), matplotlib (optional)
|
|
|
- **HTTP**: requests with Session pooling and retry
|
|
|
- **Parsing**: beautifulsoup4, lxml
|
|
|
|
|
|
### Key Patterns
|
|
|
|
|
|
1. **Module-Level Singletons**: `db = MotionDatabase()`, `config = Config()`
|
|
|
2. **Repository Pattern**: MotionDatabase class with method-per-query
|
|
|
3. **Service Layer**: TweedeKamerAPI, ai_provider with retry/backoff
|
|
|
4. **Pipeline Orchestration**: ThreadPoolExecutor for parallel SVD
|
|
|
5. **Short-Lived Connections**: DuckDB connections in try/finally blocks
|
|
|
6. **Graceful Degradation**: try/except around optional dependencies
|
|
|
|
|
|
### Domain Invariants
|
|
|
|
|
|
⚠️ **CRITICAL RULES** (from AGENTS.md):
|
|
|
|
|
|
1. **Right-wing parties on RIGHT**: PVV, FVD, JA21, SGP must appear on RIGHT side of all axes in visualizations
|
|
|
2. **SVD labels = voting patterns**: SVD labels reflect voting patterns, NOT semantic content
|
|
|
|
|
|
### Database Tables
|
|
|
|
|
|
| Table | Purpose |
|
|
|
|-------|---------|
|
|
|
| `motions` | Parliamentary motions with id, title, date, category |
|
|
|
| `mp_votes` | Individual MP votes on motions (Voor/Tegen/Onthouden) |
|
|
|
| `mp_metadata` | MP names, parties, tenure info |
|
|
|
| `svd_vectors` | 2D SVD-computed political positions per entity |
|
|
|
| `fused_embeddings` | Combined SVD + text embeddings |
|
|
|
| `embeddings` | Text embeddings for motions |
|
|
|
| `user_sessions` | Voting session tracking |
|
|
|
| `party_results` | Party match results per session |
|
|
|
|
|
|
### Conventions
|
|
|
|
|
|
- **Error Handling**: Catch `Exception`, return safe fallbacks (False/[]/None)
|
|
|
- **Logging**: Use `logging.getLogger(__name__)` — **never use print()**
|
|
|
- **Imports**: stdlib → 3rd party → local (3 groups)
|
|
|
- **Type Hints**: Required on public functions with typing module imports
|
|
|
- **DuckDB**: Short-lived connections with try/finally conn.close()
|
|
|
|
