chore: commit remaining modified files from refactoring

2026-04-04 18:47:12 +02:00
parent 805cc7e284
commit eb71328967
7 changed files with 651 additions and 1392 deletions
@@ -1,15 +1,74 @@
-# DO NOT EDIT - read-only until validated
-# Sanitized manifest: contains non-sensitive sample excerpts only
-files:
-  - path: src/lib/schema.ts
-    evidence_excerpt: "Defines schema for user input validation"
-    flags:
-      needs_review: true
-  - path: src/api/handler.ts
-    evidence_excerpt: "Handles API requests and routing"
-    flags:
-      needs_review: false
-  - path: README.md
-    evidence_excerpt: "Project overview and setup instructions"
-    flags:
-      needs_review: true
+name: stemwijzer
+version: 2
+description: Dutch political voting compass (Stemwijzer) - Mind Model constraints
+
+categories:
+  # Core documentation
+  - path: system.md
+    description: System overview and architecture summary
+    group: docs
+  - path: tech-stack.yaml
+    description: Technology stack with versions and purposes
+    group: docs
+  - path: conventions.yaml
+    description: Coding conventions and style guide
+    group: docs
+  - path: domain.yaml
+    description: Domain entities, terms, and relationships
+    group: docs
+  
+  # Design patterns
+  - path: patterns/architecture.yaml
+    description: Repository, Facade, Pipeline architectural patterns
+    group: patterns
+  - path: patterns/python.yaml
+    description: Python-specific patterns (Singleton, dataclass, context manager)
+    group: patterns
+  - path: patterns/database.yaml
+    description: DuckDB connection patterns and ORM usage
+    group: patterns
+  - path: patterns/api.yaml
+    description: API client patterns with retry logic and pagination
+    group: patterns
+  - path: patterns/streamlit.yaml
+    description: Streamlit session state and page patterns
+    group: patterns
+  
+  # Coding constraints
+  - path: constraints/error-handling.yaml
+    description: Error handling patterns with safe fallbacks
+    group: constraints
+  - path: constraints/logging.yaml
+    description: Logging conventions and best practices
+    group: constraints
+  - path: constraints/naming.yaml
+    description: File, class, function naming rules
+    group: constraints
+  - path: constraints/imports.yaml
+    description: Import organization and module structure
+    group: constraints
+  - path: constraints/types.yaml
+    description: Type hint conventions
+    group: constraints
+  
+  # Code examples
+  - path: examples/database-example.py
+    description: MotionDatabase usage example
+    group: examples
+  - path: examples/api-client-example.py
+    description: TweedeKamerAPI usage
+    group: examples
+  - path: examples/pipeline-example.py
+    description: Pipeline phase example
+    group: examples
+  - path: examples/streamlit-page-example.py
+    description: Streamlit page pattern
+    group: examples
+  
+  # Anti-patterns and workflows
+  - path: anti-patterns.yaml
+    description: Known anti-patterns to avoid
+    group: meta
+  - path: workflows.yaml
+    description: Key workflows (VotingSession, DataIngestion, EmbeddingGeneration)
+    group: meta
@@ -1,14 +1,69 @@
-# System Overview: Stemwijzer
+# System Overview

-This mindmodel documents constraints, conventions and patterns for the Stemwijzer
-project (Python Streamlit app with DuckDB-backed pipeline for parliamentary
-motions embedding analysis).
+## Project: Stemwijzer (Dutch Political Voting Compass)

-Key points:
- Language: Python >=3.13
- UI: Streamlit multi-page app (Home.py, pages/)
- Storage: DuckDB with JSON fallback for tests/dev (database.py)
- Pipeline: ETL and SVD/text fusion pipeline (pipeline/run_pipeline.py)
- AI: ai_provider adapter uses HTTP-based OpenRouter/OpenAI-compatible API with retry/backoff and local fallback. QWEN via OpenRouter is the recommended path; prefer OPENROUTER_API_KEY with OPENAI_API_KEY as a fallback where applicable.
+**Purpose**: A web application that maps the Dutch Tweede Kamer (House of Representatives) based on real parliamentary votes, helping citizens discover which political party aligns best with their views.

-Use the .mindmodel/ constraints files to guide code changes, CI, and onboarding.
+## Architecture Summary
+
+### Data Flow
+```
+TweedeKamer OData API
+        ↓
+  API Client (api_client.py)
+        ↓
+  DuckDB Database (database.py)
+        ↓
+  Pipeline Processing (pipeline/)
+        ├── fetch_mp_metadata     # MP party + tenure
+        ├── extract_mp_votes     # voting_results → mp_votes
+        ├── svd_pipeline          # SVD on vote matrix + Procrustes
+        ├── text_pipeline         # AI embeddings via OpenRouter
+        └── fusion                # Combine SVD + text vectors
+        ↓
+  Streamlit Web App (app.py, pages/)
+        ├── Home.py               # Landing page
+        ├── 1_Stemwijzer.py       # Voting quiz
+        └── 2_Explorer.py        # Political compass explorer
+```
+
+### Key Components
+
+| Component | Purpose | File(s) |
+|-----------|---------|---------|
+| **Database** | Motion storage, MP votes, embeddings | `database.py` |
+| **API Client** | TweedeKamer OData API integration | `api_client.py` |
+| **AI Provider** | OpenRouter API for embeddings/summaries | `ai_provider.py` |
+| **Pipeline** | Orchestrated data processing | `pipeline/run_pipeline.py` |
+| **Analysis** | SVD, clustering, trajectory computation | `analysis/*.py` |
+| **Similarity** | Motion similarity search | `similarity/*.py` |
+| **Web App** | Streamlit UI | `app.py`, `pages/*.py` |
+
+### Data Models
+
+**Core Entities**:
+- `Motion`: Parliamentary motion with voting results
+- `MP` / `MPMetadata`: Member of Parliament with party/tenure
+- `MPVote`: Individual vote record (Voor/Tegen/Onthouden/Geen stem/Afwezig)
+- `Party`: Political party
+- `UserSession` / `UserVote`: Voting session tracking
+- `SVDVector`: Dimensionality-reduced vote vectors
+- `FusedEmbedding`: Combined SVD + text embedding
+- `SimilarityCache`: Pre-computed motion similarities
+
+### Technical Decisions
+
+1. **DuckDB over SQLite**: Chosen for OLAP performance with complex analytical queries
+2. **ibis ORM**: Database-agnostic query building (currently using DuckDB backend)
+3. **SVD + Procrustes**: Aligns voting vectors across time windows
+4. **UMAP for visualization**: Non-linear dimensionality reduction for compass display
+5. **OpenRouter API**: Abstraction layer for AI embeddings (currently using Qwen)
+6. **Module-level singletons**: `db = MotionDatabase()` pattern for shared state
+
+### Key Conventions
+
+- **DuckDB connections**: Short-lived per method, always close
+- **Error handling**: Catch `Exception`, return safe fallbacks (False/[]/None)
+- **Logging**: Use `logging.getLogger(__name__)` - avoid print()
+- **Type hints**: Required on public functions with typing module imports
+- **Config**: Dataclass `Config` in `config.py`, accessed as `from config import config`