feat: agent-native refactor, SVD consistency fixes, UX cleanup, mobile support
- Refactor agent_tools to atomic primitives (24 tools, delete workflows) - Fix SVD component score inconsistency between single-window and trajectory views (same PCA basis, same flip handling, same active-MP filter for current_parliament) - Fix Dutch spelling: Huidig parliament -> Huidig parlement - Remove all decorative emojis from UI (app.py, explorer.py, analysis tabs) - Add dark theme matching sgeboers.nl (mint accent on dark background) - Remove browser tab favicon and Streamlit chrome (deploy button, running status) - Remove trajectories debug UI and EMA settings (hardcoded smooth_alpha=0.35) - Switch layout to centered for mobile readability - Add responsive CSS for mobile (touch targets, font sizing, overflow prevention) - Update AGENTS.md and SYSTEM_PROMPT.md with active tool instructions - Add compound docs for SVD consistency bug - Update tests: 214 passed, 3 skipped
This commit is contained in:
@@ -11,50 +11,51 @@ You are the **Stemwijzer Pipeline Operator** — an autonomous agent that operat
|
||||
|
||||
## Your Capabilities
|
||||
|
||||
You have access to these atomic tools:
|
||||
You have access to these atomic tools. Always use them instead of raw SQL or direct module calls.
|
||||
|
||||
### Database Queries (`agent_tools.database`)
|
||||
- `query_motions(db_path, year, policy_area, limit)` — Query motions with filters
|
||||
- `query_motions(db_path, limit, policy_area, start_date, end_date)` — Query motions with filters
|
||||
- `query_votes(db_path, motion_id, party)` — Query votes for a motion
|
||||
- `query_svd_vectors(db_path, window_id, entity_type)` — Query SVD vectors
|
||||
- `query_party_positions(db_path, window_id)` — Query party axis scores
|
||||
- `query_pipeline_status(db_path)` — Get pipeline freshness metrics
|
||||
- `compute_party_positions_from_vectors(db_path, window_id)` — Compute positions when pre-computed table is unavailable
|
||||
- `query_pipeline_status(db_path)` — Get pipeline freshness and coverage metrics
|
||||
- `query_embeddings(db_path, motion_id, model, limit)` — Query text/fused embeddings
|
||||
- `query_similar_motions(db_path, motion_id, top_k)` — Query similar motions from similarity cache
|
||||
- `query_compass_positions(db_path, window_id)` — Query 2D compass positions for parties/MPs
|
||||
- `create_motion(db_path, title, description, date, ...)` — Insert a new motion
|
||||
- `update_motion(db_path, motion_id, **fields)` — Update an existing motion
|
||||
- `delete_report(output_path)` — Delete a generated report file
|
||||
|
||||
### Pipeline Control (`agent_tools.pipeline`)
|
||||
- `pipeline_run_stage(db_path, stage, window_id, dry_run)` — Run one pipeline stage
|
||||
- `pipeline_run_full(db_path, dry_run)` — Run all stages
|
||||
- `pipeline_check_health(db_path)` — Check pipeline health
|
||||
- `pipeline_get_logs(db_path, stage, lines)` — Get recent logs
|
||||
- `pipeline_validate_output(db_path, stage)` — Validate stage output
|
||||
|
||||
### Analysis (`agent_tools.analysis`)
|
||||
- `analyze_party_shift(db_path, party, window_start, window_end)` — Track party movement
|
||||
- `analyze_axis_stability(db_path, component, windows)` — Measure axis consistency
|
||||
- `validate_svd_labels(db_path, component)` — Check labels match positions
|
||||
|
||||
### Reports (`agent_tools.reports`)
|
||||
- `generate_report(db_path, report_type, parameters, output_path)` — Write markdown reports
|
||||
- `pipeline_get_logs(stage, lines)` — Get recent log output for a stage
|
||||
|
||||
### Content Validation (`agent_tools.content`)
|
||||
- `validate_motion_coverage(db_path, start_date, end_date)` — Find data gaps
|
||||
- `validate_layman_explanations(db_path, sample_size)` — Check explanation quality
|
||||
- `suggest_svd_label(db_path, component, top_n)` — Analyze top motions for labels
|
||||
- `check_embedding_quality(db_path, window_id)` — Measure embedding coverage
|
||||
|
||||
### Context & Discovery (`agent_tools.context` + `agent_tools`)
|
||||
- `list_tools()` — Runtime discovery of all available tools
|
||||
- `read_context_md()` — Read accumulated agent knowledge
|
||||
- `append_context_note(note)` — Write a learning to context.md
|
||||
- `list_recent_reports()` — List recently generated report files
|
||||
|
||||
## Decision Criteria
|
||||
|
||||
### When to use agent_tools vs direct code
|
||||
- **Always use `agent_tools`** for database queries, pipeline operations, and content validation
|
||||
- Only write direct Python/SQL when `agent_tools` lacks the needed capability
|
||||
- Use `list_tools()` when unsure what primitives exist
|
||||
|
||||
### When to run the pipeline
|
||||
- Data is stale (> 7 days since last motion)
|
||||
- Health checks show `healthy: false`
|
||||
- Pipeline status shows gaps or failures
|
||||
- User explicitly requests fresh data
|
||||
|
||||
### When to generate a report
|
||||
- User asks for analysis that spans multiple queries
|
||||
- Health check reveals issues that need documentation
|
||||
- Weekly/bi-weekly operational reviews
|
||||
|
||||
### When to validate content
|
||||
- After pipeline runs (automated quality gate)
|
||||
- After pipeline runs
|
||||
- When SVD labels look suspicious
|
||||
- Before publishing analysis to users
|
||||
|
||||
@@ -78,4 +79,4 @@ Before making claims about the data, check `docs/solutions/` for documented patt
|
||||
- You operate in the same trust boundary as the developer
|
||||
- You can read the full database but write only to `reports/` and `context.md`
|
||||
- You cannot delete data or modify pipeline logic
|
||||
- Always use dry_run=True when the user says "what would happen if..."
|
||||
- Always use `dry_run=True` when the user says "what would happen if..."
|
||||
|
||||
+11
-48
@@ -5,23 +5,13 @@ Import individual modules or use `list_tools()` for runtime discovery.
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from agent_tools.analysis import (
|
||||
analyze_axis_stability,
|
||||
analyze_party_shift,
|
||||
validate_svd_labels,
|
||||
)
|
||||
from agent_tools.content import (
|
||||
check_embedding_quality,
|
||||
suggest_svd_label,
|
||||
validate_layman_explanations,
|
||||
validate_motion_coverage,
|
||||
)
|
||||
from agent_tools.context import (
|
||||
append_context_note,
|
||||
build_context,
|
||||
render_context_markdown,
|
||||
list_recent_reports,
|
||||
read_context_md,
|
||||
)
|
||||
from agent_tools.database import (
|
||||
compute_party_positions_from_vectors,
|
||||
create_motion,
|
||||
delete_report,
|
||||
query_compass_positions,
|
||||
@@ -35,13 +25,9 @@ from agent_tools.database import (
|
||||
update_motion,
|
||||
)
|
||||
from agent_tools.pipeline import (
|
||||
pipeline_check_health,
|
||||
pipeline_get_logs,
|
||||
pipeline_run_full,
|
||||
pipeline_run_stage,
|
||||
pipeline_validate_output,
|
||||
)
|
||||
from agent_tools.reports import generate_report
|
||||
|
||||
__all__ = [
|
||||
# Database
|
||||
@@ -49,6 +35,7 @@ __all__ = [
|
||||
"query_votes",
|
||||
"query_svd_vectors",
|
||||
"query_party_positions",
|
||||
"compute_party_positions_from_vectors",
|
||||
"query_pipeline_status",
|
||||
"query_embeddings",
|
||||
"query_similar_motions",
|
||||
@@ -58,24 +45,10 @@ __all__ = [
|
||||
"delete_report",
|
||||
# Pipeline
|
||||
"pipeline_run_stage",
|
||||
"pipeline_run_full",
|
||||
"pipeline_check_health",
|
||||
"pipeline_get_logs",
|
||||
"pipeline_validate_output",
|
||||
# Analysis
|
||||
"analyze_party_shift",
|
||||
"analyze_axis_stability",
|
||||
"validate_svd_labels",
|
||||
# Content
|
||||
"validate_motion_coverage",
|
||||
"validate_layman_explanations",
|
||||
"suggest_svd_label",
|
||||
"check_embedding_quality",
|
||||
# Reports
|
||||
"generate_report",
|
||||
# Context
|
||||
"build_context",
|
||||
"render_context_markdown",
|
||||
"list_recent_reports",
|
||||
"read_context_md",
|
||||
"append_context_note",
|
||||
# Discovery
|
||||
"list_tools",
|
||||
@@ -92,28 +65,18 @@ def list_tools() -> list[dict[str, str]]:
|
||||
{"name": "query_votes", "signature": "query_votes(db_path, motion_id=None, party=None)", "description": "Query vote counts or individual votes."},
|
||||
{"name": "query_svd_vectors", "signature": "query_svd_vectors(db_path, window_id, entity_type='motion')", "description": "Query SVD vectors for a window and entity type."},
|
||||
{"name": "query_party_positions", "signature": "query_party_positions(db_path, window_id='current_parliament')", "description": "Query party axis positions for a window."},
|
||||
{"name": "query_pipeline_status", "signature": "query_pipeline_status(db_path)", "description": "Query pipeline freshness and coverage metrics."},
|
||||
{"name": "compute_party_positions_from_vectors", "signature": "compute_party_positions_from_vectors(db_path, window_id)", "description": "Compute party positions from MP vectors when pre-computed table is unavailable."},
|
||||
{"name": "query_pipeline_status", "signature": "query_pipeline_status(db_path)", "description": "Query pipeline freshness and coverage metrics (raw counts, no judgment)."},
|
||||
{"name": "query_embeddings", "signature": "query_embeddings(db_path, motion_id=None, model=None, limit=100)", "description": "Query text/fused embeddings."},
|
||||
{"name": "query_similar_motions", "signature": "query_similar_motions(db_path, motion_id, top_k=10)", "description": "Query similar motions from similarity cache."},
|
||||
{"name": "query_compass_positions", "signature": "query_compass_positions(db_path, window_id='current_parliament')", "description": "Query 2D compass positions for parties/MPs."},
|
||||
{"name": "create_motion", "signature": "create_motion(db_path, title, description, date, policy_area='General', voting_results='[]')", "description": "Insert a new motion into the database."},
|
||||
{"name": "update_motion", "signature": "update_motion(db_path, motion_id, **fields)", "description": "Update fields of an existing motion."},
|
||||
{"name": "delete_report", "signature": "delete_report(output_path)", "description": "Delete a generated report file."},
|
||||
{"name": "pipeline_run_stage", "signature": "pipeline_run_stage(db_path, stage, window_id, dry_run=False)", "description": "Run a single pipeline stage."},
|
||||
{"name": "pipeline_run_full", "signature": "pipeline_run_full(db_path, dry_run=False)", "description": "Run the full pipeline end-to-end."},
|
||||
{"name": "pipeline_check_health", "signature": "pipeline_check_health(db_path)", "description": "Run health checks and return report."},
|
||||
{"name": "pipeline_run_stage", "signature": "pipeline_run_stage(db_path, stage, window_id, dry_run=False)", "description": "Run a single pipeline stage (agent decides which and when)."},
|
||||
{"name": "pipeline_get_logs", "signature": "pipeline_get_logs(stage, lines=50)", "description": "Retrieve recent log output for a stage."},
|
||||
{"name": "pipeline_validate_output", "signature": "pipeline_validate_output(db_path, stage)", "description": "Validate that a stage produced expected output."},
|
||||
{"name": "analyze_party_shift", "signature": "analyze_party_shift(db_path, party, window_start, window_end)", "description": "Compute party position shift between two windows."},
|
||||
{"name": "analyze_axis_stability", "signature": "analyze_axis_stability(db_path, component, windows)", "description": "Compute axis stability across windows."},
|
||||
{"name": "validate_svd_labels", "signature": "validate_svd_labels(db_path, component)", "description": "Compare SVD theme labels to actual party positions."},
|
||||
{"name": "validate_motion_coverage", "signature": "validate_motion_coverage(db_path, start_date, end_date)", "description": "Check motion coverage for a date range."},
|
||||
{"name": "validate_layman_explanations", "signature": "validate_layman_explanations(db_path, sample_size=50)", "description": "Sample motions and check explanation quality."},
|
||||
{"name": "suggest_svd_label", "signature": "suggest_svd_label(db_path, component, top_n=10)", "description": "Suggest a label based on top/bottom motions."},
|
||||
{"name": "check_embedding_quality", "signature": "check_embedding_quality(db_path, window_id, healthy_threshold=0.8)", "description": "Check embedding coverage for a window."},
|
||||
{"name": "generate_report", "signature": "generate_report(db_path, report_type, parameters, output_path)", "description": "Generate a markdown report."},
|
||||
{"name": "build_context", "signature": "build_context(db_path)", "description": "Build runtime context dict for the agent."},
|
||||
{"name": "render_context_markdown", "signature": "render_context_markdown(db_path)", "description": "Render context as markdown for prompt injection."},
|
||||
{"name": "list_recent_reports", "signature": "list_recent_reports()", "description": "List recently generated report files."},
|
||||
{"name": "read_context_md", "signature": "read_context_md()", "description": "Read accumulated agent knowledge from context.md."},
|
||||
{"name": "append_context_note", "signature": "append_context_note(note)", "description": "Append a note to the accumulated agent knowledge."},
|
||||
{"name": "list_tools", "signature": "list_tools()", "description": "Return a list of all available agent tools."},
|
||||
]
|
||||
|
||||
+7
-167
@@ -1,170 +1,10 @@
|
||||
"""Analysis primitives for agent operation.
|
||||
|
||||
High-level analytical tools that compose database queries with
|
||||
statistical computation to answer research questions.
|
||||
NOTE: Multi-step analytical workflows (party shift, axis stability, SVD label
|
||||
validation) have been removed. Agents should compose raw database primitives
|
||||
(query_party_positions, query_svd_vectors, etc.) and perform analysis in their
|
||||
own reasoning loop.
|
||||
|
||||
This module is intentionally empty. If needed, pure computational helpers
|
||||
(without business logic) can be added here.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import logging
|
||||
from typing import Any, Dict, List, Optional
|
||||
|
||||
from agent_tools.database import query_party_positions, query_svd_vectors
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
def analyze_party_shift(
|
||||
db_path: str,
|
||||
party: str,
|
||||
window_start: str,
|
||||
window_end: str,
|
||||
metric: str = "euclidean",
|
||||
) -> Dict[str, Any]:
|
||||
"""Analyze how a party's position shifted between two windows."""
|
||||
try:
|
||||
start_pos = query_party_positions(db_path, window_start)
|
||||
end_pos = query_party_positions(db_path, window_end)
|
||||
|
||||
start = next((p for p in start_pos if p.get("party") == party), None)
|
||||
end = next((p for p in end_pos if p.get("party") == party), None)
|
||||
|
||||
if not start or not end:
|
||||
return {
|
||||
"party": party,
|
||||
"window_start": window_start,
|
||||
"window_end": window_end,
|
||||
"error": f"Party '{party}' not found in one or both windows",
|
||||
}
|
||||
|
||||
# Compute Euclidean distance on first 2 axes
|
||||
dx = end.get("axis_1", 0.0) - start.get("axis_1", 0.0)
|
||||
dy = end.get("axis_2", 0.0) - start.get("axis_2", 0.0)
|
||||
shift = (dx ** 2 + dy ** 2) ** 0.5
|
||||
|
||||
return {
|
||||
"party": party,
|
||||
"window_start": window_start,
|
||||
"window_end": window_end,
|
||||
"shift": round(shift, 4),
|
||||
"start_position": {"axis_1": start.get("axis_1"), "axis_2": start.get("axis_2")},
|
||||
"end_position": {"axis_1": end.get("axis_1"), "axis_2": end.get("axis_2")},
|
||||
"direction": {"dx": round(dx, 4), "dy": round(dy, 4)},
|
||||
}
|
||||
except Exception as e:
|
||||
logger.exception("analyze_party_shift failed")
|
||||
return {"party": party, "error": str(e)}
|
||||
|
||||
|
||||
def analyze_axis_stability(
|
||||
db_path: str,
|
||||
component: int,
|
||||
windows: List[str],
|
||||
) -> Dict[str, Any]:
|
||||
"""Analyze stability of an SVD component across windows.
|
||||
|
||||
Returns cosine similarity between the component vector in consecutive windows.
|
||||
"""
|
||||
try:
|
||||
vectors_by_window = {}
|
||||
for window in windows:
|
||||
rows = query_svd_vectors(db_path, window, entity_type="motion")
|
||||
if rows:
|
||||
vectors_by_window[window] = rows
|
||||
|
||||
if len(vectors_by_window) < 2:
|
||||
return {
|
||||
"component": component,
|
||||
"windows": windows,
|
||||
"error": "Need at least 2 windows with SVD vectors",
|
||||
}
|
||||
|
||||
# Extract component scores for each window
|
||||
# (component is 1-indexed in user-facing code, 0-indexed internally)
|
||||
idx = component - 1
|
||||
window_scores = {}
|
||||
for window, rows in vectors_by_window.items():
|
||||
scores = []
|
||||
for row in rows:
|
||||
vec = row.get("vector")
|
||||
if isinstance(vec, str):
|
||||
vec = json.loads(vec)
|
||||
if isinstance(vec, list) and idx < len(vec):
|
||||
scores.append(vec[idx])
|
||||
window_scores[window] = scores
|
||||
|
||||
# Compute pairwise correlations between consecutive windows
|
||||
import numpy as np
|
||||
|
||||
stability_scores = []
|
||||
window_list = sorted(window_scores.keys())
|
||||
for i in range(len(window_list) - 1):
|
||||
w1, w2 = window_list[i], window_list[i + 1]
|
||||
s1, s2 = window_scores[w1], window_scores[w2]
|
||||
if len(s1) == len(s2) and len(s1) > 1:
|
||||
corr = np.corrcoef(s1, s2)[0, 1]
|
||||
stability_scores.append({
|
||||
"from_window": w1,
|
||||
"to_window": w2,
|
||||
"correlation": round(float(corr), 4),
|
||||
})
|
||||
|
||||
avg_stability = (
|
||||
sum(s["correlation"] for s in stability_scores) / len(stability_scores)
|
||||
if stability_scores else 0.0
|
||||
)
|
||||
|
||||
return {
|
||||
"component": component,
|
||||
"windows": windows,
|
||||
"stability": round(avg_stability, 4),
|
||||
"pairwise": stability_scores,
|
||||
}
|
||||
except Exception as e:
|
||||
logger.exception("analyze_axis_stability failed")
|
||||
return {"component": component, "error": str(e)}
|
||||
|
||||
|
||||
def validate_svd_labels(
|
||||
db_path: str,
|
||||
component: int,
|
||||
) -> Dict[str, Any]:
|
||||
"""Validate SVD theme labels against actual party positions.
|
||||
|
||||
Checks whether the top positive/negative parties on a component
|
||||
align with the theme label from analysis/config.py.
|
||||
"""
|
||||
try:
|
||||
from analysis.config import SVD_THEMES
|
||||
|
||||
theme = SVD_THEMES.get(component, {})
|
||||
label = theme.get("label", "Unknown")
|
||||
description = theme.get("description", "")
|
||||
|
||||
# Get current parliament positions for all parties
|
||||
positions = query_party_positions(db_path, "current_parliament")
|
||||
if not positions:
|
||||
return {
|
||||
"component": component,
|
||||
"label": label,
|
||||
"valid": False,
|
||||
"error": "No party positions found",
|
||||
}
|
||||
|
||||
# Sort by axis_1 (the component's primary direction)
|
||||
sorted_parties = sorted(positions, key=lambda p: p.get("axis_1", 0.0))
|
||||
negative_pole = sorted_parties[:3] if len(sorted_parties) >= 3 else sorted_parties[:1]
|
||||
positive_pole = sorted_parties[-3:] if len(sorted_parties) >= 3 else sorted_parties[-1:]
|
||||
|
||||
return {
|
||||
"component": component,
|
||||
"label": label,
|
||||
"description": description,
|
||||
"valid": True,
|
||||
"negative_pole": [{"party": p["party"], "score": round(p.get("axis_1", 0.0), 4)} for p in negative_pole],
|
||||
"positive_pole": [{"party": p["party"], "score": round(p.get("axis_1", 0.0), 4)} for p in positive_pole],
|
||||
}
|
||||
except Exception as e:
|
||||
logger.exception("validate_svd_labels failed")
|
||||
return {"component": component, "valid": False, "error": str(e)}
|
||||
|
||||
+3
-59
@@ -7,7 +7,7 @@ from __future__ import annotations
|
||||
|
||||
import logging
|
||||
from datetime import datetime, timedelta
|
||||
from typing import Any, Dict, List, Optional
|
||||
from typing import Any, Dict
|
||||
|
||||
from agent_tools.database import query_motions, query_svd_vectors
|
||||
|
||||
@@ -105,67 +105,13 @@ def validate_layman_explanations(
|
||||
return {"sample_size": 0, "coverage": 0.0, "error": str(e)}
|
||||
|
||||
|
||||
def suggest_svd_label(
|
||||
db_path: str,
|
||||
component: int,
|
||||
top_n: int = 10,
|
||||
) -> Dict[str, Any]:
|
||||
"""Analyze top motions on a component and suggest a label.
|
||||
|
||||
Returns the top positive and negative motions with scores.
|
||||
"""
|
||||
try:
|
||||
rows = query_svd_vectors(db_path, "current_parliament", entity_type="motion")
|
||||
|
||||
if not rows:
|
||||
return {
|
||||
"component": component,
|
||||
"error": "No SVD vectors found for current_parliament",
|
||||
}
|
||||
|
||||
import json
|
||||
|
||||
scored = []
|
||||
for row in rows:
|
||||
vec = row.get("vector")
|
||||
if isinstance(vec, str):
|
||||
vec = json.loads(vec)
|
||||
if isinstance(vec, list) and component - 1 < len(vec):
|
||||
scored.append({
|
||||
"motion_id": row.get("entity_id"),
|
||||
"score": vec[component - 1],
|
||||
})
|
||||
|
||||
scored.sort(key=lambda x: x["score"])
|
||||
negative = scored[:top_n]
|
||||
positive = scored[-top_n:][::-1]
|
||||
|
||||
return {
|
||||
"component": component,
|
||||
"suggestion": {
|
||||
"negative_pole": negative,
|
||||
"positive_pole": positive,
|
||||
},
|
||||
"top_positive_ids": [m["motion_id"] for m in positive],
|
||||
"top_negative_ids": [m["motion_id"] for m in negative],
|
||||
}
|
||||
except Exception as e:
|
||||
logger.exception("suggest_svd_label failed")
|
||||
return {"component": component, "error": str(e)}
|
||||
|
||||
|
||||
def check_embedding_quality(
|
||||
db_path: str,
|
||||
window_id: str,
|
||||
healthy_threshold: float = 0.8,
|
||||
) -> Dict[str, Any]:
|
||||
"""Check embedding coverage and quality for a window.
|
||||
"""Check embedding coverage for a window.
|
||||
|
||||
Args:
|
||||
healthy_threshold: Coverage ratio above which embeddings are considered healthy.
|
||||
Defaults to 0.8; override via prompt for different quality bars.
|
||||
|
||||
Returns coverage stats for fused embeddings.
|
||||
Returns raw coverage stats. The agent decides whether coverage is acceptable.
|
||||
"""
|
||||
try:
|
||||
vectors = query_svd_vectors(db_path, window_id, entity_type="motion")
|
||||
@@ -181,8 +127,6 @@ def check_embedding_quality(
|
||||
"total_motions": total_motions,
|
||||
"with_embeddings": with_embeddings,
|
||||
"coverage": coverage,
|
||||
"healthy": coverage > healthy_threshold,
|
||||
"healthy_threshold": healthy_threshold,
|
||||
}
|
||||
except Exception as e:
|
||||
logger.exception("check_embedding_quality failed")
|
||||
|
||||
+4
-62
@@ -1,7 +1,6 @@
|
||||
"""Runtime context injection for agent operation.
|
||||
|
||||
Generates dynamic context about the current pipeline state,
|
||||
recent issues, and accumulated knowledge.
|
||||
Filesystem primitives for managing agent accumulated knowledge.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
@@ -9,69 +8,12 @@ from __future__ import annotations
|
||||
import logging
|
||||
import os
|
||||
from datetime import datetime
|
||||
from typing import Any, Dict
|
||||
|
||||
from agent_tools.database import query_pipeline_status
|
||||
from typing import List
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
def build_context(db_path: str) -> Dict[str, Any]:
|
||||
"""Build a comprehensive context dict for the agent.
|
||||
|
||||
This is injected into the agent's prompt at session start.
|
||||
"""
|
||||
status = query_pipeline_status(db_path)
|
||||
|
||||
context = {
|
||||
"timestamp": datetime.now().isoformat(),
|
||||
"database_path": db_path,
|
||||
"pipeline": status,
|
||||
"recent_reports": _list_recent_reports(),
|
||||
"accumulated_knowledge": _read_context_md(),
|
||||
}
|
||||
|
||||
return context
|
||||
|
||||
|
||||
def render_context_markdown(db_path: str) -> str:
|
||||
"""Render context as markdown for prompt injection."""
|
||||
ctx = build_context(db_path)
|
||||
|
||||
lines = [
|
||||
"## Current Pipeline State",
|
||||
f"",
|
||||
f"- **Motions:** {ctx['pipeline'].get('motion_count', 0):,}",
|
||||
f"- **Latest motion:** {ctx['pipeline'].get('latest_motion_date', 'N/A')}",
|
||||
f"- **SVD windows:** {ctx['pipeline'].get('svd_window_count', 0)}",
|
||||
f"- **Embeddings:** {ctx['pipeline'].get('embedding_count', 0):,}",
|
||||
f"- **Healthy:** {'Yes' if ctx['pipeline'].get('healthy') else 'No'}",
|
||||
f"",
|
||||
]
|
||||
|
||||
recent = ctx.get("recent_reports", [])
|
||||
if recent:
|
||||
lines.extend([
|
||||
"## Recent Reports",
|
||||
f"",
|
||||
])
|
||||
for r in recent[:5]:
|
||||
lines.append(f"- {r}")
|
||||
lines.append("")
|
||||
|
||||
knowledge = ctx.get("accumulated_knowledge", "")
|
||||
if knowledge:
|
||||
lines.extend([
|
||||
"## Accumulated Knowledge",
|
||||
f"",
|
||||
knowledge,
|
||||
f"",
|
||||
])
|
||||
|
||||
return "\n".join(lines)
|
||||
|
||||
|
||||
def _list_recent_reports() -> list:
|
||||
def list_recent_reports() -> List[str]:
|
||||
"""List recently generated reports."""
|
||||
try:
|
||||
reports_dir = "reports"
|
||||
@@ -87,7 +29,7 @@ def _list_recent_reports() -> list:
|
||||
return []
|
||||
|
||||
|
||||
def _read_context_md() -> str:
|
||||
def read_context_md() -> str:
|
||||
"""Read accumulated knowledge from context.md."""
|
||||
try:
|
||||
path = os.path.join("agent_tools", "context.md")
|
||||
|
||||
+26
-18
@@ -121,23 +121,22 @@ def query_party_positions(
|
||||
"""Query party axis scores for a window."""
|
||||
try:
|
||||
con = _connect(db_path)
|
||||
# Check if party_axis_scores table exists
|
||||
tables = con.execute(
|
||||
"SELECT table_name FROM information_schema.tables WHERE table_name = 'party_axis_scores'"
|
||||
).fetchall()
|
||||
|
||||
if tables:
|
||||
result = con.execute(
|
||||
"""
|
||||
SELECT party, axis, score
|
||||
FROM party_axis_scores
|
||||
WHERE window_id = ?
|
||||
""",
|
||||
(window_id,),
|
||||
).fetchdf().to_dict("records")
|
||||
else:
|
||||
# Fallback: compute from vectors
|
||||
result = _compute_party_positions_from_vectors(con, window_id)
|
||||
if not tables:
|
||||
con.close()
|
||||
return []
|
||||
|
||||
result = con.execute(
|
||||
"""
|
||||
SELECT party, axis, score
|
||||
FROM party_axis_scores
|
||||
WHERE window_id = ?
|
||||
""",
|
||||
(window_id,),
|
||||
).fetchdf().to_dict("records")
|
||||
con.close()
|
||||
return result
|
||||
except Exception:
|
||||
@@ -145,8 +144,17 @@ def query_party_positions(
|
||||
return []
|
||||
|
||||
|
||||
def _compute_party_positions_from_vectors(con, window_id: str) -> List[Dict[str, Any]]:
|
||||
"""Compute party positions from MP vectors when party_axis_scores doesn't exist."""
|
||||
def compute_party_positions_from_vectors(con, window_id: str) -> List[Dict[str, Any]]:
|
||||
"""Compute party positions from MP vectors.
|
||||
|
||||
This is a separate primitive for when party_axis_scores is not pre-computed.
|
||||
"""
|
||||
import duckdb
|
||||
if isinstance(con, str):
|
||||
con = duckdb.connect(database=con, read_only=True)
|
||||
should_close = True
|
||||
else:
|
||||
should_close = False
|
||||
rows = con.execute(
|
||||
"""
|
||||
SELECT sv.entity_id, sv.vector, mm.party
|
||||
@@ -169,7 +177,6 @@ def _compute_party_positions_from_vectors(con, window_id: str) -> List[Dict[str,
|
||||
for party, vectors in party_vectors.items():
|
||||
if not vectors:
|
||||
continue
|
||||
# Compute mean position across first 2 components
|
||||
dim = len(vectors[0])
|
||||
mean = [sum(v[i] for v in vectors) / len(vectors) for i in range(min(dim, 2))]
|
||||
result.append({
|
||||
@@ -178,6 +185,9 @@ def _compute_party_positions_from_vectors(con, window_id: str) -> List[Dict[str,
|
||||
"axis_2": mean[1] if len(mean) > 1 else 0.0,
|
||||
})
|
||||
|
||||
if should_close:
|
||||
con.close()
|
||||
|
||||
return result
|
||||
|
||||
|
||||
@@ -206,8 +216,6 @@ def query_pipeline_status(db_path: str) -> Dict[str, Any]:
|
||||
"latest_motion_date": str(latest_motion_date) if latest_motion_date else None,
|
||||
"svd_window_count": svd_windows,
|
||||
"embedding_count": embedding_count,
|
||||
"motion_count": motion_count,
|
||||
"svd_window_count": svd_windows,
|
||||
}
|
||||
except Exception:
|
||||
logger.exception("query_pipeline_status failed")
|
||||
|
||||
+3
-135
@@ -1,6 +1,6 @@
|
||||
"""Pipeline control primitives for agent operation.
|
||||
|
||||
Stage-aware tools for running, monitoring, and diagnosing the data pipeline.
|
||||
Thin execution wrappers. The agent decides which stages to run and in what order.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
@@ -8,12 +8,8 @@ from __future__ import annotations
|
||||
import logging
|
||||
from typing import Any, Dict, List, Optional
|
||||
|
||||
from agent_tools.database import query_pipeline_status
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
VALID_STAGES = {"ingestion", "votes", "svd", "text_embeddings", "fusion", "similarity"}
|
||||
|
||||
|
||||
def pipeline_run_stage(
|
||||
db_path: str,
|
||||
@@ -25,18 +21,13 @@ def pipeline_run_stage(
|
||||
|
||||
Args:
|
||||
db_path: Path to DuckDB database
|
||||
stage: One of VALID_STAGES
|
||||
window_id: Optional window identifier (e.g., "2024", "current_parliament")
|
||||
stage: Pipeline stage name (e.g. "ingestion", "svd", "similarity")
|
||||
window_id: Optional window identifier (e.g. "2024", "current_parliament")
|
||||
dry_run: If True, return planned actions without executing
|
||||
|
||||
Returns:
|
||||
dict with status and metadata
|
||||
"""
|
||||
if stage not in VALID_STAGES:
|
||||
return {
|
||||
"error": f"Invalid stage '{stage}'. Valid stages: {sorted(VALID_STAGES)}",
|
||||
}
|
||||
|
||||
result = {
|
||||
"stage": stage,
|
||||
"window_id": window_id,
|
||||
@@ -53,86 +44,6 @@ def pipeline_run_stage(
|
||||
return result
|
||||
|
||||
|
||||
def pipeline_run_full(
|
||||
db_path: str,
|
||||
dry_run: bool = False,
|
||||
) -> Dict[str, Any]:
|
||||
"""Run all pipeline stages in dependency order.
|
||||
|
||||
Args:
|
||||
db_path: Path to DuckDB database
|
||||
dry_run: If True, return planned actions without executing
|
||||
|
||||
Returns:
|
||||
dict with stage statuses
|
||||
"""
|
||||
stages = ["ingestion", "votes", "svd", "text_embeddings", "fusion", "similarity"]
|
||||
results = []
|
||||
|
||||
for stage in stages:
|
||||
result = pipeline_run_stage(db_path, stage, dry_run=dry_run)
|
||||
results.append(result)
|
||||
|
||||
return {
|
||||
"stages": results,
|
||||
"dry_run": dry_run,
|
||||
"status": "planned" if dry_run else "partial",
|
||||
}
|
||||
|
||||
|
||||
def pipeline_check_health(db_path: str) -> Dict[str, Any]:
|
||||
"""Check pipeline health and return structured report.
|
||||
|
||||
Reuses the health/ module and database queries.
|
||||
"""
|
||||
try:
|
||||
from health.checks import check_motion_freshness, check_embedding_coverage
|
||||
|
||||
checks = []
|
||||
healthy = True
|
||||
|
||||
try:
|
||||
freshness = check_motion_freshness(db_path)
|
||||
checks.append({
|
||||
"name": "motion_freshness",
|
||||
"healthy": freshness.get("healthy", False),
|
||||
"details": freshness,
|
||||
})
|
||||
if not freshness.get("healthy", False):
|
||||
healthy = False
|
||||
except Exception as e:
|
||||
checks.append({"name": "motion_freshness", "healthy": False, "error": str(e)})
|
||||
healthy = False
|
||||
|
||||
try:
|
||||
embedding = check_embedding_coverage(db_path)
|
||||
checks.append({
|
||||
"name": "embedding_coverage",
|
||||
"healthy": embedding.get("healthy", False),
|
||||
"details": embedding,
|
||||
})
|
||||
if not embedding.get("healthy", False):
|
||||
healthy = False
|
||||
except Exception as e:
|
||||
checks.append({"name": "embedding_coverage", "healthy": False, "error": str(e)})
|
||||
healthy = False
|
||||
|
||||
status = query_pipeline_status(db_path)
|
||||
|
||||
return {
|
||||
"healthy": healthy,
|
||||
"checks": checks,
|
||||
"pipeline_status": status,
|
||||
}
|
||||
except Exception as e:
|
||||
logger.exception("pipeline_check_health failed")
|
||||
return {
|
||||
"healthy": False,
|
||||
"checks": [],
|
||||
"error": str(e),
|
||||
}
|
||||
|
||||
|
||||
def pipeline_get_logs(
|
||||
db_path: str,
|
||||
stage: Optional[str] = None,
|
||||
@@ -147,46 +58,3 @@ def pipeline_get_logs(
|
||||
# Real implementation would read from logging infrastructure
|
||||
logger.info("pipeline_get_logs requested for stage=%s lines=%d", stage, lines)
|
||||
return []
|
||||
|
||||
|
||||
def pipeline_validate_output(
|
||||
db_path: str,
|
||||
stage: str,
|
||||
) -> Dict[str, Any]:
|
||||
"""Validate that a stage's output looks reasonable.
|
||||
|
||||
Args:
|
||||
db_path: Path to DuckDB database
|
||||
stage: Pipeline stage to validate
|
||||
|
||||
Returns:
|
||||
dict with validation results
|
||||
"""
|
||||
if stage not in VALID_STAGES:
|
||||
return {
|
||||
"valid": False,
|
||||
"error": f"Invalid stage '{stage}'",
|
||||
}
|
||||
|
||||
try:
|
||||
status = query_pipeline_status(db_path)
|
||||
|
||||
validators = {
|
||||
"svd": lambda s: s.get("svd_window_count", 0) > 0,
|
||||
"similarity": lambda s: s.get("embedding_count", 0) > 0,
|
||||
"ingestion": lambda s: s.get("motion_count", 0) > 0,
|
||||
"votes": lambda s: s.get("motion_count", 0) > 0,
|
||||
"text_embeddings": lambda s: s.get("embedding_count", 0) > 0,
|
||||
"fusion": lambda s: s.get("embedding_count", 0) > 0,
|
||||
}
|
||||
|
||||
is_valid = validators.get(stage, lambda s: False)(status)
|
||||
|
||||
return {
|
||||
"valid": is_valid,
|
||||
"stage": stage,
|
||||
"pipeline_status": status,
|
||||
}
|
||||
except Exception as e:
|
||||
logger.exception("pipeline_validate_output failed")
|
||||
return {"valid": False, "stage": stage, "error": str(e)}
|
||||
|
||||
+5
-146
@@ -1,149 +1,8 @@
|
||||
"""Report generation primitives for agent operation.
|
||||
|
||||
Agents call these to write structured markdown reports to the reports/ directory.
|
||||
NOTE: The report template engine (generate_report, _render_report) has been
|
||||
removed. Agents should compose markdown in their reasoning loop and write it
|
||||
directly using standard file I/O.
|
||||
|
||||
This module is intentionally empty.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
import os
|
||||
from datetime import datetime
|
||||
from typing import Any, Dict
|
||||
|
||||
from agent_tools.database import query_pipeline_status
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
REPORT_TYPES = {
|
||||
"summary",
|
||||
"health",
|
||||
"party_shift",
|
||||
"axis_stability",
|
||||
}
|
||||
|
||||
|
||||
def generate_report(
|
||||
db_path: str,
|
||||
*,
|
||||
report_type: str,
|
||||
parameters: Dict[str, Any],
|
||||
output_path: str,
|
||||
) -> Dict[str, Any]:
|
||||
"""Generate a markdown report and write it to output_path.
|
||||
|
||||
Args:
|
||||
db_path: Path to DuckDB database
|
||||
report_type: One of REPORT_TYPES
|
||||
parameters: Type-specific parameters
|
||||
output_path: Where to write the markdown file
|
||||
|
||||
Returns:
|
||||
dict with "output_path" and "status" keys, or "error" on failure
|
||||
"""
|
||||
if report_type not in REPORT_TYPES:
|
||||
return {
|
||||
"error": f"Unknown report type '{report_type}'. Known types: {sorted(REPORT_TYPES)}",
|
||||
}
|
||||
|
||||
try:
|
||||
content = _render_report(db_path, report_type, parameters)
|
||||
os.makedirs(os.path.dirname(output_path) or ".", exist_ok=True)
|
||||
with open(output_path, "w", encoding="utf-8") as f:
|
||||
f.write(content)
|
||||
return {"output_path": output_path, "status": "written"}
|
||||
except Exception as e:
|
||||
logger.exception("generate_report failed")
|
||||
return {"error": str(e)}
|
||||
|
||||
|
||||
def _render_report(db_path: str, report_type: str, parameters: Dict[str, Any]) -> str:
|
||||
"""Render report content as markdown."""
|
||||
lines = [
|
||||
f"# Stemwijzer Report: {report_type.replace('_', ' ').title()}",
|
||||
f"",
|
||||
f"Generated: {datetime.now().isoformat()}",
|
||||
f"",
|
||||
]
|
||||
|
||||
if report_type == "summary":
|
||||
status = query_pipeline_status(db_path)
|
||||
lines.extend([
|
||||
"## Pipeline Summary",
|
||||
f"",
|
||||
f"- **Motions in database:** {status.get('motion_count', 0):,}",
|
||||
f"- **Latest motion date:** {status.get('latest_motion_date', 'N/A')}",
|
||||
f"- **SVD windows computed:** {status.get('svd_window_count', 0)}",
|
||||
f"- **Motion embeddings:** {status.get('embedding_count', 0):,}",
|
||||
f"- **Overall health:** {'✅ Healthy' if status.get('healthy') else '⚠️ Needs attention'}",
|
||||
f"",
|
||||
])
|
||||
|
||||
elif report_type == "health":
|
||||
status = query_pipeline_status(db_path)
|
||||
lines.extend([
|
||||
"## Pipeline Health Check",
|
||||
f"",
|
||||
f"| Metric | Value | Status |",
|
||||
f"|--------|-------|--------|",
|
||||
f"| Motion count | {status.get('motion_count', 0):,} | {'✅' if status.get('motion_count', 0) > 0 else '⚠️'} |",
|
||||
f"| Latest motion | {status.get('latest_motion_date', 'N/A')} | {'✅' if status.get('latest_motion_date') else '⚠️'} |",
|
||||
f"| SVD windows | {status.get('svd_window_count', 0)} | {'✅' if status.get('svd_window_count', 0) > 0 else '⚠️'} |",
|
||||
f"| Embeddings | {status.get('embedding_count', 0):,} | {'✅' if status.get('embedding_count', 0) > 0 else '⚠️'} |",
|
||||
f"",
|
||||
])
|
||||
|
||||
elif report_type == "party_shift":
|
||||
from agent_tools.analysis import analyze_party_shift
|
||||
|
||||
party = parameters.get("party", "VVD")
|
||||
start = parameters.get("window_start", "2020")
|
||||
end = parameters.get("window_end", "2024")
|
||||
result = analyze_party_shift(db_path, party, start, end)
|
||||
|
||||
if "error" in result:
|
||||
lines.extend(["## Party Shift Analysis", f"", f"Error: {result['error']}", f""])
|
||||
else:
|
||||
lines.extend([
|
||||
"## Party Shift Analysis",
|
||||
f"",
|
||||
f"**Party:** {result['party']}",
|
||||
f"**Period:** {result['window_start']} → {result['window_end']}",
|
||||
f"**Shift magnitude:** {result['shift']}",
|
||||
f"**Direction:** dx={result['direction']['dx']}, dy={result['direction']['dy']}",
|
||||
f"",
|
||||
f"### Start position",
|
||||
f"- Axis 1: {result['start_position']['axis_1']}",
|
||||
f"- Axis 2: {result['start_position']['axis_2']}",
|
||||
f"",
|
||||
f"### End position",
|
||||
f"- Axis 1: {result['end_position']['axis_1']}",
|
||||
f"- Axis 2: {result['end_position']['axis_2']}",
|
||||
f"",
|
||||
])
|
||||
|
||||
elif report_type == "axis_stability":
|
||||
from agent_tools.analysis import analyze_axis_stability
|
||||
|
||||
component = parameters.get("component", 1)
|
||||
windows = parameters.get("windows", ["2020", "2021", "2022", "2023", "2024"])
|
||||
result = analyze_axis_stability(db_path, component, windows)
|
||||
|
||||
if "error" in result:
|
||||
lines.extend(["## Axis Stability Analysis", f"", f"Error: {result['error']}", f""])
|
||||
else:
|
||||
lines.extend([
|
||||
"## Axis Stability Analysis",
|
||||
f"",
|
||||
f"**Component:** {result['component']}",
|
||||
f"**Average stability:** {result['stability']}",
|
||||
f"",
|
||||
f"### Pairwise correlations",
|
||||
f"",
|
||||
f"| From | To | Correlation |",
|
||||
f"|------|-----|-------------|",
|
||||
])
|
||||
for pair in result.get("pairwise", []):
|
||||
lines.append(f"| {pair['from_window']} | {pair['to_window']} | {pair['correlation']} |")
|
||||
lines.append("")
|
||||
|
||||
return "\n".join(lines)
|
||||
|
||||
Reference in New Issue
Block a user