chore: add .mindmodel/ project constraints and conventions
Generated mindmodel covering stack, architecture, domain glossary, coding conventions, DuckDB/requests/embeddings/error-handling patterns, anti-patterns, and explicit constraints for naming, imports, DB access, error handling, and testing.
This commit is contained in:
@@ -0,0 +1,70 @@
|
||||
name: duckdb_access
|
||||
|
||||
rules:
|
||||
- Prefer using read_only=True for compute-only subprocesses (e.g., SVD compute) to allow concurrent readers.
|
||||
- Prefer "with duckdb.connect(db_path, read_only=True) as conn" for scoped connections so conn.close() is automatic.
|
||||
- If a long-lived connection is created at module level, provide explicit close() or ensure operation is safe for Streamlit's lifecycle.
|
||||
- Prefer parameterizing db_path in pipelines and creating connections locally (avoid global connections that cross threads).
|
||||
|
||||
examples:
|
||||
- path: database.py
|
||||
excerpt: |
|
||||
```python
|
||||
conn = duckdb.connect(self.db_path)
|
||||
...
|
||||
conn.execute("""
|
||||
CREATE TABLE IF NOT EXISTS fused_embeddings (
|
||||
id INTEGER DEFAULT nextval('fused_embeddings_id_seq'),
|
||||
motion_id INTEGER NOT NULL,
|
||||
window_id TEXT NOT NULL,
|
||||
vector JSON NOT NULL,
|
||||
svd_dims INTEGER NOT NULL,
|
||||
text_dims INTEGER NOT NULL,
|
||||
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
|
||||
PRIMARY KEY (id)
|
||||
)
|
||||
""")
|
||||
conn.close()
|
||||
```
|
||||
note: explicit connect/close used when initializing schema
|
||||
|
||||
- path: pipeline/svd_pipeline.py
|
||||
excerpt: |
|
||||
```python
|
||||
conn = duckdb.connect(db_path, read_only=True)
|
||||
try:
|
||||
rows = conn.execute(
|
||||
"SELECT motion_id, mp_name, vote FROM mp_votes WHERE date BETWEEN ? AND ?",
|
||||
(start_date, end_date),
|
||||
).fetchall()
|
||||
finally:
|
||||
conn.close()
|
||||
```
|
||||
note: read_only connection used for compute-heavy worker
|
||||
|
||||
- path: similarity/compute.py
|
||||
excerpt: |
|
||||
```python
|
||||
try:
|
||||
import duckdb
|
||||
except Exception:
|
||||
logger.exception("duckdb import failed; cannot load vectors")
|
||||
return 0
|
||||
|
||||
with duckdb.connect(db.db_path) as conn:
|
||||
rows = conn.execute(query, params).fetchall()
|
||||
```
|
||||
note: preferred 'with' context for automatic close
|
||||
|
||||
anti_patterns:
|
||||
- Bad: creating a connection without closure in a long-running process
|
||||
remediation: use "with" context or ensure conn.close() in finally block
|
||||
example: |
|
||||
```python
|
||||
# BAD: connection may leak if exception occurs before explicit close
|
||||
conn = duckdb.connect(db_path)
|
||||
rows = conn.execute("SELECT ...").fetchall()
|
||||
# missing finally/close
|
||||
```
|
||||
- Bad: Opening write connections from many parallel workers without coordination
|
||||
remediation: open read_only for compute processes and centralize writes via short-lived connections or a single writer worker.
|
||||
@@ -0,0 +1,63 @@
|
||||
name: embeddings_similarity_pipeline
|
||||
|
||||
rules:
|
||||
- Keep embedding calls batched where possible; fallback to per-item attempts on persistent batch failure.
|
||||
- Store raw embeddings, SVD vectors, and fused_embeddings separately; fused_embeddings are typically concatenation [svd + text].
|
||||
- Compute similarity as normalized cosine on padded vectors; record top-k neighbors in similarity_cache.
|
||||
- Use read_only DuckDB connections in compute workers to allow parallel runs.
|
||||
|
||||
examples:
|
||||
- path: pipeline/ai_provider_wrapper.py
|
||||
excerpt: |
|
||||
```python
|
||||
for start in range(0, len(texts), batch_size):
|
||||
chunk = texts[start : start + batch_size]
|
||||
resp = _post_with_retries("/embeddings", json={"model": model, "input": chunk})
|
||||
...
|
||||
for j in range(i, end):
|
||||
t = texts[j]
|
||||
single, single_exc = _attempt_batch([t], j)
|
||||
if single:
|
||||
results[j] = single[0]
|
||||
```
|
||||
note: batched embed + fallback per-item retry
|
||||
|
||||
- path: pipeline/fusion.py
|
||||
excerpt: |
|
||||
```python
|
||||
try:
|
||||
svd_vec = json.loads(svd_json)
|
||||
except Exception:
|
||||
_logger.exception("Invalid SVD vector JSON for entity %s", entity_id)
|
||||
skipped_missing_svd += 1
|
||||
continue
|
||||
...
|
||||
fused = list(svd_vec) + list(text_vec)
|
||||
res = db.store_fused_embedding(
|
||||
int(entity_id),
|
||||
window_id,
|
||||
fused,
|
||||
svd_dims=len(svd_vec),
|
||||
text_dims=len(text_vec),
|
||||
)
|
||||
```
|
||||
note: concatenation of vectors and storage via MotionDatabase
|
||||
|
||||
- path: similarity/compute.py
|
||||
excerpt: |
|
||||
```python
|
||||
# Normalize rows
|
||||
norms = np.linalg.norm(matrix, axis=1, keepdims=True)
|
||||
norms[norms == 0] = 1.0
|
||||
normalized = matrix / norms
|
||||
sim = normalized @ normalized.T
|
||||
...
|
||||
# pick top-k neighbors and write to similarity_cache
|
||||
```
|
||||
note: numeric pipeline and padding to consistent dimensionality
|
||||
|
||||
anti_patterns:
|
||||
- Bad: Assuming consistent vector length without checks (leads to shape errors).
|
||||
remediation: Detect inconsistent lengths, pad with zeros, and log a warning (as seen in compute.py).
|
||||
- Bad: Recomputing heavy pipelines inline in UI requests.
|
||||
remediation: schedule heavy work in scripts/subprocesses and read precomputed results in UI.
|
||||
@@ -0,0 +1,54 @@
|
||||
name: error_handling
|
||||
|
||||
rules:
|
||||
- Use explicit exceptions for domain/error classification (e.g., ProviderError, ValueError).
|
||||
- Prefer logging.exception when catching an exception where stack trace is useful.
|
||||
- Avoid broad except: clauses that swallow exceptions; if broad except is used for "best-effort" fallback, log at warning and include original exception context.
|
||||
- For public library-like functions, prefer raising typed exceptions instead of returning magic values ([], False) — only return safe defaults where documented.
|
||||
|
||||
examples:
|
||||
- path: ai_provider.py
|
||||
excerpt: |
|
||||
```python
|
||||
except requests.ConnectionError as exc:
|
||||
if attempt == retries:
|
||||
raise ProviderError(
|
||||
f"Connection error when calling provider: {exc}"
|
||||
) from exc
|
||||
...
|
||||
```
|
||||
note: mapping network error to ProviderError with re-raise chaining
|
||||
|
||||
- path: pipeline/ai_provider_wrapper.py
|
||||
excerpt: |
|
||||
```python
|
||||
except Exception:
|
||||
_logger.exception("Failed to append audit event for embedding failure")
|
||||
results[j] = None
|
||||
```
|
||||
note: logs and assigns None for failure; fallback behavior documented earlier in wrapper rule
|
||||
|
||||
- path: similarity/compute.py
|
||||
excerpt: |
|
||||
```python
|
||||
try:
|
||||
import duckdb
|
||||
except Exception:
|
||||
logger.exception("duckdb import failed; cannot load vectors")
|
||||
return 0
|
||||
```
|
||||
note: defensive import handling and early return on failure
|
||||
|
||||
anti_patterns:
|
||||
- Bad: Broad except without logging and without re-raising (silently hides bugs)
|
||||
remediation: Narrow exception types or at minimum log.exception() and re-raise or convert to a domain error if truly handled.
|
||||
example: |
|
||||
```python
|
||||
try:
|
||||
do_work()
|
||||
except Exception:
|
||||
return []
|
||||
# BAD: hides the root cause and returns an ambiguous default
|
||||
```
|
||||
- Bad: Mixing print() and logging for errors
|
||||
remediation: Replace print() calls with logger.* calls; use structured logging configuration.
|
||||
@@ -0,0 +1,33 @@
|
||||
name: module_singletons
|
||||
|
||||
rules:
|
||||
- Module-level singletons (e.g., db = MotionDatabase()) are acceptable but should be created carefully:
|
||||
- Avoid expensive initialization at import time.
|
||||
- Provide a way to construct with a test DB path or to reinitialize in tests.
|
||||
- If a singleton holds resources (DB connections, sessions), ensure safe shutdown on program exit.
|
||||
|
||||
examples:
|
||||
- path: database.py
|
||||
excerpt: |
|
||||
```python
|
||||
class MotionDatabase:
|
||||
def __init__(self, db_path: str = config.DATABASE_PATH):
|
||||
self.db_path = db_path
|
||||
# If duckdb is not available, operate in lightweight file-backed mode
|
||||
self._file_mode = duckdb is None
|
||||
self._init_database()
|
||||
```
|
||||
note: class is safe to instantiate and creates DB at init; consider lazy init if heavy
|
||||
|
||||
- path: similarity/lookup.py
|
||||
excerpt: |
|
||||
```python
|
||||
db = MotionDatabase(db_path=db_path) if db_path else MotionDatabase()
|
||||
if hasattr(db, "get_cached_similarities"):
|
||||
rows = db.get_cached_similarities(...)
|
||||
```
|
||||
note: consumers create local MotionDatabase instances, not relying on a single global
|
||||
|
||||
anti_patterns:
|
||||
- Bad: Creating connections and performing heavy schema migrations during import
|
||||
remediation: Move heavy init to an explicit initialize() method and keep import fast.
|
||||
@@ -0,0 +1,65 @@
|
||||
name: requests_http
|
||||
|
||||
rules:
|
||||
- Reuse requests.Session when making multiple calls to the same host to benefit from connection pooling.
|
||||
- Wrap outbound HTTP calls with retry/backoff logic and respect Retry-After on 429.
|
||||
- Treat 5xx as transient and retry; surface 4xx as configuration/client errors (do not retry unless 429).
|
||||
- Raise or wrap non-OK responses into domain ProviderError to make behavior consistent across the codebase.
|
||||
|
||||
examples:
|
||||
- path: ai_provider.py
|
||||
excerpt: |
|
||||
```python
|
||||
resp = requests.post(url, json=json, headers=headers, timeout=10)
|
||||
...
|
||||
if getattr(resp, "status_code", 0) == 429:
|
||||
if attempt == retries:
|
||||
raise ProviderError(f"Provider returned HTTP {resp.status_code}")
|
||||
retry_after = None
|
||||
raw = resp.headers.get("Retry-After") if getattr(resp, "headers", None) else None
|
||||
if raw:
|
||||
try:
|
||||
retry_after = int(raw)
|
||||
except Exception:
|
||||
...
|
||||
if retry_after is not None:
|
||||
time.sleep(retry_after)
|
||||
continue
|
||||
```
|
||||
note: explicit handling of 429 and Retry-After
|
||||
|
||||
- path: api_client.py
|
||||
excerpt: |
|
||||
```python
|
||||
response = self.session.get(
|
||||
base_url, params=params, timeout=config.API_TIMEOUT
|
||||
)
|
||||
response.raise_for_status()
|
||||
data = response.json()
|
||||
```
|
||||
note: uses session + raise_for_status() to surface HTTP errors
|
||||
|
||||
- path: pipeline/ai_provider_wrapper.py
|
||||
excerpt: |
|
||||
```python
|
||||
def _attempt_batch(chunk_texts, start_index):
|
||||
backoff = 0.5
|
||||
for attempt in range(1, retries + 1):
|
||||
try:
|
||||
emb_chunk = _embedder(
|
||||
chunk_texts, model=model, batch_size=len(chunk_texts)
|
||||
)
|
||||
return emb_chunk, None
|
||||
except Exception as exc:
|
||||
if attempt == retries:
|
||||
break
|
||||
sleep = backoff * (2 ** (attempt - 1))
|
||||
time.sleep(sleep)
|
||||
continue
|
||||
```
|
||||
note: wrapper adds retry/backoff and per-item fallback
|
||||
|
||||
anti_patterns:
|
||||
- Bad: Blindly catching all requests exceptions and returning empty response
|
||||
remediation: map network exceptions to retryable vs terminal (ProviderError) and log details.
|
||||
- Bad: Using print() for network errors instead of structured logging (see api_client.py where print() is used; prefer logging).
|
||||
@@ -0,0 +1,29 @@
|
||||
name: validation
|
||||
|
||||
rules:
|
||||
- Validate inputs early and raise ValueError or domain-specific exceptions (ProviderError) for invalid contract inputs.
|
||||
- Tests should assert that invalid inputs raise the expected exceptions.
|
||||
- Use explicit checks for types and shapes on public APIs (e.g., ensure text is str before embedding).
|
||||
|
||||
examples:
|
||||
- path: ai_provider.py
|
||||
excerpt: |
|
||||
```python
|
||||
if not isinstance(text, str):
|
||||
raise ProviderError("text must be a string")
|
||||
```
|
||||
note: explicit type validation before network call
|
||||
|
||||
- path: pipeline/ai_provider_wrapper.py
|
||||
excerpt: |
|
||||
```python
|
||||
if not texts:
|
||||
return []
|
||||
if motion_ids is None:
|
||||
motion_ids = [None for _ in texts]
|
||||
```
|
||||
note: defensive handling of empty inputs
|
||||
|
||||
anti_patterns:
|
||||
- Bad: Allowing invalid values to propagate into heavy computation (e.g., non-string into embedding pipeline).
|
||||
remediation: Fail fast with a typed exception and add unit tests to cover validations.
|
||||
Reference in New Issue
Block a user