DEBUG OBSERVABILITY: live error feed + auto-triage bot + dashboard

PHASE 1 — DEBUG mode:
- /etc/systemd/system/pgz-sport.service.d/debug.conf: DEBUG=1, LOG_LEVEL=DEBUG, PYTHONUNBUFFERED=1, UVICORN_LOG_LEVEL=debug

PHASE 2 — Error stream:
- /opt/pgz-sport/scripts/debug_tail.sh: tail journalctl + nginx → /var/log/pgz-sport-debug/{stream,errors}.jsonl
- pgz-debug-tail.service (always restart, multiplexes 4 sources)

PHASE 3 — Auto-triage bot:
- /opt/pgz-sport/scripts/auto_triage.py: classifies errors, dispatches CC agents
- Patterns: 5xx spike → CC4, 401/403 spike → CC2, 4xx API → CC3, ImportError/DB → CC4
- Rate limit: 6 telegram/5min
- Records decisions in triage_decisions.jsonl
- pgz-auto-triage.service

PHASE 4 — Live dashboard:
- routers/debug_router.py mounted in pgz_sport_api
- GET /api/debug/health — services + DB + error count
- GET /api/debug/errors?limit=N — last N errors (JSON)
- GET /api/debug/decisions — auto-fix decisions
- GET /api/debug/stream — full log tail
- GET /api/debug/dashboard — live HTML refresh 5s

Damir admin tier dashboard: https://sport.rinet.one/sport/api/debug/dashboard
This commit is contained in:
2026-05-05 08:46:09 +02:00
parent 7adcec3309
commit 63ca005b6e
9 changed files with 861 additions and 16 deletions
+11 -2
View File
@@ -1322,8 +1322,17 @@ def _apply_to_db(kind: str, eid: int, fields: dict, sources: list, user_email: O
params.append(json.dumps(meta_in, ensure_ascii=False, default=str))
params.append(eid)
cur.execute(f"UPDATE {table} SET {', '.join(sets)} WHERE id=%s RETURNING *", params)
after = dict(cur.fetchone())
try:
cur.execute(f"UPDATE {table} SET {', '.join(sets)} WHERE id=%s RETURNING *", params)
after = dict(cur.fetchone())
except psycopg2.errors.UniqueViolation as _uve:
# Race condition — fetch existing row instead
conn.rollback()
cur.execute(f"SELECT * FROM {table} WHERE id=%s", (eid,))
row = cur.fetchone()
after = dict(row) if row else {}
import logging as _lg
_lg.getLogger("enrich").info(f"UniqueViolation race avoided table={table} id={eid}")
cur.execute(
"""INSERT INTO pgz_sport.enrichment_log