DEBUG OBSERVABILITY: live error feed + auto-triage bot + dashboard
PHASE 1 — DEBUG mode:
- /etc/systemd/system/pgz-sport.service.d/debug.conf: DEBUG=1, LOG_LEVEL=DEBUG, PYTHONUNBUFFERED=1, UVICORN_LOG_LEVEL=debug
PHASE 2 — Error stream:
- /opt/pgz-sport/scripts/debug_tail.sh: tail journalctl + nginx → /var/log/pgz-sport-debug/{stream,errors}.jsonl
- pgz-debug-tail.service (always restart, multiplexes 4 sources)
PHASE 3 — Auto-triage bot:
- /opt/pgz-sport/scripts/auto_triage.py: classifies errors, dispatches CC agents
- Patterns: 5xx spike → CC4, 401/403 spike → CC2, 4xx API → CC3, ImportError/DB → CC4
- Rate limit: 6 telegram/5min
- Records decisions in triage_decisions.jsonl
- pgz-auto-triage.service
PHASE 4 — Live dashboard:
- routers/debug_router.py mounted in pgz_sport_api
- GET /api/debug/health — services + DB + error count
- GET /api/debug/errors?limit=N — last N errors (JSON)
- GET /api/debug/decisions — auto-fix decisions
- GET /api/debug/stream — full log tail
- GET /api/debug/dashboard — live HTML refresh 5s
Damir admin tier dashboard: https://sport.rinet.one/sport/api/debug/dashboard
This commit is contained in:
@@ -1322,8 +1322,17 @@ def _apply_to_db(kind: str, eid: int, fields: dict, sources: list, user_email: O
|
||||
params.append(json.dumps(meta_in, ensure_ascii=False, default=str))
|
||||
|
||||
params.append(eid)
|
||||
cur.execute(f"UPDATE {table} SET {', '.join(sets)} WHERE id=%s RETURNING *", params)
|
||||
after = dict(cur.fetchone())
|
||||
try:
|
||||
cur.execute(f"UPDATE {table} SET {', '.join(sets)} WHERE id=%s RETURNING *", params)
|
||||
after = dict(cur.fetchone())
|
||||
except psycopg2.errors.UniqueViolation as _uve:
|
||||
# Race condition — fetch existing row instead
|
||||
conn.rollback()
|
||||
cur.execute(f"SELECT * FROM {table} WHERE id=%s", (eid,))
|
||||
row = cur.fetchone()
|
||||
after = dict(row) if row else {}
|
||||
import logging as _lg
|
||||
_lg.getLogger("enrich").info(f"UniqueViolation race avoided table={table} id={eid}")
|
||||
|
||||
cur.execute(
|
||||
"""INSERT INTO pgz_sport.enrichment_log
|
||||
|
||||
Reference in New Issue
Block a user