devpulse: watchdog/feedback README rewrite (was stale, missed everything that tripped VERA) + fix thread-unsafe watchdog test + true up branch-prompt timeout. README (last touched 06-23) predated the owner gate and Monitor-tool wake: now documents owner-only gating (seated owner:true, doctor --fix repairs), the 3-step wake mechanic (dispatch -> arm via Monitor TOOL -> Monitor return IS the wake; '1 monitor' IS the indicator), why passive wake-back can NEVER reach an interactive session (BLOCKED line = by design), cross-project @target resolution, 600s default + --timeout, stall events; feedback re-documented as THE owner-to-owner cross-project channel (Patrick ruling: cross-project comms impossible by design except feedback). TEST FIX: test_watchdog_agent _fake_clock_sleep patched GLOBAL time.sleep with a stateful fake -> prax logger's 3 daemon threads raced the fake clock forward and unlinked the fixture lock before watch_agent read it (nondeterministic 'no active lock' + uptime-sized elapsed; failed 4x today, passed in the same-day full-repo sweep). Fix: thread-scope the fake via caller-frame check (only agent-module sleeps advance the clock; foreign threads get a real 1ms sleep). Verified 17 pass 3x deterministic, devpulse suite 407 green. Branch prompt: watchdog default timeout claim corrected 1800s -> 600s (hit live: 3 watchdogs expired mid-build at ~600s).

This commit is contained in:
AIOSAI
2026-07-11 17:58:18 -07:00
parent d511576fc0
commit 766d697e08
3 changed files with 59 additions and 9 deletions
@@ -86,7 +86,7 @@ drone @flow list open # active plans
# Watchdog # Watchdog
Devpulse module. After dispatch, arm as a background task — it polls the dispatch lock and exits when the agent finishes. Resolves @target → branch path → `.ai_mail.local/.dispatch.lock`. Default timeout 1800s; `drone @devpulse watchdog --help` for the full reference. Devpulse module. After dispatch, arm as a background task — it polls the dispatch lock and exits when the agent finishes. Resolves @target → branch path → `.ai_mail.local/.dispatch.lock`. Default timeout **600s** — pass `--timeout <s>` for longer builds (verified live S300; `drone @devpulse watchdog --help` for the full reference).
``` ```
drone @ai_mail dispatch @target "Subject" "Body" drone @ai_mail dispatch @target "Subject" "Body"
+40 -6
View File
@@ -44,7 +44,7 @@ src/aipass/devpulse/
│ │ └── watchdog/ # Agent, timer, schedule, registry │ │ └── watchdog/ # Agent, timer, schedule, registry
│ └── plugins/ # Plugin extension point │ └── plugins/ # Plugin extension point
├── devpulse_json/ # JSON handler storage (config, data, logs per module) ├── devpulse_json/ # JSON handler storage (config, data, logs per module)
├── tests/ # 282 tests ├── tests/ # 407 tests
├── artifacts/ # Birth certificate, reports ├── artifacts/ # Birth certificate, reports
├── dropbox/ # Received files, archived plans, install audit ├── dropbox/ # Received files, archived plans, install audit
├── docs/ # Transition notes ├── docs/ # Transition notes
@@ -55,11 +55,38 @@ src/aipass/devpulse/
All commands via `drone @devpulse <command>`: All commands via `drone @devpulse <command>`:
### Watchdog — directed wake system ### Watchdog — directed wake system (owner-only)
**Who may call it:** the project OWNER only — the first agent, seated as `owner: true`
in the project's sealed `*_REGISTRY.json`. Portable: `@devpulse` in AIPass, `@vera` in
Vera Studio, whoever owns elsewhere. A refusal means your project's owner isn't seated —
run `aipass doctor` to see why and `aipass doctor --fix` to repair (DPLAN-0239).
**How the wake works (read this once, save a debugging session):**
1. `drone @ai_mail dispatch @target "Subject" "Body"` — hand off the work.
2. **Immediately arm the watchdog via the harness Monitor TOOL** — never Bash
`run_in_background` (its output goes nowhere and cannot wake you):
`drone @devpulse watchdog agent @target --timeout 600`
3. The status line shows **"1 monitor"** the moment it's armed — that IS the
active-dispatch indicator. When `@target` finishes, the watchdog exits, the
Monitor completes, and **your session is re-invoked with the result — that IS
the wake.**
There is no passive wake: ai_mail's wake-back spawns a new headless process and can
never inject into a live interactive session (`BLOCKED — interactive session` in the
logs is that guard working as designed; it only serves senders whose session closed).
If you dispatched and idle without arming, nothing will ever wake you.
`@target` resolves in the **caller's own project** (then falls back to scanning
`~/Projects` registries) — external-project owners monitor their own agents with it.
Default timeout is **600 s**; pass `--timeout <s>` for longer builds. Mid-watch it
also emits `[watchdog.stall]` / `[watchdog.resumed]` events (no JSONL activity 120 s
with no in-flight tool = probable stuck agent).
| Command | What it does | | Command | What it does |
|---|---| |---|---|
| `watchdog agent @target` | Monitor dispatched agent until it finishes | | `watchdog agent @target [--timeout s]` | Wake when the dispatched agent exits (default 600 s) |
| `watchdog timer <duration>` | Wake after duration (5m, 30s, 2h, 1h30m) | | `watchdog timer <duration>` | Wake after duration (5m, 30s, 2h, 1h30m) |
| `watchdog timer start/stop <name>` | Named duration tracking | | `watchdog timer start/stop <name>` | Named duration tracking |
| `watchdog schedule <HH:MM>` | Wait until a specific time | | `watchdog schedule <HH:MM>` | Wait until a specific time |
@@ -67,7 +94,14 @@ All commands via `drone @devpulse <command>`:
| `watchdog cancel <id>` | Cancel a running watchdog | | `watchdog cancel <id>` | Cancel a running watchdog |
| `watchdog list` | List all watchdog entries | | `watchdog list` | List all watchdog entries |
### Feedback — personal cross-branch mailbox ### Feedback — the owner-to-owner channel (owner-only)
ai_mail and dispatch stop at the project boundary — **cross-project comms is
impossible by design, except feedback.** Project owners (managers) talk owner-to-owner
through it: an external project's owner runs `drone @devpulse feedback send ...` from
their project and it lands in devpulse's feedback mailbox; devpulse answers with
`feedback reply`. Same owner gate as watchdog — unseated projects are refused until
`aipass doctor --fix` seats them.
| Command | What it does | | Command | What it does |
|---|---| |---|---|
@@ -75,7 +109,7 @@ All commands via `drone @devpulse <command>`:
| `feedback inbox` | List all messages | | `feedback inbox` | List all messages |
| `feedback view <id>` | Read a message | | `feedback view <id>` | Read a message |
| `feedback reply <id> "msg"` | Reply to sender | | `feedback reply <id> "msg"` | Reply to sender |
| `feedback send "subject" "body"` | Receive feedback from another agent | | `feedback send "subject" "body"` | Send feedback to devpulse (any project's owner may call) |
### Compass — rated decision store ### Compass — rated decision store
@@ -122,7 +156,7 @@ drone @git log # Recent commits
All branches via dispatch orchestration. Watchdog monitoring for any dispatched agent. Feedback channel for cross-branch communication. Git operations (commit, PR, merge) for the entire project. All branches via dispatch orchestration. Watchdog monitoring for any dispatched agent. Feedback channel for cross-branch communication. Git operations (commit, PR, merge) for the entire project.
*Last Updated: 2026-06-23* *Last Updated: 2026-07-11*
--- ---
@@ -20,6 +20,7 @@ a live ai_mail dispatch flow. They're skipped by default in CI.
import json import json
import os import os
import sys
import time import time
from pathlib import Path from pathlib import Path
@@ -346,12 +347,27 @@ def test_stalltracker_resume_clears_stall(monkeypatch, capsys):
def _fake_clock_sleep(agent_module, monkeypatch, lock_file, unlink_at=200.0, step=60.0): def _fake_clock_sleep(agent_module, monkeypatch, lock_file, unlink_at=200.0, step=60.0):
"""Patch monotonic + sleep with a fake clock that advances `step`s per sleep """Patch monotonic + sleep with a fake clock that advances `step`s per sleep
and unlinks the dispatch lock once the clock passes `unlink_at` (loop exit).""" and unlinks the dispatch lock once the clock passes `unlink_at` (loop exit).
THREAD-SCOPED (S300): ``agent_module.time`` is the shared stdlib module, so
patching ``time.sleep`` is process-global — background daemon threads (prax
logger spawns three on first log) also hit the fake and would race the
clock forward, unlinking the lock before ``watch_agent`` even reads it
(flaked exactly so: 'no active lock' + uptime-sized elapsed). Only sleeps
called FROM the agent module advance the clock; foreign callers get a tiny
real sleep so they don't spin hot.
"""
clock = {"t": 0.0} clock = {"t": 0.0}
agent_file = Path(agent_module.__file__).resolve()
real_sleep = time.sleep
monkeypatch.setattr(agent_module.time, "monotonic", lambda: clock["t"]) monkeypatch.setattr(agent_module.time, "monotonic", lambda: clock["t"])
def fake_sleep(_seconds): def fake_sleep(_seconds):
"""Advance the fake clock and drop the lock once past unlink_at (loop exit).""" """Advance the fake clock for agent-module callers only."""
caller = Path(sys._getframe(1).f_code.co_filename).resolve()
if caller != agent_file:
real_sleep(0.001) # background thread — keep it off the fake clock
return
clock["t"] += step clock["t"] += step
if clock["t"] >= unlink_at: if clock["t"] >= unlink_at:
lock_file.unlink(missing_ok=True) lock_file.unlink(missing_ok=True)