devpulse: watchdog/feedback README rewrite (was stale, missed everything that tripped VERA) + fix thread-unsafe watchdog test + true up branch-prompt timeout. README (last touched 06-23) predated the owner gate and Monitor-tool wake: now documents owner-only gating (seated owner:true, doctor --fix repairs), the 3-step wake mechanic (dispatch -> arm via Monitor TOOL -> Monitor return IS the wake; '1 monitor' IS the indicator), why passive wake-back can NEVER reach an interactive session (BLOCKED line = by design), cross-project @target resolution, 600s default + --timeout, stall events; feedback re-documented as THE owner-to-owner cross-project channel (Patrick ruling: cross-project comms impossible by design except feedback). TEST FIX: test_watchdog_agent _fake_clock_sleep patched GLOBAL time.sleep with a stateful fake -> prax logger's 3 daemon threads raced the fake clock forward and unlinked the fixture lock before watch_agent read it (nondeterministic 'no active lock' + uptime-sized elapsed; failed 4x today, passed in the same-day full-repo sweep). Fix: thread-scope the fake via caller-frame check (only agent-module sleeps advance the clock; foreign threads get a real 1ms sleep). Verified 17 pass 3x deterministic, devpulse suite 407 green. Branch prompt: watchdog default timeout claim corrected 1800s -> 600s (hit live: 3 watchdogs expired mid-build at ~600s).
This commit is contained in:
@@ -86,7 +86,7 @@ drone @flow list open # active plans
|
|||||||
|
|
||||||
# Watchdog
|
# Watchdog
|
||||||
|
|
||||||
Devpulse module. After dispatch, arm as a background task — it polls the dispatch lock and exits when the agent finishes. Resolves @target → branch path → `.ai_mail.local/.dispatch.lock`. Default timeout 1800s; `drone @devpulse watchdog --help` for the full reference.
|
Devpulse module. After dispatch, arm as a background task — it polls the dispatch lock and exits when the agent finishes. Resolves @target → branch path → `.ai_mail.local/.dispatch.lock`. Default timeout **600s** — pass `--timeout <s>` for longer builds (verified live S300; `drone @devpulse watchdog --help` for the full reference).
|
||||||
|
|
||||||
```
|
```
|
||||||
drone @ai_mail dispatch @target "Subject" "Body"
|
drone @ai_mail dispatch @target "Subject" "Body"
|
||||||
|
|||||||
@@ -44,7 +44,7 @@ src/aipass/devpulse/
|
|||||||
│ │ └── watchdog/ # Agent, timer, schedule, registry
|
│ │ └── watchdog/ # Agent, timer, schedule, registry
|
||||||
│ └── plugins/ # Plugin extension point
|
│ └── plugins/ # Plugin extension point
|
||||||
├── devpulse_json/ # JSON handler storage (config, data, logs per module)
|
├── devpulse_json/ # JSON handler storage (config, data, logs per module)
|
||||||
├── tests/ # 282 tests
|
├── tests/ # 407 tests
|
||||||
├── artifacts/ # Birth certificate, reports
|
├── artifacts/ # Birth certificate, reports
|
||||||
├── dropbox/ # Received files, archived plans, install audit
|
├── dropbox/ # Received files, archived plans, install audit
|
||||||
├── docs/ # Transition notes
|
├── docs/ # Transition notes
|
||||||
@@ -55,11 +55,38 @@ src/aipass/devpulse/
|
|||||||
|
|
||||||
All commands via `drone @devpulse <command>`:
|
All commands via `drone @devpulse <command>`:
|
||||||
|
|
||||||
### Watchdog — directed wake system
|
### Watchdog — directed wake system (owner-only)
|
||||||
|
|
||||||
|
**Who may call it:** the project OWNER only — the first agent, seated as `owner: true`
|
||||||
|
in the project's sealed `*_REGISTRY.json`. Portable: `@devpulse` in AIPass, `@vera` in
|
||||||
|
Vera Studio, whoever owns elsewhere. A refusal means your project's owner isn't seated —
|
||||||
|
run `aipass doctor` to see why and `aipass doctor --fix` to repair (DPLAN-0239).
|
||||||
|
|
||||||
|
**How the wake works (read this once, save a debugging session):**
|
||||||
|
|
||||||
|
1. `drone @ai_mail dispatch @target "Subject" "Body"` — hand off the work.
|
||||||
|
2. **Immediately arm the watchdog via the harness Monitor TOOL** — never Bash
|
||||||
|
`run_in_background` (its output goes nowhere and cannot wake you):
|
||||||
|
`drone @devpulse watchdog agent @target --timeout 600`
|
||||||
|
3. The status line shows **"1 monitor"** the moment it's armed — that IS the
|
||||||
|
active-dispatch indicator. When `@target` finishes, the watchdog exits, the
|
||||||
|
Monitor completes, and **your session is re-invoked with the result — that IS
|
||||||
|
the wake.**
|
||||||
|
|
||||||
|
There is no passive wake: ai_mail's wake-back spawns a new headless process and can
|
||||||
|
never inject into a live interactive session (`BLOCKED — interactive session` in the
|
||||||
|
logs is that guard working as designed; it only serves senders whose session closed).
|
||||||
|
If you dispatched and idle without arming, nothing will ever wake you.
|
||||||
|
|
||||||
|
`@target` resolves in the **caller's own project** (then falls back to scanning
|
||||||
|
`~/Projects` registries) — external-project owners monitor their own agents with it.
|
||||||
|
Default timeout is **600 s**; pass `--timeout <s>` for longer builds. Mid-watch it
|
||||||
|
also emits `[watchdog.stall]` / `[watchdog.resumed]` events (no JSONL activity 120 s
|
||||||
|
with no in-flight tool = probable stuck agent).
|
||||||
|
|
||||||
| Command | What it does |
|
| Command | What it does |
|
||||||
|---|---|
|
|---|---|
|
||||||
| `watchdog agent @target` | Monitor dispatched agent until it finishes |
|
| `watchdog agent @target [--timeout s]` | Wake when the dispatched agent exits (default 600 s) |
|
||||||
| `watchdog timer <duration>` | Wake after duration (5m, 30s, 2h, 1h30m) |
|
| `watchdog timer <duration>` | Wake after duration (5m, 30s, 2h, 1h30m) |
|
||||||
| `watchdog timer start/stop <name>` | Named duration tracking |
|
| `watchdog timer start/stop <name>` | Named duration tracking |
|
||||||
| `watchdog schedule <HH:MM>` | Wait until a specific time |
|
| `watchdog schedule <HH:MM>` | Wait until a specific time |
|
||||||
@@ -67,7 +94,14 @@ All commands via `drone @devpulse <command>`:
|
|||||||
| `watchdog cancel <id>` | Cancel a running watchdog |
|
| `watchdog cancel <id>` | Cancel a running watchdog |
|
||||||
| `watchdog list` | List all watchdog entries |
|
| `watchdog list` | List all watchdog entries |
|
||||||
|
|
||||||
### Feedback — personal cross-branch mailbox
|
### Feedback — the owner-to-owner channel (owner-only)
|
||||||
|
|
||||||
|
ai_mail and dispatch stop at the project boundary — **cross-project comms is
|
||||||
|
impossible by design, except feedback.** Project owners (managers) talk owner-to-owner
|
||||||
|
through it: an external project's owner runs `drone @devpulse feedback send ...` from
|
||||||
|
their project and it lands in devpulse's feedback mailbox; devpulse answers with
|
||||||
|
`feedback reply`. Same owner gate as watchdog — unseated projects are refused until
|
||||||
|
`aipass doctor --fix` seats them.
|
||||||
|
|
||||||
| Command | What it does |
|
| Command | What it does |
|
||||||
|---|---|
|
|---|---|
|
||||||
@@ -75,7 +109,7 @@ All commands via `drone @devpulse <command>`:
|
|||||||
| `feedback inbox` | List all messages |
|
| `feedback inbox` | List all messages |
|
||||||
| `feedback view <id>` | Read a message |
|
| `feedback view <id>` | Read a message |
|
||||||
| `feedback reply <id> "msg"` | Reply to sender |
|
| `feedback reply <id> "msg"` | Reply to sender |
|
||||||
| `feedback send "subject" "body"` | Receive feedback from another agent |
|
| `feedback send "subject" "body"` | Send feedback to devpulse (any project's owner may call) |
|
||||||
|
|
||||||
### Compass — rated decision store
|
### Compass — rated decision store
|
||||||
|
|
||||||
@@ -122,7 +156,7 @@ drone @git log # Recent commits
|
|||||||
|
|
||||||
All branches via dispatch orchestration. Watchdog monitoring for any dispatched agent. Feedback channel for cross-branch communication. Git operations (commit, PR, merge) for the entire project.
|
All branches via dispatch orchestration. Watchdog monitoring for any dispatched agent. Feedback channel for cross-branch communication. Git operations (commit, PR, merge) for the entire project.
|
||||||
|
|
||||||
*Last Updated: 2026-06-23*
|
*Last Updated: 2026-07-11*
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
|
|||||||
@@ -20,6 +20,7 @@ a live ai_mail dispatch flow. They're skipped by default in CI.
|
|||||||
|
|
||||||
import json
|
import json
|
||||||
import os
|
import os
|
||||||
|
import sys
|
||||||
import time
|
import time
|
||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
|
|
||||||
@@ -346,12 +347,27 @@ def test_stalltracker_resume_clears_stall(monkeypatch, capsys):
|
|||||||
|
|
||||||
def _fake_clock_sleep(agent_module, monkeypatch, lock_file, unlink_at=200.0, step=60.0):
|
def _fake_clock_sleep(agent_module, monkeypatch, lock_file, unlink_at=200.0, step=60.0):
|
||||||
"""Patch monotonic + sleep with a fake clock that advances `step`s per sleep
|
"""Patch monotonic + sleep with a fake clock that advances `step`s per sleep
|
||||||
and unlinks the dispatch lock once the clock passes `unlink_at` (loop exit)."""
|
and unlinks the dispatch lock once the clock passes `unlink_at` (loop exit).
|
||||||
|
|
||||||
|
THREAD-SCOPED (S300): ``agent_module.time`` is the shared stdlib module, so
|
||||||
|
patching ``time.sleep`` is process-global — background daemon threads (prax
|
||||||
|
logger spawns three on first log) also hit the fake and would race the
|
||||||
|
clock forward, unlinking the lock before ``watch_agent`` even reads it
|
||||||
|
(flaked exactly so: 'no active lock' + uptime-sized elapsed). Only sleeps
|
||||||
|
called FROM the agent module advance the clock; foreign callers get a tiny
|
||||||
|
real sleep so they don't spin hot.
|
||||||
|
"""
|
||||||
clock = {"t": 0.0}
|
clock = {"t": 0.0}
|
||||||
|
agent_file = Path(agent_module.__file__).resolve()
|
||||||
|
real_sleep = time.sleep
|
||||||
monkeypatch.setattr(agent_module.time, "monotonic", lambda: clock["t"])
|
monkeypatch.setattr(agent_module.time, "monotonic", lambda: clock["t"])
|
||||||
|
|
||||||
def fake_sleep(_seconds):
|
def fake_sleep(_seconds):
|
||||||
"""Advance the fake clock and drop the lock once past unlink_at (loop exit)."""
|
"""Advance the fake clock for agent-module callers only."""
|
||||||
|
caller = Path(sys._getframe(1).f_code.co_filename).resolve()
|
||||||
|
if caller != agent_file:
|
||||||
|
real_sleep(0.001) # background thread — keep it off the fake clock
|
||||||
|
return
|
||||||
clock["t"] += step
|
clock["t"] += step
|
||||||
if clock["t"] >= unlink_at:
|
if clock["t"] >= unlink_at:
|
||||||
lock_file.unlink(missing_ok=True)
|
lock_file.unlink(missing_ok=True)
|
||||||
|
|||||||
Reference in New Issue
Block a user