Will Kinsey

Agent Lab · Playbook 01 · Pattern: Triage

Let a cheap model decide what’s routine

My agent asks a lot of questions that don’t matter. A small decision model answers the low-risk ones and screens risky shell commands, so only real decisions reach my phone. Anything involving money, messages, deletes, credentials or production skips the model and comes to me.

Status
Live since October 27, 2026
Agent
Hermes Agent (my instance is called Iris)
Built with
Jev on OpenRouter, Python, Telegram
Last updated
October 28, 2026

The problem

Hermes is careful, which is good. It also asks a steady stream of questions that don’t need me: “May I proceed?”, “Should I save this?”, “Port 3001 or 3002?” Each one interrupts whatever I’m doing, and the important questions get lost in the noise.

Turning approvals off isn’t the answer. I want the agent to stop asking about things that are routine and reversible, and keep asking about everything else.

How it works

Two layers, one rule: when in doubt, ask me. Jev is a decision model, not a chatbot; here’s what that means.

Replay with sample questions

Layer 1: the agent’s own questions

My plugin, jev-approval-triage. Before Hermes asks me a permission or choice question, it calls a tool that does this:

  1. Hard stopsPurchases, money, outbound messages, deleting or overwriting, credentials and access, security, production and legal questions come straight to me. No model is called.
  2. TriageJev rates the question low, medium or high risk, and flags questions only I can answer.
  3. DecideOnly confident low-risk questions get a second request. Jev picks one of the agent’s options or “ask the owner.” Below 0.7 confidence, I’m asked.
  4. ReportThe agent tells me “Decided by Jev: question → answer” in its next update, so I can reverse it.

Layer 2: risky shell commands

Hermes’s built-in “smart” approval mode, with Jev set as the reviewer through a community plugin (jev-approvals by anpicasso). I configured it; I didn’t write it.

  1. FlagHermes matches a command against its list of dangerous patterns, such as recursive deletes or sudo.
  2. VerdictJev answers APPROVE, DENY or ESCALATE.
  3. AskAPPROVE runs. DENY and ESCALATE come to my phone as a Telegram approval card. I can override a DENY for that one command only.
  4. Time out safeIf I don’t answer within 60 seconds, the command does not run. In scheduled jobs, with nobody there to approve, flagged commands are blocked.

First results

From the logs for October 27 and 28, 2026. It’s a small sample, so read it as a first look, not a benchmark.

LayerTotalHandled by JevCame to me
Agent questions123 decided9 (3 hard stops, 6 after triage)
Shell commands93 approved6 (4 denied, 2 escalated)

Five of the nine commands were a test set. Jev approved python3 -c "print('hello')" and rm -rf ./build/__pycache__. It denied rm -rf ~/Documents, deleting a notes folder, and a command that piped an SSH key to a paste site. Command verdicts took 0.26 to 0.40 seconds.

Setup steps

  1. Connect OpenRouter to Hermes. The plugin uses the key Hermes already holds, so most setups need nothing extra. 
    hermes auth add openrouter
  2. Install the triage plugin. The public GitHub repo is being prepared; this page will link to it.
  3. Start in shadow mode. Jev’s answers are logged, but you still get asked. Add this to ~/.hermes/config.yaml: 
    plugins:
  entries:
    jev-approval-triage:
      settings:
        model: typesafe/jev-1.14
        mode: shadow            # switch to "decide" once you trust it
        min_confidence: 0.7     # below this, you are asked
        extra_hard_stops:       # words that always come to you
          - customer
          - school
  4. Read the log after a few days, then switch mode to decide. Every result is in ~/.hermes/logs/jev-approval-triage.jsonl.
  5. For shell commands, install the jev-approvals community plugin and turn on smart approvals: 
    approvals:
  mode: smart
  timeout: 60          # seconds; no answer means it does not run
  cron_mode: deny      # scheduled jobs: flagged commands are blocked
auxiliary:
  approval:
    provider: typesafe-jev
    model: typesafe/jev-1.14
    base_url: https://openrouter.ai/api/alpha

Copy-ready skill

This is the skill that ships with the plugin. It tells the agent when to call the triage tool and, just as important, when not to.

---
name: jev-approval-triage
description: How and when to use the jev_triage tool so the user is not interrupted by low-risk approval questions.
---

# Jev Approval Triage

The user installed this plugin because they are tired of answering approval questions that don't matter. Use `jev_triage` before every permission or choice question you would otherwise send them.

## When to Call It
- "May I proceed with …?", "Should I …?", "Do you want me to …?"
- "Which of these options …?" where a sensible default exists.
- Follow-up confirmations inside a task the user already asked for.

## When Not to Call It
- The answer depends on facts only the user knows: what they meant, what they see on their screen, something in the physical world, personal taste between equal options. Just ask.
- The action already needs explicit approval under your rules or the tool's own approval prompt. The triage never replaces those.

## How to Call It
- `question`: the exact question you would send.
- `proposed_action`: what you will actually do. Be concrete ("configure nginx for the arcade on port 3001 with its own certificate"), not vague ("make the change").
- `options`: the choices, your recommendation first.
- `context`: the user's original request, what is done, and the cost of a wrong choice.

Describe the action honestly. Never reword a question to dodge a hard stop: that defeats the point of the plugin and the user will lose trust in it.

## Acting on the Result
- `outcome: decided`: do what `decision` says. In your next report add: "Decided by Jev: <question> → <decision>" so the user can reverse it.
- `outcome: ask_user`: ask the user normally. The `reason` field is for you; don't repeat it unless it helps.
- If the tool errors or is unavailable, ask the user.

## What Happens Inside
1. Hard stops (purchases, money, outbound messages, deleting or overwriting, credentials and access, security, production, legal) go straight to the user with no model call.
2. Jev, a decision model, rates risk low/medium/high and flags questions only the user can answer.
3. Only confident low-risk questions get a second Jev request that picks one of your options or "ask the owner". Low confidence goes to the user.
4. Only Jev is ever called. Every result is logged to `~/.hermes/logs/jev-approval-triage.jsonl` with the model, tokens and cost.
5. Jev returns no written rationale, so report its decision as-is.

Guardrails

  • Hard stops are code, not judgment. The categories that always come to me are checked before any model is called.
  • Jev only. Every request pins a Jev model and restricts OpenRouter to the TypeSafe provider with fallbacks off. A reply from any other model is rejected.
  • Fail safe. An error, a timeout or an unreadable reply means I get asked as usual.
  • Visible. Every decision is logged and reported back to me, so I can reverse it.
  • Not for my facts or my taste. Questions about what I meant, what I can see, or a choice between equal options go to me directly.
  • Commands wait for me. In the command layer, nothing escalated runs without an answer.

What broke and what I changed

  • It was slow at first. The first triage calls took 3 to 10 seconds. I switched the plugin to call Jev directly on OpenRouter’s decisions endpoint, and calls since then have taken 0.2 to 0.7 seconds.
  • It needed its own key. Setup asked for a separate OpenRouter key. Now it uses the key Hermes already holds, and a dedicated key is optional.
  • “Deletes always come to me” only holds in one layer. In the command layer, Jev approved removing a cache folder. That’s a sensible call, but it means the hard stops apply to the agent’s questions, not to every command. I’m stating that plainly rather than overselling the rule.
  • Some questions still need me. Facts only I know and personal taste can’t be delegated, so the skill tells the agent to ask me directly.
  • Known limits. The agent calls the tool because its instructions say to; nothing forces it. Hard stops are keyword-based, so they lean toward asking me and can’t catch an action described misleadingly.

Status: live since October 27, 2026. Last updated October 28, 2026. Back to the Agent Lab