Will Kinsey

Agent Lab · Playbook 08 · Pattern: Meter

Measuring my own AI usage without logging content

I wanted to know which models actually do my work, without building a log of everything I’ve asked them. Nightly collectors pull usage metadata from Codex and Claude into a local database. Prompts, responses and tool contents are never stored.

Status
Live
Agents
OpenAI Codex, Claude Code
Built with
Python, SQLite, scheduled jobs
Last updated
October 28, 2026

The problem

I use several models every day and route work between them on purpose. Without measurement, “I mostly use the cheaper model for that” is a guess. But the obvious way to measure, keeping full transcripts, creates a sensitive archive I don’t want.

How it works

Replay with real Codex numbers

  1. Collect nightlyThe Codex collector runs at 11:55 PM and the Claude tracker at 2 AM. Each reads local session logs.
  2. Keep metadata onlyModel, token counts, timestamps, session and request IDs, and tool names. No prompts, responses, conversation titles or tool contents.
  3. Store locallyEverything goes into SQLite on this Mac.
  4. Report weeklyEvery Monday at 8:30 AM a report shows each model family’s share and flags any family above 60% or 75% of the workload.

What it shows

Codex, Sep 26 to Oct 26Share of tokens
GPT-6 Sol65.8%
GPT-6 Luna18.5%
Astra14.9%
GPT-6.1 Terra0.8%

That window covered 1,321 primary turns across 318 sessions, and there are nine history snapshots so far. On the Claude side, over the 30 days to October 28, Opus carried 84.0% of tokens, Sonnet 15.3% and Haiku 0.7%. The Claude ledger only starts on October 6, so that history is short.

The Haiku share is small because delegating to it only started this month. That’s exactly the kind of thing I want a number for rather than a feeling.

Setup steps

  1. Decide what you will never store, and write it down before writing the collector.
  2. Write a narrow schema: only the columns you need. Anything not in the schema can’t leak into it.
  3. Collect nightly from local logs with a scheduled job, and test the parser. Mine have 14 tests (Codex) and 42 (Claude).
  4. Scan the database for privacy after the first import, for example that title and content columns are empty.
  5. Report weekly with shares by model family and a flag when one family dominates.

Copy-ready collector spec

Hand this to a coding agent to build your own. A GitHub version will come with the skills shelf.

Build a private AI usage collector

Goal: a nightly job that records how much I use each AI model, without storing what I said or what it answered.

Store only these fields per request:
  timestamp, tool (e.g. Codex or Claude), session ID, request ID,
  model ID, role (main or helper agent), input tokens, cached input tokens,
  output tokens, tool names used (names only)

Never store: prompts, responses, conversation titles, tool arguments,
tool outputs, file contents.

Requirements
- Read the tool's local session logs; write to a local SQLite file.
- Idempotent: re-running the same night must not double count (unique on request ID).
- Narrow schema: no free-text columns.
- After import, run a privacy check that fails if any stored value looks like
  message text (long strings outside the allowed fields).
- Tests for the parser, including malformed log lines and repeated runs.
- A weekly report: token share by model family for the last 30 days, and a flag
  when one family exceeds 60% or 75%. Open the database read-only.
- Note in the report that rolling windows overlap and tokens are workload, not billing.

Guardrails

  • No content. No prompts, responses, titles or tool contents, enforced by the schema and checked after import.
  • Local only. The database stays on this Mac.
  • Read-only reporting. The weekly report opens the database read-only.
  • Honest about limits. The database does store project folder paths, so I don’t claim it’s anonymous.

What broke and what I changed

  • Rolling windows overlap. Two 30-day reports a week apart share most of their data, so they’re trends, not separate samples.
  • Tokens aren’t a bill. Token totals measure workload, not cost; cached input alone was about 97% of Codex input in the October window.
  • Exports can be incomplete. Claude Cowork exports may leave out the final reply, so those counts can run low.
  • Done isn’t success. A task finishing technically says nothing about whether it helped; usage data can’t answer that.

Last updated October 28, 2026. Back to the Agent Lab