FlagshipAgent OrchestrationHuman-in-the-LoopEvalsLive system
I run a team of agents. Here's the org design.
I designed, built, and now operate an agentic operations system — twenty-plus production AI skills that scan markets autonomously at 7 AM, tier what they find against standing rules, draft and validate documents against external scoring loops, and hand work between each other. This is a case study in the discipline the industry now calls agent UX — except the user was me, the stakes were real, and the system runs every day.
20+ production agents · running daily
Role
Designer, PM & operator
Team
1 human + 20+ agents
Timeline
In production
Stack
Claude Code · MCP · custom skills
01The problem
A one-person operation has the same jobs as an org: market intelligence, competitive research, document production, quality assurance, pipeline management, publishing. Doing them manually consumed the hours that should have gone to judgment-heavy work. The standard answer — "use AI tools" — produced a drawer full of disconnected chats: no memory, no handoffs, no quality gates, and no trust. A prompt is not a process.
The reframe: this is an organizational design problem, not a tooling problem.Which jobs get delegated entirely? Which need supervision? Where does a human approval gate belong, and what does the agent need to show me at that gate to earn the approval? These are the same questions every enterprise deploying agents is asking right now — Gartner projects 40% of enterprise apps will embed task agents by end of 2026. I've been answering them in production, on my own operations.
02The architecture
The system is a skill library — each skill a versioned, documented unit with its own workflow, reference data, and defined autonomy level. Skills compose: the morning market scan feeds the evaluation pipeline; evaluation feeds document production; production feeds an external validation loop that must clear a 75 Jobscan floor — 85+ for strong — before anything ships.
// The org chart, as it actually runs INTELLIGENCE market-scan (autonomous, 7 AM weekdays) → board-scan → skip-list governance EVALUATION tiering rules · location/comp hard filters · standing-rule memory PRODUCTION resume-build → humanizer pass → Jobscan validation ≥75 [GATE] MAINTENANCE skill authoring → versioned deploy → dual-location sync [SUPERVISED] PUBLISHING content drafts → voice check → human approval [GATE] → post // Every arrow is a designed handoff. Every [GATE] is a placed decision.
FIG. 01 — SYSTEM MAPFIVE LANES · TWENTY-PLUS SKILLS
AUTONOMOUSSUPERVISEDHUMAN GATE
INTEL
market-scan · 7 AM weekdays/board-scan · 7 boardsRUNS UNATTENDED
skip-list & standing rules↑ written by Evaluation · read by Intelligence — settled decisions never re-litigated
Five lanes, twenty-plus skills. Every arrow is a designed handoff; every gate is a placed decision, sized to blast radius.
03Design decisions
Autonomy is assigned per job, not per tool
Shipped
Blanket autonomy vs. graduated trust
The market scan runs fully autonomously — wrong output costs minutes. Document production runs supervised — wrong output costs an opportunity. Publishing has a hard human gate — wrong output costs reputation. Autonomy maps to blast radius, not to what the model can technically do.
Impact: zero irreversible actions taken without approval; no autonomous output has ever reached a third party
Memory as governance, not just context
Shipped
Stateless runs vs. standing rules
Every evaluation decision writes back to a persistent skip-list with standing rules ("category exclusions," "previously rejected — do not re-surface"). The agents don't re-litigate settled decisions, and I don't re-review them. This is the eval loop made durable: the system gets more aligned with my judgment every week I use it.
Impact: standing rules under governance; repeat-noise rate near zero
External validation beats self-assessment
Shipped
Model self-critique vs. independent scoring loop
Generated documents don't ship on the model's own confidence. They loop against Jobscan, an external scorer, until they clear the 75 floor — an objective, third-party eval the agent can't flatter its way past. Where no external scorer exists, a second agent with different instructions plays adversary.
Impact: measurable quality floor on every produced document
The human gate shows its work
Shipped
Approval theater vs. informed consent
At every gate the agent must present rationale, alternatives considered, and confidence — the preview-before-execute pattern. If I can't see why, I can't approve fast, and the whole system's speed collapses. Gate design IS the throughput design.
Impact: approval decisions in seconds, not re-review sessions
Kill the flaky, keep the mechanical
Killed
LLM judgment everywhere vs. deterministic where possible
Early versions let the model "eyeball" design diffs. It hallucinated changes. The rebuilt pipeline uses mechanical snapshot-and-diff — deterministic comparison, agent applies only what the diff proves. Lesson: agents earn trust by doing less, provably, not more, plausibly.
Impact: diff false-positive rate → ~0; trust in the pipeline → total
04A day in production
07:01 — the scan wakes up, runs seven job boards and a LinkedIn pipeline against hard filters and standing rules, writes two reports to disk, and flags anything needing same-day action. I read a two-minute summary over coffee and make the only decision that needs me: pursue or pass. The system did the org's work; I did the judgment. That ratio — machine hours to human minutes, with the human minutes spent entirely on decisions — is the whole thesis.
FIG. 02 — ONE MORNINGRUNS UNATTENDED
07:01 · WEEKDAYS · NO INPUT FROM ME
8
SOURCES SWEPT
7 job boards + professional network · each with its own filters
AUTONOMOUS
165
LISTINGS READ
posted in the last 24 hours · title, scope, location, comp band parsed
AUTONOMOUS
159
FILTERED OUT
hard rules: scope, altitude, location, standing exclusions
RULE-DRIVEN
6
SURFACED & TIERED
2 flagged time-sensitive · ranked, with reasoning attached
1
DECISION — MINE
pursue / pass · the only human minute
HUMAN GATE
One actual morning. The machine reads 165; I read 6; I decide 1.
FIG. 03 — THE CADENCEMAY 26 — JUL 24, 2026
23MORNINGS RUN
~3,800LISTINGS TRIAGED
0MINUTES SPENT TRIAGING
each mark is one weekday the system ran before I woke up
MONTUEWEDTHUFRI
RAN
IDLE — TRAVEL, HOLIDAY, OR MACHINE OFF
Not a demo that ran once. An operation that keeps running.
05Results
20+
Production skills in active rotation
hrs→min
Recurring workflows, compressed (e.g., 90 min by hand → 2-min review)
75+
Jobscan apply-band floor on shipped documents (85+ = strong)
Time figures come from my own operational logs; the system, its outputs, and its cadence are demonstrable live. This is a single-operator system. What it suggests for a team is a hypothesis I’d want to test, not a result I can claim.
06Key learning
"Designing for agents is org design. Decide what each one owns, what it must show you, and what it may never do — then get out of the way."
Every enterprise rolling out agents in 2026 is about to relearn management theory: delegation, trust-but-verify, span of control, escalation paths. Designers who have actually operated an agent workforce — not demoed one — are the people who can design these systems for users. I bring receipts.
