Agents in operations
Agents you can put into an operation, not into a demonstration.
It has to be right. It has to say why. And it has to know when to stop and hand over to a person. Those three requirements are what separate an agent you can put into an operation from a demonstration that impressed everyone in the room.
People arrive here for one of three reasons. An agent is running and nobody can tell where its answers come from. An agent is about to ship and nobody is willing to sign off on it. Or an agent answers, and the work is redone by hand anyway.
Answering and deciding are not the same product
An agent that answers is judged by whether the answer was useful. An agent that decides is judged by whether someone can put their name on it.
That second requirement changes the engineering. It means every output carries the sources it stood on. It means there is a boundary drawn around what the agent is allowed to conclude, and a rule for what happens at that boundary. It means the same question asked twice returns the same verdict, because a system that answers differently on Tuesday cannot be signed off on Monday.
Most of what is deployed in operations today was built to the first standard and is being used to the second. That gap is where our work starts.
What sits in this line
Judgment-bearing agents — compliance monitoring, audit, document verification. Work where being wrong has a cost that someone owns.
Several agents orchestrated with clear boundaries of responsibility — including the boundary that says what happens when one of them is confidently incorrect.
Grounded retrieval with source attribution — every conclusion carrying the document, the section and the revision it came from, because in a regulated setting the answer without its source is not an answer.
Escalation rules to a human — written before the agent ships, not after the first incident. What it may decide, what it must hand over, and to whom.
Regression testing and drift control — a fixed set of questions with known answers, rerun on every model change, every corpus change and every prompt change.
The knowledge layer underneath the agent — assembling the corpus, marking up sources, and keeping it current. It ships inside the system, not as a separate purchase.
No lock to a model provider — the same agent running on a different model, with the regression set as the evidence that it still behaves.
How the work runs
- Understand — sixty real questions with known correct answers, run against whatever you have now. Not a workshop.
- Design — where the boundary of the agent's authority sits, what it must escalate, and to whom.
- Build — the agents, the orchestration, and the knowledge layer underneath them.
- QA — the sixty questions run again before release, and again after every model or corpus change. Drift is reported monthly rather than noticed anecdotally.
- Integrate into your landscape — into the systems where the work actually happens, so the verdict lands in the workflow rather than in a chat window someone has to remember to open.
- Hand it over — repository, documentation, configuration, prompts, and the regression set. The regression set is the part people forget to ask for, and it is the part that lets you change vendors, including changing away from us.
- Support — the regression set keeps running and the escalation rules keep being adjusted as the corpus changes.
Ten ways an agent in production is wrong
| # | Failure mode | What it looks like | What catches it |
|---|---|---|---|
| 1 | Answered from nothing | Nothing in the corpus supports the answer. The model closed the gap itself | Empty or thin attribution |
| 2 | Answered from the wrong document | The source looks right, but it is a different revision, a different jurisdiction, a different vessel, a different contract | Almost nothing — the attribution looks healthy. Only questions with known answers catch it |
| 3 | Answered from a superseded revision | The corpus holds both the version in force and the one it replaced | Dating and versioning of sources |
| 4 | Confident and wrong | Said in an even tone, and the operator took it | A run against questions with known answers |
| 5 | Partial, presented as complete | Three conditions named out of five | Completeness marked in the reference answers |
| 6 | Escalated where it should have answered | The agent hands over too often | Escalation rate against the rate that was warranted |
| 7 | Answered where it should have escalated | The agent decided outside its own authority | Escalation rules, which usually do not exist |
| 8 | Two verdicts for one question | Two runs, two answers | Re-running a fixed set |
| 9 | Drift after a model or corpus update | It passed yesterday and fails today | Regression against a fixed set |
| 10 | Right, and unusable | The answer is correct, no trace of the reasoning survives, and a person can neither accept it nor reject it | A requirement on the output format, not on the model |
Everyone tests the first row, because the first row is visible. The money is lost on the second and the seventh — an answer sourced from a document that looks right and is not, and a decision taken that should have been handed to a person.
The four packages
| Package | Duration |
|---|---|
| B1 · Agent audit | 2 weeks |
| B2 · Pilot on one process | 5 weeks |
| B3 · Agent system | 10–14 weeks |
| B4 · Agent QA | retainer |
The audit is sixty questions with known correct answers, up to three knowledge sources. That boundary is deliberate and it is in the contract, and so is the rate for going past it.
You get the run itself, a catalogue of failure modes with your own examples in it, an attribution report saying what the agent leaned on and what was not in the corpus at all, a ranked list of causes, and a quote to fix them. Credited against a pilot or a build within 60 days.
One thing worth knowing before you buy the audit. Producing sixty questions with known correct answers is harder than it sounds, and a fair number of teams discover they cannot. That discovery is itself the finding, and it is cheaper found in an audit than at the point where someone has to sign off.
B4 is sold only after a pilot or a system. Nobody buys monitoring first, and it is offered in that order for the same reason.
What we can show you, and what we cannot
LogiSea is our own system for maritime fleets, and the agents in it monitor compliance and audit procurement — that is, they reach verdicts rather than offer suggestions. It is our strongest engineering evidence in this line.
Three vessels are in installation. Not in production, not deployed at a fleet, not running a year of data. When that changes, this paragraph changes with it.
We have not yet published a sample audit report. It is the next artefact we owe this page, and until it exists you are reading a description of a deliverable rather than looking at one. That is a fair thing to hold against us, and a fair thing to ask for on the call.
What this line assumes
Sixty questions with known correct answers are the entry requirement. Where a company cannot produce them, the problem is upstream of the agent, and the audit says so rather than solving it.
Support chatbots are a different problem and a well served one. We work on agents that reach a verdict and have to defend it.
Fine-tuning is almost never what was missing. In nearly every case somebody asked for it, the answer sat in the corpus or in the escalation rules.
Sign-off stays inside your company. We produce the evidence the person signing needs. The name on the line is theirs.
What is sold is a system that runs, not people who are available. Scope sets the price; retainers are priced by volume handled.
An agent worth putting into an operation usually stands on a data layer that had to be built first, and often lives inside a system that was built from zero.
Forty-five minutes. Bring five questions your agent gets wrong, and the answers you wish it gave. We will look at where they came from and tell you which of the ten rows above you are in.