Notes

A demonstration prompt is a kind of theatre. Someone types a question the tool was prepared for, the answer arrives in a tidy paragraph, and the room relaxes. The week after the demonstration, a clerk asks the hundred questions that actually appear in the job, including the ones that are badly spelled, half-remembered, or aimed at a file that was never scanned. Those hundred are the brief.

We ask for them before we write much code. They come from mail, from tickets, from the last month of “have you got a minute.” We sit with the person who would have to live with a wrong answer and we write down what a right one would contain, including the cases where the only honest reply is that the file does not say. The set is ugly. It is also the only way we know whether retrieval is even possible.

A hundred is a lower bound. Some builds want three hundred because the types of request keep branching. We stop adding when a new question stops teaching us anything about a missing folder or a missing field. The extra questions after that are comfort, and comfort is how evaluation sets rot.

If an organisation cannot produce the questions, we treat that as a finding. It usually means the work is still in people’s heads. In that case the first job is writing things down, and a model will not do that writing for you.

People talk about pipelines as if the gain were the messages that file themselves. The real design is about the messages that should not. Every automated row you accept is a row nobody looked at. That can be fine for a booking date that matches a known pattern. It is not fine for a cancellation that names the wrong shipment.

Attention is the scarce thing on a dispatch desk or in a clinic office. A pipeline that dumps every low-confidence case into a heap has only moved the heap. A pipeline that hides low confidence has spent the attention you will need later, when a customer rings. We spend the discovery week watching which exceptions a person actually wants to see, and we make that queue short and visible.

There is a cost in the other direction too. If you insist that a person tick every field, you have bought a typing assistant and called it automation. The honest conversation is which fields, on which types, are allowed to pass, and who is named when one of those passes is wrong.

We write that down. It looks like a dull table. Six months later it is the only artefact anyone can use to argue with a new manager who wants the queue switched off.

Hand-over day is a good day. The scores are green, the guide is printed, someone on the team can add a file. Three months later the thing that breaks is rarely the model. It is the world around the model. A folder is reorganised. A template gains a field. The person who owned the evaluation set takes another job. A vendor ships a “silent” update that changes a refusal into a guess.

The failures look polite. An assistant still answers. A pipeline still writes rows. The rows are slightly wrong in a way that only the person who left would have spotted. This is why we argue for a named owner on your side, and for a set of questions that can be re-run without us. A monthly visit is one way to keep that habit. A calendar reminder and a script is another. What does not work is hope.

The other quiet break is permission. Someone shares a drive more widely “so the assistant can see it,” and a month later the assistant can see a draft it was never meant to cite. Access should follow the folders you already trust. When that rule is bent, we would rather hear about it than discover it in a log.

None of this is dramatic. It is housekeeping. The studio exists for the builds; the housekeeping is how a build stays a tool instead of becoming a story people tell about the month it worked.