Six Months in Air-Gapped ManufacturingSixMonthsinAir-GappedManufacturing

The Engineering Handbook Assistant summary on our deployments page is two sentences long, which is about right for that context and does no justice to how the project actually went. This is the longer version, with the parts that didn't work included.

The starting problem

Forty years of process documentation and drawing annotations, accumulated across format changes, department reorganisations, and at least three document management systems that were each supposed to be the last one. Shop-floor staff needed answers in seconds, not after a search through a folder structure nobody fully understood anymore. The hardware constraint was non-negotiable from day one: air-gapped, on-premise, no exceptions — the corpus included drawing annotations the client was contractually barred from letting touch any external network, full stop.

What "forty years of documentation" actually meant

Not a clean corpus. Scanned drawings with handwritten margin notes, superseded revisions still sitting alongside current ones with no reliable way to tell which was which from metadata alone, and process documents written in a house shorthand that assumed the reader already knew the equipment. Ingestion took longer than the retrieval architecture did — roughly ten weeks against the project's twelve-month total, most of it spent on:

• OCR passes tuned specifically for the drawing annotation style, since general-purpose OCR consistently mangled the domain-specific abbreviations
• A revision-resolution pass to establish which version of a document was current, built in cooperation with two engineers who'd been there long enough to know the informal versioning conventions no system had ever documented
• Chunking tuned to preserve the relationship between a drawing and its annotations — naive text-chunking broke that relationship constantly in early passes

The retrieval architecture, briefly

Hybrid retrieval — dense embeddings plus keyword/BM25 — with re-ranking, on-prem, using the citation-grounding approach we've written about separately. Every answer resolves back to the specific drawing or document paragraph it came from, which mattered enormously here specifically because shop-floor staff needed to trust an answer enough to act on it without independently re-verifying it every time — the entire point of the system was time saved, and a system nobody trusts saves no time.

What went wrong, honestly

The first retrieval pipeline over-indexed on keyword match because the domain vocabulary was specific enough that dense embeddings alone kept surfacing plausible-sounding but wrong documents — a failure mode that didn't show up until real shop-floor queries started arriving, because our initial gold set was built by people too familiar with the material to ask the ambiguous questions an actual technician asks under time pressure. We rebuilt the evaluation set from logged real queries about six weeks in and re-tuned the hybrid weighting from there. That delay is the single biggest reason the project ran ten months instead of the eight we scoped.

Where it landed

In production for six months as of this writing, running entirely on the client's own hardware with no network path out. Query volume and latency figures aren't ones we're able to publish for this deployment, but the retrieval-and-generation-scored-separately evaluation approach described in our evaluating models on your own data note is directly descended from what we learned rebuilding this system's gold set mid-project — this deployment is the reason that's now how we start, not how we recover.