Quantising for Constrained HardwareQuantisingforConstrainedHardware

Quantisation gets talked about as a single lever: pick a bit-width, turn the dial, trade quality for footprint. In practice it's a family of decisions, and the ones that matter most for edge deployment are rarely the ones that show up first in a benchmark table.

The number nobody asks about first

Everyone asks "how much smaller" and "how much quality do we lose." The question that actually determines whether a deployment works is: quality loss on what, exactly? A model quantised to 4 bits might lose almost nothing on a general knowledge benchmark and lose a meaningful amount on the specific extraction or classification task your deployment depends on. Aggregate benchmarks average over failure modes that don't average over your use case.

Our process inverts the usual order: we build the task-specific evaluation set first, from real (or representative) queries against the actual corpus, and only then start comparing quantisation levels against it. A 4-bit model that's fine on average and bad specifically at the multi-step reasoning your workflow needs is not a 4-bit model you can ship.

Where precision actually gets spent

Three places consistently matter more than the headline bit-width:

• Attention vs. feed-forward layers. Uniform quantisation across the whole network is the easy default and usually not the right one. Mixed-precision schemes that keep attention layers at higher precision while quantising feed-forward blocks more aggressively tend to preserve reasoning quality better than a flat cut, at a smaller footprint cost than you'd expect.
• The embedding and output layers. These are cheap to keep at higher precision relative to their share of total parameters, and quantising them aggressively has an outsized effect on how confidently — and how correctly — the model commits to an answer near the end of generation.
• Calibration data. Post-training quantisation methods calibrate against a sample of data to decide how to allocate precision. Calibrating against generic text when your deployment is entirely domain-specific (maintenance logs, contract language, telemetry) quietly wastes precision on distributions the model will never see in production.

What "constrained hardware" actually constrains

For ruggedised edge nodes specifically — the case that shows up most often in field deployments — the binding constraint usually isn't raw model size, it's the combination of memory bandwidth and thermal headroom under sustained inference load, not a single burst. A model that fits in memory but throttles the device after twenty minutes of continuous queries hasn't actually solved the deployment problem. Our benchmark harness runs sustained-load passes on the target hardware profile, not just a cold-start latency number, specifically because that number has been the wrong number more than once.

The spec sheet tells you what the hardware can do for one query. The deployment lives or dies on what it can do for the six-hundredth query that hour.

Where we've landed, for now

For most field-diagnostics-class deployments we're currently running mixed 4/8-bit configurations, task-calibrated, with attention layers held at 8-bit — a compromise that's held up across the ruggedised deployments we've run it on so far. That's not a universal number; it's where our evaluation process currently lands for that specific class of hardware and task. The next node profile, or the next task, gets re-evaluated from the eval set up, not assumed from this post.