How quantized models get compiled and run continuously

You can't ship one artifact to every system — the compiled graph is tied to the FPGA fabric and its precision.

The flow is always quantize → compile → verify: take a portable ONNX, quantize to the precision the accuracy budget allows, and compile with Vitis AI to the target's overlay or .xmodel. Because compilation is ahead-of-time, cold-start is instant — the device just loads the graph.

A standard OTA-for-models flow, hardened for the field:

1. Model registry + object store / CDN — signed, versioned artifacts in a private S3 / R2 bucket, fronted by our Rust backend.
2. Device pull, not push — each device's agent polls GET /models/manifest?device=<id> for the models assigned to its hardware target. Pull wins for fleets that are intermittently online.
3. Signed + hashed artifacts — every blob is Ed25519-signed and SHA-256-verified before it is ever loaded. Non-negotiable for anything safety- or defense-adjacent — it's the supply-chain integrity gate.
4. Resumable / delta downloads — HTTP range requests so a dropped link resumes; ship deltas when only a fine-tune changed.
5. A/B model slots — download into an inactive slot, verify, then atomically flip. Keep the previous slot as last-known-good for instant rollback.

A long-lived inference daemon — a systemd service on the edge module, a host process alongside the datacenter card — that:

• Loads once, stays warm. The active overlay lives in a persistent context; frames stream through it. Re-loading per frame is the classic mistake that kills throughput.
• Hot-reloads without a gap. Configure the new graph into a second context, warm it, swap the pointer, free the old one (double-buffering). No process restart, no blind spot in perception.
• Runs a watchdog. If the new model faults or a latency / accuracy canary regresses, auto-revert to the last-known-good slot.
• Bounds resources. Pin the DPU / dataflow region and cap the power envelope — on a drone or a vehicle, thermal and power are hard limits; in the rack, tail latency is the SLA.

• Fetch on a link, run without one. Sync models whenever a drone is at base or a vehicle is on Wi-Fi; in operation the device is autonomous on whatever is cached. Manifest reconciliation happens opportunistically when a link returns.
• Encrypt at rest + secure / measured boot so a recovered airframe or a salvaged ECU doesn't leak models or weights.
• Telemetry back-channel — report active model version, inference latency, and accuracy canaries per device so you can manage the fleet and stage rollouts.