The uncomfortable thing about running models on edge devices is that the failure you care about most doesn't look like a failure. The process is up. Latency is normal. CPU and memory are unremarkable. Every check you have is green, and the model has been quietly wrong for three weeks.
In a data centre you'd catch this eventually - someone samples predictions, compares against labels, runs an eval set. At the edge you usually have none of that. No labels, limited bandwidth, and a device you can't easily interrogate. So the question becomes: what can you compute on the device, cheaply, that tells you the model has changed behaviour?
Ground truth isn't coming
Start by accepting the constraint. You are not going to label field data at any useful rate. Whatever you build has to detect change rather than correctness - that the model is behaving differently than it was, not that it's wrong in some absolute sense.
That sounds weaker than it is. Most real regressions show up as a distribution shift long before anyone notices a business impact, and you already have a reference: the same device, the same location, last month.
Four signals worth computing on-device
Confidence distribution. Keep a rolling histogram of output confidence. Not the mean - the shape. A detector whose confidence mass drifts downward is telling you something changed, in the model or in what it's looking at.
Per-class prediction rate. How often each class fires, per hour. A camera that detected 400 objects a day last month and detects 250 now has either a quieter environment or a blinder model.
Empty-output rate. How often inference returns nothing at all. This one is blunt and it catches a surprising share of real breakage - a preprocessing mismatch after an update often manifests as silence, not errors.
Input statistics. Mean brightness, contrast, blur estimate. Cheap to compute, and they separate "the model got worse" from "someone moved the camera or a light burned out." Without this you'll spend a week debugging a model problem that's a maintenance problem.
All four are histograms or counters, not raw data. That matters: a device on a metered link can send a few hundred bytes an hour and nothing else.
Comparing distributions without doing statistics badly
You need a number that says "these two histograms differ." Population Stability Index is the usual choice - well understood, cheap, and its conventional thresholds are a reasonable starting point: below 0.1 means no meaningful shift, 0.1 to 0.2 warrants a look, above 0.2 means something genuinely changed. KL divergence works too and is more sensitive to small differences in low-probability bins, which may or may not be what you want.
Two things that matter more than the choice of metric. First, compare a device against its own history, not against the fleet average - devices see different things, and fleet-wide comparison generates noise forever. Second, use a long enough window. Hourly comparisons on a camera will fire every day at dusk. Daily windows against the same weekday are usually the smallest unit that isn't dominated by normal cycles.
Thresholds that survive contact with a real fleet
The first version of this always alerts too much. Some defences:
Require a shift to persist across multiple consecutive windows before alerting. One bad afternoon is weather.
Alert on fleet-level patterns separately from device-level ones. Five devices drifting together after a rollout is a model problem. One device drifting alone is usually a hardware or environment problem, and it's a different ticket for a different person.
Suppress device-level alerts during known environmental events if you can - you'll learn which ones matter within a month of running this.
Send the interesting cases, not the data
The natural next thought is "let me look at what it's seeing." Resist sending everything. A reasonable pattern is uncertainty sampling: when the model's confidence sits in an ambiguous band, keep that frame. Ship a small number of them on a schedule, when the device is on a cheap link.
You end up with a slow trickle of exactly the examples that are hard, which is what you'd want for retraining anyway. And review remains a human decision - an ambiguous frame is not a label.
The part nobody builds until it's too late
All of this only works if you kept a baseline. Before any rollout, you want at least a couple of weeks of these four signals under the model currently in the field. Without that, the first time something regresses you'll be comparing to nothing and arguing from intuition.
It costs almost nothing to collect and it's the difference between "the model changed on day 3 across the twelve devices we updated" and "something feels off, can we roll back and see?"
I'm running a live cohort on operating edge-AI inference in production - fleet rollouts, observability on devices you can't reach, hardware selection. Four sessions, Wednesdays from November 4. Details and dates.
What do you use for this? I'd genuinely like to know what other people are computing on-device - especially anyone doing it on hardware tighter than a Jetson.