Does that hand have a glove on?

A press has a danger zone. A hand enters it. The only question that matters is whether that hand is wearing a glove — and the answer has to arrive fast enough to be worth having, and be right often enough to be worth trusting.

Those two requirements pull in opposite directions, and they are not symmetric. A false alarm stops a machine that did not need stopping, and a line that cries wolf gets switched off by the people it protects. A missed alarm injures somebody. So this is not a "get the F1 up" problem; it is a problem about which mistakes you are willing to make.

I built GloveWatch to find out how far you can get with pretrained models only — no dataset collection, no training run, nothing to label before it works at all.

The trap that shapes everything

The obvious first move is to find hands with a hand detector. MediaPipe Hands and its relatives are excellent, fast, and free.

They are also trained overwhelmingly on bare hands, and they degrade exactly where this application needs them most. Put a thick work glove on and the landmark model starts missing the hand entirely — which produces the worst possible failure mode for a safety system: the gloved hand is invisible, so the only hands the system sees are the bare ones, and the only thing it can conclude is what it already assumed. A detector that can't see gloves makes a glove-detector look like it works.

So the glove is never allowed anywhere near the job of finding the hand. Localisation comes from body pose only:

1. YOLO26x-pose gives 17 COCO keypoints per person — including elbows and wrists.
2. The hand is placed by extending the elbow→wrist vector past the wrist by 0.6 of the forearm length.
3. A square box 1.5× that extension, floored at 48 px so distant hands survive.

Nothing in that chain looks at skin. A glove, a bandage, a sleeve pulled over the hand — the box lands in the same place, because it is inferred from the arm.

Classifying without a dataset

With a hand box in a zone, the actual question is one binary call on a small crop. The first version does it zero-shot, with a SigLIP 2 prompt ensemble — six phrasings for each class, scores averaged:

glove_prompts:
  - "a photo of a hand wearing a work glove"
  - "a gloved hand"
  # …
bare_prompts:
  - "a bare human hand"
  - "a hand with no glove"
  - "a close-up photo of bare skin on a hand"
  # …

Each hand is inspected at two scales — a tight crop and one padded by 25% — and the two probabilities averaged, because a crop tight enough to be mostly hand often cuts off the cuff, which is the most glove-looking part of a glove.

This works on the first run with nothing trained. That is the entire point of the zero-shot tier: the system is useful before anybody has labelled anything, and only then does it earn the right to ask for data.

The upgrade, and what it honestly bought

The accuracy endgame is few-shot: collect in-zone crops from real footage, sort them into two folders, embed them with a frozen DINOv3 backbone and fit a StandardScaler + LogisticRegression on top. The app picks up models/glove_head.pkl automatically when it appears.

I collected 386 crops — 287 bare, 99 gloved — and trained:

CV accuracy: 0.837 +/- 0.019
folds:       0.833, 0.831, 0.857, 0.805, 0.857

83.7%. For a safety system, that number is not good, and I would rather say so plainly than dress it up. The reason is visible the moment you look at what the classifier is actually being handed:

These are small, motion-blurred, oddly lit patches from a hand that is in motion — a hand at rest is not the interesting case. On top of that, all 386 came from one person, in one room, wearing two pairs of gloves. A model fitted to that has learned my kitchen lighting as much as it has learned gloves.

The honest reading is that 83.7% is what this dataset supports, not what the method supports. The fix is more and more varied data, from the actual line — not a better classifier head.

Why one frame is never allowed to decide

A classifier at 84% firing on every frame at 30 fps would produce roughly five wrong calls a second. No machine can be driven from that.

So no single frame is permitted to conclude anything. A sliding window holds the last 9 in-zone frames for each tracked hand, and a verdict requires 7 of them to agree. Tracks are associated between frames by wrist distance, the window resets when a hand leaves the zone, and once an alert fires it latches until the hand has genuinely left — a hand hovering on the zone boundary should not be able to flicker an alarm on and off.

That buys a great deal of stability and costs latency, and the cost is measurable from the event log:

Median time from a hand entering the zone to NO_GLOVE_ALERT is 292 ms, against a hard floor of 7 frames at 30 fps = 233 ms. The design and the measurement agree, which is the most reassuring thing in this whole write-up — the voter is doing exactly what it says and adding almost nothing on top.

GLOVE_OK is slower: median 511 ms. Both verdicts need 7 of 9, so the asymmetry has to come from the classifier — gloved frames are classified less consistently than bare ones, so the window takes longer to fill with agreement. Which is the same 99-vs-287 imbalance in the training set showing up again, one layer down.

The number I actually worry about

Panel B is the one that matters for deployment, and it is not the median.

Alert confidences run from 0.566 to 1.000. A third of all alerts — 102 of 308 — fired below 0.8, and 13% below 0.7. The threshold is 0.5, so the bottom of that tail is a decision made on barely more than a coin flip, and it is a decision that stops a machine.

Verdict latency

292 ms

Head CV accuracy

83.7%

End-to-end

~20 FPS

Alerts under 0.8

33%

The median tells you the system works. The tail tells you what it will do on its worst day, and on a safety system the worst day is the whole specification. The obvious next move is not a better model — it is refusing to act on low-confidence verdicts at all, and escalating them to a human instead of to the machine stop.

Where it stands

It runs end-to-end at about 20 FPS on an RTX 4080, against a target of 15, with YOLO26x-pose exported to a TensorRT FP16 engine at 960 px. That engine is built for that specific GPU and has to be re-exported for any other one, which is easy to forget and confusing when it bites.

What it is not is finished. Everything above was measured in a room, by one person, waving one pair of gloves at a webcam. The pipeline holds up and the timing is real, but the classifier has only ever seen my hands. That is the next job, and it is a data job, not a model job.

The part I would build the same way again is the first decision — refusing to locate hands with anything that can see the glove. Everything downstream can be improved by feeding it better data. That one could not have been fixed later at all.