How the watch finds a set, names it, and counts it

Lifting is hard for a watch. A session is mostly rest, exercises differ by small rotations of the forearm, a rep count is only useful if it is exact, and all of it has to run on the wrist because the phone is in a locker. This is how Rheo handles that, and how well it works so far.

100%

of sets named correctly

87.7% for hand-built features

92.7%

of sets found without a tap

91.6% of what it finds is a real set

71.4%

of sets counted to the exact rep

35% for the best classical counter

Held-out sessions and workouts, 84 sets from one lifter. What that does and does not show is further down.

Labelled by the lifter

The training data comes from the product. A tap starts a set and another ends it, and the lifter names the exercise and logs the reps, so every logged set is a labelled example with boundaries placed at the moment it happened.

The watch records the whole workout at 200 Hz, rest included, because the rest is what the detector learns to say no to.

1Barbell Bench Press, 8 reps logged
2Barbell Overhead Press, 6 reps logged
3Dumbbell Biceps Curl, 12 reps logged

A synthetic recording, drawn to show the labels. This is the replay fixture the Swift and Python decoders are tested against, generated rather than recorded, at 115 s long. Real recordings are a continuous trace of a person’s body and stay in private storage. The trace is the magnitude of acceleration with gravity removed; colour marks what falls between a start tap and an end tap.

Labelled sets: 84
Workouts: 13
Exercises: 10
Logged reps: 649
Recorded: 3.1h

All from one lifter’s own watch. That is the biggest limit on everything below.

Cleaning it

Each recording is low-passed at 20 Hz and resampled to 50 Hz, keeping the 6 channels public datasets also have. Sets whose logged reps could not physically fit the tapped span are skipped, and the 1.5 s around every tap is left out of the detector’s loss: a tap is a hand reaching for the wrist, not the first rep.

The most useful bug was a sign. RecoFit, a 94-subject forearm dataset, transferred at 35–45%. Its armband reads the forearm’s long axis with the opposite sign to the Apple Watch, and one fixed rotation took zero-shot accuracy to 78–100%.

Push-up

Shoulder press

Lateral raise

Biceps curl

Squat

Mean gravity along the forearm’s long axis, per exercise. RecoFit’s armband reads a hanging hand as negative; the watch reads it as positive. The fix is one fixed rotation, applied to acceleration and rotation rate together, worked out from Apple’s documented frame and RecoFit’s own data without using any of our labels.

Small models, in stages

A causal network decides, every 20 ms, whether a set is under way. A second names the exercise from the samples inside it. A signal-processing counter counts the reps with a configuration chosen for that exercise. Each stage had a classical baseline to beat before a network could replace it, which is why the counter is still signal processing.

Core Motion200 Hz, 13 channels
Anti-alias and resample50 Hz, 6 channels
Set detectorCausal TCN, P(set) every 20 ms
Set ruleEnter at 0.6, leave at 0.4
Exercise classifierTCN over 6 s windows
Rep counterTurning points; the classifier picks its config
ProposalExercise, reps, last weight; confirm or correct

Why temporal convolutions. Both networks are stacks of dilated 1-D convolutions, so how far back each one sees is set by its dilations and can be matched to the movement. The classifier sees about a rep. The detector sees 10.1 s of the past, enough to tell a set from walking to the rack, and never looks ahead.

Together they are 52k parameters and 350 KB on the watch, with no hidden state to drift over an hour-long session.

Set detector causal, k = 5, width 48

d = 1

0.18 s

d = 2

0.50 s

d = 4

1.1 s

d = 8

2.4 s

d = 16

5.0 s

d = 32

10.1 s

Exercise classifier centred, k = 5, width 32

d = 1

0.18 s

d = 2

0.50 s

d = 4

1.1 s

d = 8

2.4 s

What each block can see, relative to the frame it is answering about. Each row is one residual block of two dilated convolutions; the last row is the whole network. The detector ends at 10.1 s of past signal (505 samples at 50 Hz), enough to see that movement is rhythmic; the classifier ends at 2.4 s (121 samples), about one or two reps, and pools over a 6 s window.

How well it works

Every split holds out whole sessions or workouts, and every model is scored next to the baseline it had to beat. On the boundaries the detector finds, rather than the ones the lifter tapped, the classifier is still 95.5% correct, and end to end the rep count is exact on 49% of sets, most misses one rep short.

Sets named correctly

Temporal network Hand-built features

Leave one session out (18 folds)100%87.7%
Leave one workout out (3 seeds)100%84.6%
Temporal: oldest 16 sessions train, newest 8 test (3 seeds)97.2–100%77.8%
Per-class 80/20 holdout, not groupedoptimistic100%86.2%

Five exercises, on the boundaries the lifter tapped. A range spans three training seeds, and the bar shows the lowest. The per-class holdout splits sets rather than sessions, so it is optimistic.

Sets counted to the exact rep

Peak picking8%
Autocorrelation10%
Matched filter35%
Turning point, one config for every exercise58%
Turning point, config chosen by the classifierships71.4%
Turning point, told the true exerciseoracle70%

All 84 logged sets, leave one workout out, on the boundaries the lifter tapped. The first three are classical baselines. The oracle is told the true exercise, which a watch never is, and the shipped counter matches it.

What these numbers are not

They come from one lifter, so none of them says how the models do on a new person; that needs the first users. The classifier knows 5 exercises and forces anything else into one of them. With 84 sets, the intervals are wide.

What did not work

Pretraining the detector on public data made it worse (85.4% against 92.7%). A learned rep counter reached 42% exact against signal processing’s 71%: it had learned how long a set lasts, not how many reps it holds.

The next model learns from the first users

Every set a new lifter confirms or corrects is a labelled example from a wrist the models have never seen.