Consider a resistance-training setup: a user takes hold of the handles, the equipment provides resistance, a phone captures video, and a screen displays the workout. Research into systems that understand training begins with a more basic question: what exercise is this person doing right now?

The answer shapes every step that follows. Each exercise has its own rhythm, force-production characteristics, and points to observe. The system must first understand the current exercise before it can identify repetition cycles, decide which body parts to focus on in movement analysis, or organize the data into workout logs and reports. Beyond displaying an exercise name, recognition determines how the capabilities that follow should proceed.

Exercise recognition also has to happen in time. Analyzing video after a workout can help organize records, but it cannot contribute to the set in progress. To be useful in a product, exercise understanding must keep pace with the user and deliver feedback while it can still help.

At first, this sounds like a straightforward visual classification problem: if a camera can see the user, why not ask a general-purpose vision-language model (VLM) to identify the exercise? Real workouts are more complicated. A user may adjust an attachment, test the resistance, pause, and only then begin a working set. Several exercises may share almost identical underlying movements. The same exercise can look different depending on the user's height, range of motion, speed, camera angle, and occlusion. Meanwhile, the entire capability has to run continuously on the device without interrupting the workout or making the user wait.

Taking these factors together, our research explores exercise recognition as a continuous process of understanding training in real time. As video enters the system, four things need to happen in sequence: establish that training is taking place, understand the main movement, distinguish exercise variations, and provide stable results in time for feedback. Each stage addresses a different problem. Together, they explore how ordinary video can become exercise and training-state information that supports smart fitness equipment.

Through these four stages, we turn continuous video into real-time results for interface feedback, workout records, and subsequent analysis by an agent.

1. Before identifying the exercise, establish that training is happening

When a camera captures continuous footage for exercise recognition, that footage includes noise and unrelated activity. The actual exercise occupies only part of it.

Consider a standing curl. The user walks into view, bends down to adjust an attachment, stands up, makes a test pull, performs a set, and finally puts down the handles. All these moments involve a person, equipment, and visible changes, but they mean different things. A system looking at a single frame might mistake preparation for a working set, or interpret a brief pause as the end of an exercise. Getting these training boundaries wrong affects more than whether an exercise name appears a second earlier or later. Repetition counting, logs, and reports inherit the error: a test pull might be counted as part of the working set, or attachment adjustments might be treated as part of the exercise. Even a correct category is of little use when assigned to the wrong time interval.

Our first research decision was therefore to treat changes over time as evidence. The system considers changes across preceding and subsequent frames to determine when the user has actually begun training and whether the movement is continuing. This produces a training sequence with a beginning, continuity, and changes over time. Frame-level labels become meaningful within that temporal context.

Reliable observation has another easily overlooked prerequisite: the camera must be in a usable position. When filming training with a phone, checking its position first helps establish whether the system can see what it needs to see before attempting exercise recognition. Separating camera positioning from exercise recognition prevents the model from guessing from an unreliable view from the outset.

The training data also needs to reflect the factors that affect real workouts.

First, we found that general-purpose footage of everyday life can teach a model to recognize people and motion, but rarely covers the attachments, cables, benches, and exercise names used in smart fitness. We therefore need to keep collecting data around the exercise taxonomy and use motion-capture equipment to obtain human-motion annotations synchronized with the video.

Second, collecting only textbook demonstrations would be easier: clean footage, full ranges of motion, and consistent tempos naturally lead to attractive offline results. But someone trying an exercise for the first time may use a shorter range of motion, vary their speed, or reposition their body while applying force. How the system handles these moments, which look different from demonstration videos, determines whether the capability can work in real product settings.

We therefore care more about covering real variation than making every sample look perfect. This data helps the model cope with frame-rate fluctuations and imperfect exercise form. The model learns how an exercise can unfold in practice, rather than simply memorizing an ideal demonstration.

2. Understanding the underlying movement takes more than memorizing a reference image

Once training has been established, the next question is which exercise the user is doing. Fitness exercises, however, are not always as easy to distinguish as walking, running, and jumping. They often share the same underlying movement. Standing bar curls, bilateral handle curls, rope hammer curls, and reverse-grip curls can all involve repeated arm flexion and extension. Conversely, the same standing curl can look different because of range of motion, tempo, compensatory body movements, and camera angle.

This is a central difficulty of exercise recognition: different exercises may look too similar, while different people performing the same exercise may look quite different.

Exercise recognition must account for similarities between exercises, variation within an exercise, changing camera conditions, and uncertain training boundaries.

Given these differences, what evidence should the system use? Body trajectories alone may miss the grip, attachment, and equipment position. Visual appearance alone is vulnerable to clothing, backgrounds, lighting, and occlusion. Our second research decision was therefore to use complementary sources of information: one describes how the body moves over time, while the other preserves the relationship between the user, equipment, attachments, and surroundings.

This combination is especially useful when information is incomplete, because the two sources are not always equally clear. When part of the body is hidden, continuous movement may still reveal the underlying pattern. When movement trajectories are very similar, equipment and grip may provide the distinction. The system needs to use the evidence currently available so that the two sources complement each other, without expecting either one to solve every problem alone.

Combining the two sources does not mean every detail in an image should affect the prediction. The system still needs to distinguish which information deserves more trust. Clothing color, walls, and lighting may change from one workout to the next; they should not readily change the exercise prediction. Persistent movement relationships between the body and equipment are closer to the exercise itself. The system therefore needs to use the richness of the image while avoiding shortcuts based on incidental background features. Distinguishing stable evidence from incidental cues matters more than memorizing additional reference images.

With this more stable evidence, body motion and visual information can together establish a reasonably reliable range of possibilities: is the user mainly pushing, pulling, squatting, or performing some type of curl? Once that context is established, subsequent decisions can follow a more focused line of inquiry instead of comparing the movement against every exercise.

3. For highly similar exercises, narrow the question before answering it

After establishing the main movement, the next step is to distinguish specific exercises within that movement family. The taxonomy used in our research therefore has two levels: broad categories and subcategories. Dozens of broad categories are defined by posture and movement pattern, then divided into more specific subcategories according to equipment and execution.

The harder task is distinguishing small differences between categories. A change in grip, attachment, or cable exit position may be enough to change the exercise type. When a user's form is imperfect, otherwise clear boundaries become less distinct. Asking a lightweight system to choose from all subcategories in one step, without separating these levels of judgment, means combining questions with different levels of difficulty and different evidence requirements. It must understand the main movement, recognize equipment details, and cope with variations in real users' movements at the same time. As categories become more specific, similar options increasingly interfere with the decision.

For example, when the user's hands are briefly hidden behind their body, the system may already be able to establish that they are performing a standing curl without having enough information to determine the grip. Forcing a complete exercise name at that point is no better than retaining a reliable broad-category judgment. Exercise understanding needs to recognize how specific an answer the current evidence can support.

Our third research decision follows from this: work from coarse to fine. The system first identifies a relatively stable exercise family, then examines the details that distinguish subcategories within that family. For the small number of exercises that depend more heavily on equipment and semantic information, the system can bring in stronger visual understanding within a controlled scope, without asking it to guess freely across the entire taxonomy.

This hierarchy follows the different evidence requirements of broad categories and subcategories. Broad categories depend more on sustained movement structure and are suited to the primary real-time decision. Subcategories depend more on grip, attachments, and execution details, which can be confirmed within an established exercise context. Establishing that context before interpreting the details therefore reduces interference between similar exercises and applies stronger semantic capabilities where they are needed.

Beyond distinguishing similar exercises, this hierarchy also suggests a useful interaction pattern: the main judgment can keep working even when details are temporarily unclear. As evidence accumulates, the system progressively refines a stable broad category into a specific exercise. This gradual refinement fits the way a set unfolds over time. The system first establishes the standing-curl movement family, then uses finer details to identify the variation.

4. Deliver results in time and keep them stable

Hierarchical judgment explains how an exercise can be identified progressively. For the result to contribute to training in progress, it must also arrive in time. An exercise name that appears long after the workout is over may still be a correct label, but it can no longer function as a real-time product capability.

Meeting this requirement means considering the whole process from observation to feedback. When a phone camera is used to observe training, the phone can also be the real-time entry point for exercise understanding. The research challenge is to keep observation, interpretation, result stabilization, and potential interface responses working together throughout the workout. Meanwhile, the phone must handle video, interaction, and other tasks as available compute, memory, and temperature change. Users will not hold a textbook pose while waiting for a model.

A fast individual computation, however, does not guarantee a real-time system. If incoming frames keep accumulating, the model may accurately identify an exercise from several seconds ago while missing the fact that the user has already stopped or switched exercises. During a workout, a late correct result can be as confusing as an incorrect one. We therefore measure whether results reflect the user's current state as well as how quickly they are computed.

To address these delays, our fourth research decision was to design the entire pipeline around timely, stable results. With a continuous stream of incoming frames, the system prioritizes the user's current state. Keeping up with the current movement also requires handling brief fluctuations: the system smooths information over time to maintain a stable conclusion and avoid switching exercises because of a single noisy observation. The intended experience is an interface that avoids flickering between similar exercise names or continuing to describe the previous exercise after the movement has changed.

Achieving both timeliness and stability requires a balance between sensitivity and conservatism. If the system is too sensitive, exercise names fluctuate with occlusion and posture changes. If it is too conservative, it updates too slowly after the user switches exercises. A usable real-time experience must therefore respond to actual changes without turning every brief disturbance into a new conclusion.

Alongside balancing update speed and stability, the system needs to prioritize different tasks. In the research pipeline, we separate the primary real-time decision from more detailed semantic enhancement: core exercise information is generated continuously on the phone without depending on a network round trip. Finer classifications that need more semantic information can follow, but must not block the interaction in progress. This establishes priorities among accuracy, stability, and timeliness.

Through these choices across the pipeline, our research aims to turn continuous phone footage into structured information about exercises and training states. If these results are used for feedback, the model should run in real time and produce information as the user's workout unfolds, without requiring users to change their training pace to accommodate the recognition algorithm.

5. From identifying an exercise to understanding the whole workout

Looking at the workout as a whole, these four stages form an approach to understanding training: when a set begins, what the main movement is, which specific variation it may represent, and how long that interpretation remains valid. Attaching a label to a video is only the most visible result.

In end-to-end, on-phone research testing covering nearly 100 exercise subcategories, real-time recognition accuracy exceeded 90% on both iOS and Android. To relate that figure to a workout, consider that a one-hour session typically includes four to five exercises. A rough conversion from the overall accuracy suggests fewer than 0.5 misclassifications per session on average. This estimate suggests the possibility of a more continuous experience when using exercise recognition, with fewer stops to operate the interface at each exercise change. That continuity is the goal: results need to be correct, stable, and available in time during training.

Continuous recognition also provides information for deeper analysis. Identifying what the user is doing creates a shared foundation for movement analysis and form correction: knowing the current exercise tells the system which repetition-cycle logic to use, which bodily changes to observe, how to organize workout records, and what context to preserve for reports.

Real-time exercise information provides context for repetition counting, movement feedback, workout records, reports, and subsequent analysis in a coaching-agent workflow.

Among these downstream capabilities, a coaching agent is especially dependent on understanding the workout as it unfolds. Moving from answering questions to participating throughout a workout requires a reliable starting point for perception: alongside knowing what the user has done in the past, it needs to understand the current exercise, whether the set is continuing, and whether the next piece of feedback can still make a difference. Real-time exercise understanding is one foundation for these capabilities. It offers a path toward continuous analysis during training, from seeing a person in motion to understanding which exercise they are performing. This is a starting point for fitness AI that draws on multiple kinds of information.