AI & ML Engineering

Flick

I built Flick to distinguish gestures by their paths through time, even when they pass through a similar hand pose. I process circle, swipe-right and zigzag trajectories and connect an accepted gesture to a visible in-app action.

Following the trajectoryExplore the interface ↘
Project
Hand-trajectory classification and stateful action triggering
My role
I built trajectory preprocessing, two classifiers, the trigger state machine and local replay.
Year
2026
Contents

Reading the movement between poses

I built Flick around a small interaction problem: a hand can hold a similar pose while tracing a circle, moving to the right or drawing a zigzag. I wanted to carry that movement through a local recognition pipeline and show what happens when its decision reaches an interface.

The intended interaction is easy to describe. Someone makes a movement, the application accepts it and one visible action appears. Making that behavior dependable involves several decisions before the action: how to represent the path, when to accept a label and how to keep the same episode from firing repeatedly. I kept the scope to one hand, three gestures and background.

Following one circle through replay

I use a circle clip to make the behavior concrete. In the local application I select the trajectory RandomForest backend and replay the clip. The interface displays the predicted label, draws the resampled index-finger coordinates and adds a decision to the trace. An accepted episode can produce the in-app action called Highlight circle.

Pressing Replay again grows the decision history while the action count stays at one. Classification can happen several times during one episode. I added Hand absent and Release / rearm so the reader can supply an explicit background observation, then start another episode after cooldown. Reset replay has a broader purpose: it clears the presentation clock, trace and accumulated actions together.

A creature holding a latch after allowing one token through
I used this small latch to illustrate one action from a continuing gesture.

Giving each decision a visible place

I organized the Streamlit interface around replay controls and their consequences. Model selection and clip selection stay explicit. The optional JSON field lets someone use a raw request instead of the selected fixture. Below them, the current result and coordinate chart explain that particular decision; the trace and action count retain what happened across requests.

This arrangement makes an otherwise hidden state visible. A recognized circle with no new action can be a correct held-episode result. The message explains that background, a held episode or cooldown can prevent another action. I also show measured evaluation separately from the current replay. Switching a clip should not make an old benchmark look like a fresh measurement, and seeing a label should not imply another application has received a command.

Application screens

1 / 3
Following the trajectory

I replayed recorded landmarks so the trajectory and accepted gesture could be inspected together.

Open full-size image

Keeping the displacement in the coordinates

The representation needed to preserve the movement I was trying to recognize. I subtract the wrist position from the first frame and divide by the initial palm scale, measured between the wrist and the middle-finger base landmark. Later positions retain their displacement relative to that starting reference.

Subtracting the wrist independently in every frame would keep the hand locally centered and remove much of the trajectory. For a swipe, that would discard the information needed to distinguish movement from a similar stationary pose. I use the index-tip path after normalization. The input still has a defined coordinate convention, with unmirrored image-normalized coordinates and downward y. I have not made the representation rotation invariant, so orientation remains an explicit part of its operating conditions.

Separating the clocks

I resample the path to 64 steps using its timestamps. That gives both classifiers a consistent representation while keeping the original elapsed duration available. Velocity statistics use elapsed seconds, so an uneven sampling rate is not mistaken for a faster movement.

There are three clocks in this application, and I keep their jobs separate. Source timestamps describe the recorded trajectory. The replay presentation clock orders completed windows and controls the trigger. Wall-clock measurements describe computation. A scheduled replay window does not stretch the underlying motion to fill its display duration. This distinction also explains the interface's scope: Replay submits a complete clip for classification. Detecting where a gesture starts and finishes inside an ongoing camera stream would need another component and suitable event annotations.

Starting with a path I could inspect

For the baseline I used Dynamic Time Warping against 36 foreground training templates. DTW compares the index-tip paths while allowing their progression through the sequence to differ. I divide the accumulated Euclidean cost by the sum of the sequence lengths, then use a distance threshold to reject a query that is too far from the templates.

Alongside a label, the nearest template can explain which stored movement the query resembled. The calculation uses the path itself; it does not retrieve an answer from the sample name. I wanted that reference before adding a learned classifier, because it provides a concrete way to ask whether the additional model changes speed, acceptance behavior or the mistakes that remain.

A compact learned comparison

The learned method is a seeded RandomForest with 100 trees. I give it 202 features: the 64 resampled three-dimensional path positions, axis-velocity summaries, speed summaries and duration. This keeps the input tied to the movement and lets the model run on CPU without a downloaded visual backbone.

A foreground winner still needs to pass its acceptance threshold. Otherwise the result becomes background. I fit the models on training data and choose the thresholds on validation data before scoring the held-out split. The saved artifacts include configuration and source identities, so a mismatched model produces an error when loaded. Prediction and the interface never silently train a replacement. I can inspect the behavior of that saved comparison.

Letting a held gesture remain one episode

I separated recognition from the decision to emit an action. The trigger has a foreground latch and a fixed one-second cooldown. Once a foreground episode arrives, the latch records that it has been consumed. Repeated foreground predictions remain inside that episode even after cooldown has elapsed.

Changing from one foreground label to another also leaves it latched. Background or an explicitly absent hand releases it. An onset that arrives during cooldown is consumed until another release, which can suppress a closely spaced intended action. I made that policy explicit in the UI and tests. It gives the replay a predictable state transition, while leaving natural gesture segmentation as a separate problem to investigate with continuous recordings.

Building a repeatable experiment

I generated 144 synthetic clips, each containing timestamps and 21 three-dimensional hand landmarks per frame. The split has 48 training, 48 validation and 48 test clips, with 12 examples for each gesture or background class in every split. Sample identifiers and split labels stay outside the inference features.

The session IDs are disjoint, although all 15 analytic shape prototypes recur across them. I treat this as a controlled test of variation around shared shapes. In the recorded experiment both methods scored the same test IDs after their thresholds had been frozen. For background actions I used 117 scheduled windows covering a 60-second interval union. They reuse 12 clips with 16.947564 seconds of unique source material, which matters when interpreting the zero-event result.

Keeping the rejected zigzag in the result

The recorded four-class macro F1 was 1.0 for DTW and 0.979130 for RandomForest. The difference comes from one zigzag. Its raw RF winner was correct at 0.77 confidence, but the frozen acceptance threshold was 0.8, so the final output was background. DTW accepted it. I retained that miss and the frozen threshold.

The warm median computation times were 10.90 ms for DTW and 5.66 ms for RF; their p95 values were 11.42 and 5.85 ms. These cover 25 validation, preprocessing and inference operations after warmup, excluding capture duration and UI rendering. Both emitted zero actions on the scheduled background test. I could weigh RF’s lower computation time against its extra rejection.

Making bad input recoverable

I kept error handling in the same path used by replay. An empty JSON array is a useful example: the request expects an object, and entering [] produces a contextual message instead of a plausible prediction. Clearing the field and releasing the episode lets the user recover without restarting the application.

The core also checks numeric landmarks, strictly increasing timestamps and usable palm geometry. An absent hand is an explicit background observation; malformed geometry is an error. Missing artifacts and configuration mismatches name the preparation problem. These distinctions prevent a failed request from becoming a believable action. I share the loading, inference and trigger code between CLI and UI, so the graphical demonstration exercises the same rules as the command-line workflow.

What I would measure with real recordings

Flick taught me to inspect the representation and action state alongside the classifier score. A preserved path makes the comparison meaningful; a visible latch explains why another prediction may correctly produce no action. The replay and saved reports let me inspect both when interpreting a mistake.

The next useful input would be separately acquired human recordings, split by whole session, with varied background movement and annotated positive-event boundaries. Those annotations would let me measure event recall and completion-to-trigger delay. Camera capture has not been executed in this project. For the existing local work I preserve the original experiment and use separate reproduction outputs, so later tests can develop the system without overwriting the evidence that describes this version.

Available for work