Research · Chang Gung University Intelligent Control Lab · Aug 2024 – Jun 2025

Dynamic Hand-Gesture Recognition with MobileNet + GRU

Static gesture classifiers look at one frame at a time, so they can't tell a left swipe from a right swipe. This model reads the whole clip. A pre-trained MobileNet extracts what each frame shows, and a GRU learns how it moves.

Test accuracy: 85%
Baseline (RNN + GRU): 66%
Test loss: 0.342 vs 1.1513
Presented: Poster, 20th Formosa Plastics Applied Tech. Seminar

Results: explore the confusion matrices

These are the real test-set results from the paper. Switch models to see where the baseline breaks down. It mixes up the swipes and pauses, while MobileNet + GRU's only real weakness is left swipes.

27/32 correct · 84.38%

Hover a bar to inspect it. Drag to rotate. Blue = correct, orange = mistakes.

Per-class precision / recall / F1

Class	Precision	Recall	F1	Confusion row (pred. 0–4)
Left swipe	1.00	0.50	0.67	[4, 2, 0, 2, 0]
Right swipe	0.67	1.00	0.80	[0, 6, 0, 0, 0]
Pause	1.00	1.00	1.00	[0, 0, 6, 0, 0]
Thumbs down	0.75	1.00	0.86	[0, 0, 0, 6, 0]
Thumbs up	1.00	0.83	0.91	[0, 1, 0, 0, 5]

Approach

01
Frames
Each 30-frame clip trimmed to frames 7–26 and downsampled to 96×96.
02
MobileNet
Pre-trained CNN, wrapped in TimeDistributed, extracts spatial features per frame.
03
Compress
Pooling halves height and width, then features are flattened per time step.
04
GRU (128)
Models temporal order. Dropout of 0.3 limits overfitting.
05
Softmax
Five classes: left/right swipe, pause, thumbs down/up.

Why this design: transfer learning from MobileNet gives strong spatial features without a large dataset, and MobileNet is light enough to point toward real-time, on-device use. GRUs capture temporal dependencies with fewer parameters than LSTMs and avoid the vanishing gradients of plain RNNs.

No data augmentation, on purpose: flipping or reversing a clip would turn a left swipe into a right swipe. Direction is the label.

Data & training

Open Kaggle dataset (Hand Gesture Detection System): 763 clips, each 30 frames covering 2–3 seconds, recorded waist-up.
Roughly 84 / 12 / 4% train / validation / test split, randomized and balanced across classes, with early stopping on validation.
Optuna Bayesian search (Tree-structured Parzen Estimator), 50 trials, picked: GRU 128 units, dense 64 units, learning rate 0.001.
Cross-entropy loss, batch size 32, trained on an NVIDIA RTX 4060 in about 15 minutes.

MobileNet+GRU training and validation accuracy rising to about 0.95 over 30 epochs — MobileNet + GRU accuracy: train and validation track closely.

RNN+GRU accuracy curves with a large, noisy gap between training and validation — RNN + GRU baseline: a noisy validation curve and a persistent gap from training.

MobileNet+GRU loss curves converging — MobileNet + GRU loss.

RNN+GRU loss curves with large validation spikes — RNN + GRU loss: validation spikes early in training.

What I'd do next

Evaluate on a larger held-out set. 32 test clips make each mistake worth about 3 points of accuracy.
Try temporal augmentation that keeps direction (speed jitter, crops) and Transformer-based temporal models.
Swap in MobileNetV3 and quantize for real-time inference on a phone.

Paper

Regarding a MobileNet and GRU-based Dynamic Hand-Gesture Recognition Model

Peng-Ru Hou, Tz-Jing Lee, Yeong-Hua Chang

Poster, 20th Formosa Plastics Corp. Applied Technology Seminar

Affiliations: Chang Gung University, Dept. of Electrical Engineering; Kang Chiao International School.