Action Recognition in Video using Deep Learning
A two-stream convolutional network that recognises human actions in video by combining appearance (RGB frames) with motion (optical flow).
Overview
Classifying actions in still images is largely solved by deep convolutional networks, but video is harder: an action unfolds over time, so a model needs context from the whole clip rather than from any single frame.
Approach
The architecture is inspired by the two pathways of the human visual cortex — one for recognising objects, one for recognising motion:
- Spatial stream — a ResNet-34 classifies individual RGB frames sampled from the video, capturing objects and scene.
- Temporal stream — a second ResNet-34 takes stacked dense optical flow between consecutive frames, capturing motion.
- The two streams’ class scores are fused to produce the final prediction.
Write-up
The method, experiments and results are described in the paper and poster linked above.