CASE/06 / Healthcare
HungaryReading a child's movements from a phone video
Pose estimation and a trained classifier turned parents' smartphone videos into a measured movement record a doctor could review for signs of cow milk allergy.
01 / THE ASK
The decision behind the brief.
Cow milk allergy in young children often shows up as specific body movements: arching of the back, clenched fists and curled limbs. These are hard to describe in a consultation and easy to miss in a short visit. The client wanted parents to be able to record what they see at home on a phone, and wanted doctors to receive that as something they could measure rather than a description.
02 / WHAT WE FOUND
What the work revealed.
The movements that matter are short and easy to read differently. A parent's video captures them, but two people watching the same clip can disagree about what they saw. The useful step was not another recording tool. It was a consistent way to measure each movement in the clip and say whether the pattern matched the one associated with the allergy.
03 / WHAT WE BUILT
Measure the movement. Classify the pattern. Leave the decision with the doctor.
Start from a phone video
Parents record a short clip of the child on a smartphone. No special equipment is needed.
Track the body, frame by frame
Pose estimation finds the child's key body points in every frame: head, shoulders, elbows, wrists, hips, knees and ankles. [CONFIRM: TensorFlow pose model and Keypoint R-CNN, see question 3]
Turn keypoints into movement signals
From the keypoints we measure the movements linked to the allergy across the clip: how far the back arches, when the hands close and how the limbs curl.
Keypoints from each frame become movement signals over the length of the clip. The classifier looks at those signals and marks the windows where the pattern appears.
Classify the pattern
A classification model, trained on labelled examples, marks each clip as indicative or not indicative of the movement pattern associated with cow milk allergy.
Give the doctor something to review
The doctor receives the clip, the keypoint overlay and the movement timeline together. The model's output supports their assessment. It does not replace it.
04 / THE RESULT
An objective record of what the parent saw.
Parents and doctors now discuss the same measured record instead of a remembered description. Assessment of the movements relies less on subjective judgement, and the doctor sees the pattern the model found alongside the video it came from.
[CONFIRM: any validated result we can state, such as agreement with clinician labels on a test set. If there is none, keep this section as written.]
05 / ENGINEERING DETAILUnder the hood
- Smartphone video
- Keypoints per frame
- Movement signals
- Classification
- Report for the doctor
Pose estimation. Body keypoints are detected in every frame of a smartphone video. [CONFIRM: model and framework, see question 3]
Movement signals. Keypoint positions are converted into measures of back arching, hand closure and limb flexion over time.
Classification. A model trained on labelled clips classifies the movement pattern as indicative or not indicative.
Output. The clip, keypoint overlay, movement timeline and model output are packaged for the treating doctor.
- Pose estimation
- Keypoint detection
- Movement signal extraction
- Trained classifier
- Clinician-facing report