Body Tracking Overview
The body tracking module focuses on a person’s bones detection and tracking. A detected bone is represented by its two endpoints also called keypoints. The ZED camera can provide 2D and 3D information on each detected keypoints. Furthermore, it produces local rotation between neighbor bones.
How It Works #
The overall process is very similar to the ZED SDK Object detection module. They share some information in outputs like the 3D position and 3D velocity of each person. The body tracking module also uses a neural network for keypoints detection and then calls depth and positional tracking of the ZED SDK module to get the final 3D position of each keypoint. The ZED SDK supports multiple body formats :
The BODY_18 body format contains 18 keypoints following the COCO18 skeleton representation
Each keypoint is indexed by an integer from 0 to 17 :
| keypoint index | keypoint name | keypoint index | keypoint name |
|---|---|---|---|
| NOSE | RIGHT_KNEE | ||
| NECK | RIGHT_ANKLE | ||
| RIGHT_SHOULDER | LEFT_HIP | ||
| RIGHT_ELBOW | LEFT_KNEE | ||
| RIGHT_WRIST | LEFT_ANKLE | ||
| LEFT_SHOULDER | RIGHT_EYE | ||
| LEFT_ELBOW | LEFT_EYE | ||
| LEFT_WRIST | RIGHT_EAR | ||
| RIGHT_HIP | LEFT_EAR |
The BODY_34 body format contains 34 keypoints following this configuration :
- Each keypoint is indexed by an integer from 0 to 33 :
| keypoint index | keypoint name | keypoint index | keypoint name | keypoint index | keypoint name | keypoint index | keypoint name |
|---|---|---|---|---|---|---|---|
| PELVIS | LEFT_HANDTIP | LEFT_HIP | NOSE | ||||
| NAVAL_SPINE | LEFT_THUMB | LEFT_KNEE | LEFT_EYE | ||||
| CHEST_SPINE | RIGHT_CLAVICLE | LEFT_ANKLE | LEFT_EAR | ||||
| NECK | RIGHT_SHOULDER | LEFT_FOOT | RIGHT_EYE | ||||
| LEFT_CLAVICLE | RIGHT_ELBOW | RIGHT_HIP | RIGHT_EAR | ||||
| LEFT_SHOULDER | RIGHT_WRIST | RIGHT_KNEE | LEFT_HEEL | ||||
| LEFT_ELBOW | RIGHT_HAND | RIGHT_ANKLE | RIGHT_HEEL | ||||
| LEFT_WRIST | RIGHT_HANDTIP | RIGHT_FOOT | |||||
| LEFT_HAND | RIGHT_THUMB | HEAD |
The BODY_38 body format contains 38 keypoints following this configuration :
- Each keypoint is indexed by an integer from 0 to 37 :
| keypoint index | keypoint name | keypoint index | keypoint name | keypoint index | keypoint name | keypoint index | keypoint name |
|---|---|---|---|---|---|---|---|
| PELVIS | LEFT_CLAVICLE | LEFT_KNEE | LEFT_HAND_THUMB_4 | ||||
| SPINE_1 | RIGHT_CLAVICLE | RIGHT_KNEE | RIGHT_HAND_THUMB_4 | ||||
| SPINE_2 | LEFT_SHOULDER | LEFT_ANKLE | LEFT_HAND_INDEX_1 | ||||
| SPINE_3 | RIGHT_SHOULDER | RIGHT_ANKLE | RIGHT_HAND_INDEX_1 | ||||
| NECK | LEFT_ELBOW | LEFT_BIG_TOE | LEFT_HAND_MIDDLE_4 | ||||
| NOSE | RIGHT_ELBOW | RIGHT_BIG_TOE | RIGHT_HAND_MIDDLE_4 | ||||
| LEFT_EYE | LEFT_WRIST | LEFT_SMALL_TOE | LEFT_HAND_PINKY_1 | ||||
| RIGHT_EYE | RIGHT_WRIST | RIGHT_SMALL_TOE | RIGHT_HAND_PINKY_1 | ||||
| LEFT_EAR | LEFT_HIP | LEFT_HEEL | |||||
| RIGHT_EAR | RIGHT_HIP | RIGHT_HEEL |
The ZED SDK can output 3 levels of information: raw 2D/3D body detection, 3D body tracking and 3D body fitting.
2D/3D Body detection #
The ZED SDK first uses the ZED camera image to infer all 2D bones and keypoints using neural networks. Then the SDK depth module and positional tracking module are used together to extract the correct 3D position of each bone and keypoint.
3D body tracking #
If tracking is enabled, the ZED SDK will assign an identity to each detected body over time. At the same time, by filtering the raw body detection, it will output a more stable 3D body estimation.
3D body fitting #
Moreover, a user can enable fitting to unlock even more information about each identity. The fitting process takes the history of each tracked person to deduce all missing keypoints thanks to the human kinematic’s constraint used by the body tracking module. It is also able to extract local rotation between a pair of neighbor bones by solving the inverse kinematic problem. These data will be compatible with some known software for avataring for example. Here is an example where BODY_FORMAT::BODY_34 were used to animate an avatar in Unreal.
Detection Outputs #
Each detected person is stored as a structure in the ZED SDK called sl.BodyData.
| Body Data | Description | Output |
|---|---|---|
| ID | Fixed ID for identifying an object over time. | Integer |
| Tracking state | Defines if an object is currently tracked or lost. | Ok, Off, Searching, Terminate |
| Action state | Defines if an object is currently idle or moving. | Idle, Moving |
| Position | Provides the 3D position of the object according to the camera as a 3D vector (x,y,z). | [x, y, z] |
| Velocity | Provides the velocity of the object in space as a 3D vector (x,y,z). | [vx, vy, vz] |
| Dimensions | Provides the width, height and length of the object. | [width, height, length] |
| Detection confidence | A lower confidence means the object might not be localized perfectly or that its label is uncertain. | 0 - 100 |
| 2D bounding box | Defines the box surrounding the object in the image represented as four 2D points. | Four pixel coordinates |
| 3D bounding box | Defines the box surrounding the object in space represented as eight 3D points. | Eight 3D coordinates |
| Mask | Provides the pixels which really belong to the object and those of the background. | Binary mask |
| ID | Fixed ID for identifying an object over time. | Integer |
| Tracking state | Defines if an object is currently tracked or lost. | Ok, Off, Searching, Terminate |
| Action state | Defines if an object is currently idle or moving. | Idle, Moving |
| Position | Provides the 3D position of the object according to the camera as a 3D vector (x,y,z). | [x, y, z] |
| Velocity | Provides the velocity of the object in space as a 3D vector (x,y,z). | [vx, vy, vz] |
| Dimensions | Provides the width, height and length of the object. | [width, height, length] |
| Detection confidence | A lower confidence means the object might not be localized perfectly or that its label is uncertain. | 0 - 100 |
| 2D bounding box | Defines the box surrounding the object in the image represented as four 2D points. | Four pixel coordinates |
| 3D bounding box | Defines the box surrounding the object in space represented as eight 3D points. | Eight 3D coordinates |
| Mask | Provides the pixels which really belong to the object and those of the background. | Binary mask |
| 2D keypoint | A set of useful points representing the human body, expressed in 2D. | a vector of [x,y] |
| Keypoint | A set of useful points representing the human body, expressed in 3D. | a vector of [x, y, z] |
| 2D head bounding box | bounds the head with four 2D points. | Four pixel coordinates |
| 3D head bounding box | bounds the head with eight 3D points. | Eight 3D coordinates |
| Head position | 3D head centroid | [x, y, z] |
| Keypoint confidence | Per keypoint detection confidence | a vector of float |
| Local position per joint | local position of each keypoint | a vector of [x,y,z] |
| Local orientation per joint | local rotation of each keypoint | a vector of [x,y,z,w] |
| Global root orientation | global root orientation of the Body | [x,y,z,w] |
For more information on Body Tracking, see the Using the API page.