Live Link for UE5

The ZED Live Link allows developers to send camera tracking information as well as skeleton data into Unreal Engine 5 using the Live Link Plugin integrated into UE5. It also contains an Unreal Engine sample project already set up to receive this data. This way, you can control a Skeletal Mesh in real-time using the Skeleton tracking module of the ZED SDK.

You don’t need to use the ZED Plugin for Unreal, only the ZED Live Link sample and the Unreal project we provide you. You can have an overview of the solution on the diagram below:

The machine running ZED Live Link will use the ZED SDK, so it should be installed, download it here. The Unreal Project can run on a separate machine (Windows or Linux).

Live Link is a plugin designed to stream animation data coming from external sources to UE. You can get more information about this plugin in the UE5 Live Link doc. Our custom ZED Live Link sample is using this plugin to stream skeleton data into UE5.

UE project #

We provide you with an Unreal project that already contains the different assets necessary to animate avatars properly using the ZED Live Link skeleton data.

You will be able to use directly this project by hitting play while the ZED Live Link sample is running and seeing a default avatar animate. We are showing how to add your models here: Animate New Avatar.

In this tutorial, you’ll learn how to use our ZED Live Link sample, connect it to our Unreal Project and animate a default avatar with the Live Link skeleton data. You must follow these steps:

1. Open the Unreal Project #

Open our ZEDUnrealLiveLink.uproject project.

  • In your project root directory right click on the ZEDUnrealLiveLink.uproject file and select Generate Visual Studio project files in the context menu.
  • Open Visual Studio and compile. Launch the project from the uproject file or Visual Studio.

If these options are not available, make sure your Visual Studio install is correctly set up.

You’ll see the following empty scene:

You can follow these steps if the Live Link Plugin is not enabled yet in your project.

  1. Inside your project, In the Menu Bar under Edit, select Plugins.

  1. Under the Animation section, click Enabled for Live Link, and Yes on the confirmation window, then restart the Editor.

At this point, you must run the ZED Live Link executable. It will call the ZED SDK in order to compute skeleton data and stream it to the Unreal project. So you must make sure to have the ZED SDK installed on the machine that will run the ZED Live Link sample.

A pre-built ZEDLiveLink sender for the current SDK version is located in zed-livelink repo/zed-livelink-fusion (or mono)/Releases/your system/.

To learn how to build the ZED Live link sample yourself, take a look at our Build ZED Live Link documentation page.

The release 4.0 of the ZED SDK introduced the Fusion API which allows you to fuse skeleton data from multiple cameras. This feature is also supported in the project. This project is now split into two different programs:

  • zed-livelink-mono: the “classic” live link sample that sends camera tracking and skeleton tracking data from one single camera.
  • zed-livelink-fusion: the new live link sample that sends fused skeleton data from one or multiple cameras.
  • Open a terminal in the folder containing the ZED Live Link executable, and run
$ ./ZEDLiveLink

You can see the connection status and the parameters given in the config file in the terminal.

Adjust ZED SDK Parameters #

The ZED SDK parameters are now available in a JSON file next to the executable of the Live link sample. It allows you to change the parameters without rebuilding the sample.

The configuration file looks like this :

{
   "InitParameters":
   {
       "input": "USB_ID",
       "input_path": 0,
       "resolution": "HD1080",
       "fps": 30,
       "depth_mode": "ULTRA"
   },
   "PositionalTrackingParameters":
   {
       "enable_pose_smoothing" : true,
       "enable_area_memory": false
   },
   "BodyTrackingParameters":
   {
       "enable_module": true,
       "detection_model": "HUMAN_BODY_ACCURATE",
       "body_format": "BODY_38",
       "body_selection": "FULL",
       "confidence": 40,
       "max_range": -1,
       "minimum_keypoints_threshold" : -1,
       "skeleton_smoothing": 0.2
   }
}

You can get more information about the role of the different parameters by taking a look at the API documentation.

By default, the Live Link sample will try to open a JSON file called ZEDLiveLinkConfig.json next to the executable.

But you can also give the path to your config file as an argument when running the Live Link sample :

$ ./ZEDLiveLink path/to/config/file.json
  • Open a terminal in the folder containing the ZED Live Link executable, and run
$ ./ZEDLiveLinkFusion

Adjust ZED SDK Parameters #

The ZED Live link sample requires a configuration file (JSON format) to run. This file contains all the parameters from the ZED SDK that can be modified in this sample. By default, the sample will try to open a JSON called “ZEDFusionLiveLinkConfig.json” located next to this executable.

The Fusion API requires a calibration file to be able to fuse the data from multiple cameras. This file can be generated using the ZED360 tool available with the ZED SDK. Once your calibration file is ready, in the ZEDFusionLivelink.json file, set the value of json_config_filename to the path to this calibration file.

The configuration file looks like this :

{
   "InitParameters":
   {
       "resolution": "HD1080",
       "fps": 30,
       "depth_mode": "ULTRA"
   },
   "BodyTrackingParameters":
   {
       "detection_model": "HUMAN_BODY_ACCURATE",
       "body_format": "BODY_38",
       "body_selection": "FULL",
       "confidence": 40,
       "max_range": -1,
       "minimum_keypoints_threshold" : -1
   },
   "InitFusionParameters":
   {
       "json_config_filename": "",
       "output_performance_metrics": true,
       "verbose": false
   },
   "BodyTrackingFusionParameters":
   {
       "skeleton_minimm_allowed_keypoints": -1,
       "skeleton_minimum_allowed_camera": -1,
       "skeleton_smoothing": 0
   }
}

You can get more information about the role of the different parameters by looking at the API documentation.

By default, the Live Link sample will try to open a JSON file called ZEDFusionLiveLinkConfig.json next to the executable.

But you can also give the path to your config file as an argument when running the Live Link sample :

$ ./ZEDLiveLinkFusion path/to/config/file.json

In the “LivelinkMap” level, it no longer automatically connects to a live link at Start. Now, you need to do it manually by either selecting a Source in the Live link window.

  1. Inside the UE5 Editor, from the Menu Bar under Window, select Live Link.

  1. In the Live Link window, click the Source button, then under Message Bus Source, select your ZED source.

You can also press ‘S’ in the scene which will open a basic UI with all the sources available. With this method, connecting to a source will automatically disconnect you from all other sources, meaning you can connect to only one source at a time.

5. Select the remap asset #

The ZED SDK now has multiple skeleton formats (Body 34, 38 & 70) available for animating a 3D model. You can find more information about the body formats available in the ZED SDK on the Body tracking page. You need to make sure the correct remap asset is selected in the Anim Blueprint of the Actor you are using in your level. Indeed, each body format has its own remap asset (as the name and number of joints are different). The remap asset must be set in the Anim blueprint, in the ZED Livelink pose component.

For example, if you are using the Body format Body_38, open the ABP_ZED_Manny anim blueprint, select the ZED LivelinkPose component, and, in the Detail panel, set the Remap Asset field to RemapAssetBody38

6. Hit Play in Unreal #

With the project you’re provided with, once the ZED Live Link sample is running, you can animate a default avatar by directly hitting Play in Unreal. You will see avatars animated with skeleton data coming from the connected Live Link source:

Troubleshooting #

  • Your firewall might block the data stream. If you do not see the ZED Source in the Live Link window, try to disable it.

  • If the ZED Source is not yet detected in UnrealEngine, enable Enable by default in Edit -> Project Settings -> UDP Messaging -> Enable by default

  • On Linux, if the ZED Live Link sample executable crashes at the start, try to run the ldd command onto the sl_zed_c.so library :
$ ldd sl_zed_c.so

It will show all the dependencies required by the .so and allow you to install anything that might be missing (for example lib-usb).

Note: The c wrapper used for the Live link sample is also available here : https://github.com/stereolabs/zed-c-api.

If you encounter issues running the live link sample, do not hesitate to build the wrapper yourself and place it in the lib/win64 or /linux folder.