Face Capture get started

Key takeaways
How to Live Stream Face Capture data to Unreal
How to Live Stream Face Capture data to Unreal
How to use Face Capture Data in Maya
How to use Face Capture Data in Maya
How to use Face Capture data in Blender (2.79)
How to use Face Capture data in Blender (2.79)
How to use Face Capture data in Cinema4D
How to use Face Capture data in Cinema4D
How to use Face Capture data in Blender (2.80)
How to use Face Capture data in Blender (2.80)
How to Live Stream Face Capture Data in Unity
How to Live Stream Face Capture Data in Unity
Everything you need to know about Rokoko Face Capture
Everything you need to know about Rokoko Face Capture

How to Live Stream Face Capture data to Unreal

Need to learn how to live stream face capture data to your character in Unreal? Look no further! This video breaks down that process for you.

This blog serves as a guide to users who want to learn how to use our Unreal Plugin to forward live stream face capture data to their custom characters within the Unreal Engine. Please follow along below as we break down this process!

Please note that the Live Streaming feature within Rokoko Studio is only included under the paid Studio subscription plans. If you are only on the Studio Basic plan and want to take advantage of Live Streaming, check out our subscription plans here. If you want to learn how to also stream Smartsuit Pro data in Unreal, check out our blog here.

Setting Up in Studio

  • Open Rokoko Studio (1.13.0), click on the cogwheel at the top left to access the settings
  • Highlight ‘Data Streaming’:
  • Click the Stream Data’ button to enable it
  • You can also enable Live Streaming by clicking the ‘Start Live Stream’ panel to the right of Studio
  • Make sure that the ‘Forward IP’ is the local host IP (or the IP address for the computer you are streaming the data to)
  • Ensure that the ‘Forward Port’ is set to 14043
  • It is also important that the ‘Forward Port’ must be different from the ‘Smartsuit Listen Port’ otherwise it will cause issues
  • Once these Data Streaming settings are set, open Rokokok Remote app on your smartphone to set up Face Capture
  • As of writing this blog, Face Capture is only available for iPhoneX phones with front-facing TrueDepth cameras
  • Download the Rokoko Remote app for iPhone here
  • Click the ‘Rokoko Remote’ button at the top right of Studio
  • Under “Available devices” you should see the name of your phone
  • To the right of your phone name in Studio, click the ‘Connect’ button
  • On your phone, a ‘Connect to Studio’ window will appear asking if you accept connecting to your computer
  • Tap the ‘Accept’ button to connect your phone to Studio
  • Once you tap ‘Accept’, the screen should show you that your phone is now connected to Studio
  • Tap the slider button to enable ‘Facial capture’ on your phone
  • When you tap the slider button, a prompt will appear asking if you would like “Rokoko Remote” to access the camera on your phone
  • Tap ‘OK’
  • Once Face Capture is set up, minimize Studio and open Unreal

Smartsuit Live + Live Link

  • Click the Epic Games Launcher and navigate to the Marketplace tab
  • Use the Search bar and type in “Smartsuit Live”
  • Click on the Smartsuit Live asset
  • Navigate down the product page and click the “Install to Engine” button
  • Once finished, click the “Launch” button at the top of the Epic Games Launcher window
  • Open an existing or new Unreal Project (if this is your first time, we recommend opening a new project)
  • Once Unreal is open, click ‘Edit’ then click ‘Plugins’
  • Locate the ‘Smartsuit Live’ plugin by using the search bar
  • Check the ‘Enabled’ check box
  • A prompt will appear asking if you are sure that you want to enable the plugin. Click ‘Yes’
  • A prompt to restart Unreal will appear
  • Restart Unreal and then the Smartsuit Live plugin will be installed
  • Close out of the ‘Plugins’ window
  • Now, use the search bar under the ‘Modes’ tab on the left and type “Smartsuit Receiver”
  • Click and drag the “Smartsuit Receiver” inside of the viewport
  • Click the “SmartsuitReceiver” object inside of the World Outliner
  • Make sure that the ‘Streaming Data’ field is the same number as the “Forward Port” inside of Rokoko Studio
  • Open Live Link by clicking Window > Live Link
  • Click ‘Add’ and hover your cursor over ‘Rokoko Virtual Production Source’
  • Click ‘Studio’ then click ‘OK’
  • Close the Live Link window

Setting Up Character

  • Next, import your character:
  • Right-Click within the Content Browser and select “Import to /Game…”
  • Find the FBX of your character and select the Open button
  • If your character isn’t using an existing skeleton, leave Skeleton set to ‘None’
  • Click “Import”
  • Once your character is imported, drag and drop your skeletal mesh into the scene view
  • Under the details panel, click ‘Blueprint/Add Script’
  • Click the folder you want to create a Blueprint (Character folder works)
  • Click the ‘Create Blueprint’ button
  • The Blueprint should open on it’s own, if not, double-click on your newly added blueprint within the Content Browser
  • Click the ‘Add Component’ button and type ‘LiveLink’
  • Click on ‘Live Link’ from the dropdown to add it as a component
  • Under the ‘My Blueprint’ panel, click the “+” on the ‘Functions’ panel
  • Type ‘LiveLinkFaceNode’
  • Within the Live Link Face Node function window, click ‘Live Link Face Node’
  • Under the Details panel, click the “+” on the ‘Inputs’ panel 3 times to add 3 new parameters
  • Name each parameter as follows:
  • “SubjectName”
  • “CurveName”
  • “BlendshapeName”
  • Change all 3 parameters from ‘Boolean’ to ‘String’ by clicking the dropdown and selecting ‘String’
  • Drag out from the Execution Pin on ‘Live Link Face Node’ and search “Get Subject Data (Live Link)
  • Drag out from “Subject Name” on ‘Live Link Face Node’ and attach to “Subject Name” on ‘Get Subject Data’
  • Copy the ‘Dot’ node that was created when linking to ‘Subject Name’ and paste it twice to create two additional ones
  • Drag out from ‘Curve Name’ on ‘Live Link Face Node’ and attach to one of the ‘Dot’ nodes
  • Drag out from ‘Blendshape Name’ and attach to the other ‘Dot’ node
  • Drag out from the ‘Subject Frame Handle’ on ‘Get Subject Data’ node and search ‘Get Curves’
  • Drag out from ‘Curves’ and search ‘Find’
  • Drag out from the green pin on ‘Find’ node and search ‘float / float’
  • Change the float inside of this node from ‘1.0’ to ‘100.0’
  • Drag out from the green pin on ‘float / float’ pin and search ‘Set Morph Target (SkeletalMeshComponent)
  • Drag out from the ‘Dot’ pin leading from ‘Curve Name’ on ‘Live Link Face Node’ and attach to the purple pin on ‘Find’ node
  • Drag out from the ‘Dot’ pin leading from ‘Blendshape Name’ on ‘Live Link Face Node’ and attach to ‘Morph Target Name’ on ‘Set Morph Target’ node
  • Drag out from Execution Pin of ‘Get Subject Data’ node to the Execution Pin on ‘Set Morph Target’ node
  • Click the ‘Compile’ button
  • Green check mark means it has been set up properly
  • If you don’t get a green check mark, go through these steps again to make sure the nodes were set up properly
  • Next, click the ‘Event Graph’ tab
  • Drag out from the Execution Pin on ‘Event Tick’ node and search ‘Live Link Face Node’
  • Click the box for ‘Subject Name’ and type ‘arkit’
  • Copy the ‘Live Link Face Node’ and paste one for every blendshape you have for your character (in our case there are 52 blendshapes so we create 52 nodes)
  • Line each of the created nodes up from left to right
  • Next, open your character’s Skeletal Mesh and locate the ‘Morph Target Preview’ tab
  • This tab should list all of your face blendshapes
  • You will need to input the names of each blendshape inside the ‘Curve Name’ and ‘Blendshape Name’ fields inside the ‘Live Link Face Node’ on the Event Graph
  • The quickest way to do this is to right-click and click ‘Copy Name’ within your ‘Morph Target Preview’ and paste the names inside of each node
  • Be careful of extra spaces at the end of the name - make sure to delete them
  • Once all of the names have been added to each of the nodes, drag from the Execution Pin on ‘Event Tick’ node and attach it to the Execution Pin in the first ‘Live Link Face Node’
  • Now drag each of the Execution Pins to the following ‘Live Link Face Node’ in front of it until all nodes are attached to each other
  • The very last ‘Live Link Face Node’ will not have an Execution Pin leading out to anywhere
  • Click the ‘Compile’ button
  • Green check mark means it has been set up properly
  • If you don’t get a green check mark, go through these steps again to make sure the nodes were set up properly
  • Close out of the Blueprint window
  • Press the Play button and watch as your character’s face moves along with your facial movement!
[optimize output image]

We hope this guide has helped you learn how to forward your Face Capture data from Rokoko Studio directly to your character in Unreal. If you run into any issues during this process, feel free to reach out to our support desk or email support@rokoko.com.

To learn more about Face Capture and claim your free month, click here.

Face Capture
Rokoko Studio
Smartsuit Pro
Unreal
Key takeaways
How to Live Stream Face Capture data to Unreal
How to Live Stream Face Capture data to Unreal
How to use Face Capture Data in Maya
How to use Face Capture Data in Maya
How to use Face Capture data in Blender (2.79)
How to use Face Capture data in Blender (2.79)
How to use Face Capture data in Cinema4D
How to use Face Capture data in Cinema4D
How to use Face Capture data in Blender (2.80)
How to use Face Capture data in Blender (2.80)
How to Live Stream Face Capture Data in Unity
How to Live Stream Face Capture Data in Unity
Everything you need to know about Rokoko Face Capture
Everything you need to know about Rokoko Face Capture
More courses