This blog serves as a guide to users who want to learn how to use our Unity Plugin to live stream face capture data to their custom characters within the Unity Engine. Please follow along below as we break down this process!
Please note that the Live Streaming feature within Rokoko Studio is only included under the paid Studio subscription plans. If you are only on the Studio Basic plan and want to take advantage of Live Streaming, check out our subscription plans here. It is also worth mentioning that this tutorial only goes over how to stream face capture data to your character. If you want to learn how to stream Smartsuit Pro data, check out our blog here.
It's also important to note that characters that utilize the standard ARKit 52 blendshapes are compatible with Face Capture. Make sure your character is set up with this method before proceeding. If you're looking for a way to rig your characters to take advantage of face capture, check out Polywink’s solution here.
Setting Up in Studio
Within Studio, click on the cogwheel at the top left to access the settings
Highlight ‘Data Streaming’:
Click the Stream Data’ button to enable it
You can also enable Live Streaming by clicking the ‘Start Live Stream’ panel to the right of Studio
Make sure that the ‘Forward IP’ is the local host IP (or the IP address for the computer you are streaming the data to)
Ensure that the ‘Forward Port’ is set to 14043
It is also important that the ‘Forward Port’ must be different from the ‘Smartsuit Listen Port’ otherwise it will cause issues
Once these Data Streaming settings are set, open Rokokok Remote app on your smartphone to set up Face Capture
As of writing this blog, Face Capture is only available for iPhoneX phones with front-facing TrueDepth cameras
Download the Rokoko Remote app for iphone here: https://apps.apple.com/us/app/rokoko-remote/id1465692290
Click the ‘Rokoko Remote’ button at the top right of Studio
Under “Available devices” you should see the name of your phone
To the right of your phone name in Studio, click the ‘Connect’ button
On your phone, a ‘Connect to Studio’ window will appear asking if you accept connecting to your computer
Tap the ‘Accept’ button to connect your phone to Studio
Once you tap ‘Accept’, the screen should show you that your phone is now connected to Studio
Tap the slider button to enable ‘Facial capture’ on your phone
When you tap the slider button, a prompt will appear asking if you would like “Rokoko Remote” to access the camera on your phone
Once Face Capture is set up, minimize Studio and open Unity.
Installing Rokoko Live
Click the Unity Launcher to open the Unity program.
Open an existing or new Unity Project (if this is your first time, we recommend opening a new project)
Import Smartsuit Plug-in from Asset Store
Click 'Window', then 'Asset Store' to pull up the Asset Store in-engine
Search for ‘Rokoko Live' then download and import it into your project
Setting up Smartsuit Receiver
After importing the plugin, within the Rokoko > Smartsuit folders, drag and drop the ‘SmartsuitReceiver’ prefab into your Hierarchy tab
This is required within your scene for live streaming to work
Make sure that the ‘SmartsuitReceiver’ prefab is highlighted in the Hierarchy so that you can see the attached script components in the Inspector
Paste the “Forward Port” number that you copied from Studio into both the “Port Range Start” and “Port Range End”
Setting up Character
Import your custom character into your project if you haven't already
Right-click on or within the "Assets” folder and click “Import New Asset”
Locate the FBX file(s) with your custom character, click “Import”
Drag and drop your character FBX (or prefab) into the scene
Locate the mesh that contains all of the blendshape data for the character’s face and click it within your Hierarchy to highlight it
Within the Inspector, click the ‘Add Component’s button and type ‘Face’ and click on the script named ‘Face’
With the Face script component added, you will need to add the Face ID of your face inside of Studio
The Face ID is located inside of the Scene Overview to the right of the face icon
The ID will be a combination of Upper and Lowercase letters and numbers
Type that Face ID inside of the ‘Face ID’ field inside of the Inspector in Unity
Once done, click the play button in Unity
You should now start to see the character’s face move around with your face inside of the Rokoko Remote app
We hope this guide has helped you learn how to forward your Face Capture data from Rokoko Studio directly to your character in Unity. If you run into any issues during this process, feel free to reach out to our support desk or email email@example.com.