Blender Retargeting Workflow: The Ultimate Retargeting Guide

Below, please find a transcript of the video tutorial. Enjoy!

Introduction

Hey everyone, I'm Sam Lazarus, creative director for Rokoko. And welcome to this ultimate retargeting tutorial for Blender. So we're gonna go soup to nuts in this tutorial. We're gonna take a character that we rig on Mixamo, although this workflow will work with characters that you make on your own or are not rigged with Mixamo skeletons. We're gonna take some motion capture that we made in the new Rokoko studio. We're gonna send that over to Blender, put it on our character for some basic retargeting. And then at the end of the video we're gonna go through a really interesting IK workflow using the Mixamo control rig, uh, add-on that makes it really easy to go in and edit your motion capture using IK tools. So I will mention also, you know, we say this is the ultimate retargeting tutorial kind of tongue in cheek because especially with Blender, there's so many workflows out there for retargeting that may be more appropriate for you in what you're doing. This is just a workflow that we recommend that's really easy way to do retargeting and also allows you to do some e editing of your mocap data as well, which can be really powerful to fix clipping or maybe, you know, clapping or when things need to be a little bit more precise. So without further ado, let's jump into it.

‍

Plugins / Add-ons You Will Need

So before we get into everything, we should just go over the plugins that you're going to need for this workflow. So for just the basic retargeting workflow, you're gonna need the Rokoko Retargeting plugin, uh, which if you just Google Rokoko Blender GitHub, uh, you'll, you'll pick it up. It'll also be in the description below along with the rest of these plugins. Um, you click on the link and then you would just go to code download.zip, which is what you would install in Blender and we'll go over that when we get to Blender.

‍
Um, I would also recommend picking up the cat blender plugin. Again, link will be in the description. This plugin was originally developed for VR Chat and it makes it really easy to correct APOs to TPOs errors, which can happen sometimes when you're retargeting, uh, in Blender and in other tools. So I would recommend grabbing this plugin. It's a great plugin just for Reposing characters in general. At the end of the video, we're gonna go over how to use a Mixamo control rig with a Mixamo character to really easily edit your motion capture. So you're gonna need the Mixamo add-on for that. And then finally, I would recommend picking up the animation layers add-on on the Blender marketplace. This is just an add-on that overhauls, the animation layer system in Blender gives you a nice little UI with which to create layers over your original mocap animation where you can do edits. So it's non-destructive mocap editing. And we're gonna be using this at the end of the video as well in that control rig tutorial.

Exporting Rokoko Mocap

Okay, but let's really get into it now. So here we are in the new Rokoko Studio and I've already gone ahead and recorded some body and hand mocap. We're not gonna go over how to do this. We have lots of other tutorials on how to do that. This video is gonna be focused on just the retargeting aspect of, of uh, this whole process. So as you see, you know, do some karate moves, jumping around, do my little kick and then I pat my head and you can see I don't quite make contact. Uh, also with the clap, I don't quite make contact exactly between my two hands and that's a common thing in inertial mocap and inver coco mocap. Sometimes very precise touching of the body or clapping your your head.

‍
You might not touch it exactly in the motion capture animation. So that's why knowing how to do a little bit of mocap cleanup in blunder is so useful, which is again, what we're gonna cover at the end of the video. So here I've got my motion capture, it's all recorded. I've already renamed my clip and I've cleaned it up as you can see, cleaned up my foot contacts, everything looks really good. So I'm just going to right click on the take add to export list. Then I'll go over to export. We're gonna export an FBX because we're going into Blender. We don't need the body mesh and we don't need the face because there is no facial animation for skeleton preset. I'm going to select Mixamo, I found it works really well with this Blender workflow, so that's what I would recommend you do. And then we don't need to change anything under preset overrides, but under more settings you do wanna make sure that you're going to be exporting a binary FBX Blender Doesn't like ASCII I've found in the past, so it's better to go with binary. And then fbx version, we're gonna leave alone frame rate 30 and we are going to export out our clips.

Rigging a Mixamo Character

So before we hop into Blender, we're doing this tutorial soup to nuts, right? So we are going to rig up a character on Mixamo, although again, this workflow doesn't need to have a Mixamo character for it to work. You just need a character in a t pose, uh, which we'll go over again in a second. So I've got a character here and we'll just upload it to Mixamo. There's tons of tutorials online on how to rig a character in Mixamo so I'm not gonna go really go over it in too much detail. But I do want to point out that the character that I'm uploading is in an A pose. So this is an o BJ mesh that's in an A pose. And so that's going to affect, uh, the retargeting process when we get into Blender and we'll we'll remember that for when we get to how to fix this problem. So I'm just going to do the Mixamo Auto rigging, you know the deal. Here we go. It's all rigged up. Looks good, fingers are doing their thing and everything hit next and next and there we go. And now I'm just going to go to animations in Mixamo and I'll type in T-pose. Here we go. And we can download this character, put it download, boom. Now we can jump into Blender.

‍

Importing our Character to Blender

So I'm going to be working in Blender 3.3 0.1, 3.4 just came out but some of these plugins haven't been updated yet. By the time you see this video, we'll put it in the description below if, if our plugin has been updated to 3.4, but this should work with older versions of Blender and our plugin will be updated very shortly. It's just because 3.4 just came out. So I'm sticking with 3.3. We're going to delete everything in our scene and then I'm going to go up to file import and we're going to import that character that we just downloaded.

‍
So T-pose two, whenever I import a character FBX into Blender, always make sure to click this automatic bone orientation, uh, underneath armature. So I'll make sure that's clicked. Hit import. And there we go. We got our character. So the very first thing we're going to do because this is a Mixamo character but also for other characters is that you can see that we've got some key frames that are on this character skeleton. We don't want any key frames, you know, getting in the way of our mocap. So we're gonna just make sure we delete these key frames. And I'm gonna do that by going to pose mode up here, selecting all the bones. You can see if we go to my skeleton, all my bones are selected and then I'm just going to delete key frames. There we go. No more key frames, everything nice and clean.

Fixing the A Pose / T Pose Issue

So the next thing that we need to do before we even get motion capture involved is again, make sure that this character is in a t pose. When we do retargeting, you need to make sure that the motion capture and the characters that their initial poses on frame zero match and Rokoko Studio will always spit out a t pose. Um, and so we need our character to be in a T pose and this looks like it's in a T pose, but if we select one of the meshes that make up our character and we go to edit mode, you can see that it's going to change, its posing. And that's because this pose, this a pose is the rest pose for this character. And whenever we retarget motion capture, it's going to retarget based on the rest pose and we need that rest pose to be in a T pose.

‍
So how do we fix this, right? And again, you can kind of find out what your rest pose is of your character by just clicking a mesh and going to edit and it should change. Well this is where the cat's plugin comes in. And so again, I've got all my plugins up here to install plugins. You just go to edit preferences, add-ons, install, and then you install those.zip files like the Rokoko plugin you would download as a zip and you just install it that way. And here's where they'll all show up. So I've got my cats plug in here, which is free. And if I click start pose mode, I'm just going to get rid of this camera came along with our character. If I select our character start pose mode, you can see that it hops into this, uh, this rest pose. And so we can just go in and grab the upper arms and it looks a little funky with the, you know, with the uh, the mesh there.

‍
But that's something that you could fix when you're preparing your model. It kind of happens with Mixamo and it's not really the end of the world because for most of of the time our arms are gonna be down. Um, so I wanna make sure my arms are in a T pose and then I'm also going to go and on the thumb bones, I'm just gonna try and make these live flat on the same plane as the fingers kind of like this. And that's because our skeleton that comes out of Rococo studio, this is how the thumb will be posed. Um, and so we just, again, we want to make sure everything is as close to the Rokoko initial pose as possible. And so that's just the thumb kind of lying flat along the, the hand there. So that looks good. The other thing that we can do, which we don't need to do with this character, but you should try to position the feet directly underneath the hips.

‍
Um, again, this looks good as is, but if your feet are sometimes with characters, they'll be more like this. And you just wanna make sure those feet are in a straight line, the legs are in a straight line down and the feet are under the hip. So after we've done all of our posing thumbs are looking good, arms are looking good, everything we can hit apply as rest pose. And you can also use blender's built-in tools to, to apply a rest pose. I've just found it that it's kind of hit or miss sometimes and it can cause pro unexpected problems. So I really like this cat's plugin for fixing this problem. And again, you'll see if I, if I select our mesh and we go into edit mode, now it's not moving, right? So this is ready, this character is in a T pose, it's ready to receive motion capture, and at this point I usually would just go and save Sam character. It's a little, it's got a red beanie, so it's, you know, Sam character rigged. Just save it on the desktop. And now I've got a project file with my character. It's in a t pose. It's really easy to bring in motion capture now and do my retargeting because this is already set up.

Basic Retargeting of Mocap

So let's go through that process. Now, just how do we do basic retargeting, right? We've, we've got our motion capture and we've exported it out. How do we get it onto this character? So we're gonna go up to file import fbx, and we're gonna go find our motion capture that we exported. If you ever wanna find where your motion capture, uh, was exported to, you can of course select where you want to export it to. But you can also always just go open folder and it will open it up right here.

‍
So I can copy this path. We'll hop back into blender, paste this, and then we have our retargeting demo, Mixamo skeleton. Remember always go into armature, make sure that automatic bone orientation is checked, we will hit fbx and there we go. Look, it all came in. If we play it through, here's our motion capture. And I'm just gonna expand out this scene a little bit. I think that it's 640, there we go. And so now how do we get this motion capture onto our character, right? So for this, we're going to be using the Rococo plugin, but before I do that, if I go and I move my character over a little bit, one thing that I like to do is just resize my character so that their hips are kind of at the same level as my mo cap's hips. And that's just gonna make all the foot contact with the ground just be a little bit better, stick a little bit better.

‍
It really matters that the hips of your character and your mocap are at the same level. So I'm going to just shrink down our character a little bit and that that matches the mocap for the hips a little bit better. The arms are okay if they're, if they're not aligned properly. So we're gonna go to our Rokoko plugin, which again is right here. And we installed that of course by going to edit preferences, add-ons and install right here. And we just installed that.zip file from GitHub for the plugin. And when you open up the plugin, you'll have to sign in using your Rokoko id, which is free. It's the same ID that you use in Rokoko Studio. But down here we have this little retargeting section. And so if I go under source and I select my mocap as the source, it's the source of the animation and the target is going to be our Mixamo character, that's where the animation is going.

‍
We can hit Build Bone list. And this is really great that this happened so automatically. So the bone list essentially is helping Blender understand, hey, this is the right arm of my motion capture, this is the right arm of my character. When you're retargeting that mocap, make sure that it goes to the right bone, right? So Blender doesn't inherently know which bones match each other between your mocap and your character. So you need to go in and tell Blender. And so right here we have, you know, the hips of our mocap and it matches the hips of our character because this is a Mixamo character, this filled out pretty automatically. Sometimes you might have duplicate bones or sometimes it, it messes up the left shoulder, it might duplicate it for the left arm. You can always go in manually and change these designations. Um, so if you get any errors, you might just need to go in and, and make sure that this is all correct and then you can just hit save. And the next time you use a character with that skeleton, it will all already automatically be in there. So assuming this is all right and ready to go, we can just hit retarget animation, it'll think for a second.

‍
And there we go. Look, we hit play, we've got some motion capture on our character and it all looks great. The foot contact looks awesome, the feet aren't sliding around. And again, that's because we made sure that our character's hips kind of matched to the same height as our motion capture ships. So, you know, as we saw before in our mocap, we've got some things that we might need to fix, right?

‍

FK Mocap Clean Up

There's some clipping issues right here. The hand is just going through the head. So that's a problem obviously, and we can go through and we can do a little bit of cleanup on that without getting into a control rig or anything else. But the problem with cleaning up motion capture this way is just that we have to do it fk. There's no control rig ik control rig on this character. So if we need to move the hand, that means we're gonna have to move the upper arm and then the forearm and then the hand to, to get it around position, which is possible.

‍
And we'll go through that right now. But it's not quite as good as a control rig workflow, which is what we're gonna do at the, at the end of the video. So if we wanted though, to go in and, and just change this, this intersection, what we could do is open up our non-linear animation panel. And if we actually, what we can do is we can go ahead and delete our motion capture. We don't need that anymore. And on our non-linear animation panel, if we click this little down arrow, which makes sure that I'm not blocking here, we click this little down area. You can see that it creates a new layer on top of our original motion capture layer. So that means that we can add mocap edit that are non-destructive to the original motion capture layer. So if we go into pose mode

‍
And we go and we grab that forearm, let's just try and just adjust that forearm we could, and again, remember we, we created a new layer here by hitting that down button. Then we actually need to go and make sure that blending is set to combine and not replace because we want to add and edit onto the animation we already have. We'll grab that forearm bone again. I'm gonna hit s to create a key frame or whatever, create a key frame may be in your blender setup. Go to where I pat the head, so right here. And at this point we'll just rotate this hand so that it's kind of hitting the top of the head a little bit better. And we could go in and tweak that more, but let's just say that we're satisfied with that and we'll create another key frame.

‍
There we go. We have now edited that little section and so this is great for doing small tweaks, but it can get really tiresome if, say you really needed to do some mocap editing, you needed to do a bunch of different edits. You don't want to go in and have all these different kind of messy layers, especially in the blender native, uh, non-linear animation panel. Um, so let's go through a workflow right now where we will create a Mixamo control rig again for this Mixamo character. This workflow will only work for Mixamo characters. Um, and it's gonna be even easier to edit our motion capture.

IK Mixamo Control Rig Workflow

Okay, so here I'm back at that saved project file, you know, my Mixamo character, I've gone in, I've made sure that it's in a t pose, you know, using the cat's plugin. This, this file is ready to receive motion capture. And before we do this, I should also mention that there's lots of other workflows that use IK control rigs. The Auto Rig Pro uh, plugin has a great workflow for doing something similar as well as the native rfi, um, you know, systems and blender. This is just one way to do this. And so I encourage you to go try and find other tutorials that might cover some of those other workflows in case this doesn't really, you know, you don't wanna work with a Mixamo character for example. Um, but in this case we are so, so it's pretty easy. So the first thing that we're going to do for this workflow, I've got my character, it's in a t pose. I deleted the animation from it, you know, everything we did, we did before. And I'm just going to create a copy of this character. So now we've got my character and I've also got in the scene if I hide all of these, we've got this other armature right? And both of them are blank. So I'm going to turn that back on. And if we select our original character and let's go in and actually rename this Mixamo character, we'll rename this Mixamo character copy.

And if we click on our Mixamo character, which is what we're retargeting to, and we go to the Mixamo plugin, which we, you have to install the mii mo control rig add-on, uh, which is in the description below. And we select the character. You can see characters Mixamo character, we hit create control rig, we're gonna turn off, apply animation and hit okay. And as soon as we do that, we have something pretty awesome, which is an automatic con ik control rig. If we go into pose mode and I select the risks, for example, look at this, now we have an IK control rig where we can just affect, you know, parts of the character and the rest of the character is kind of gonna react appropriately and it's gonna make it a lot easier. You can imagine fixing problems with mocap if you can just grab the wrist and the rest of the arm moves along. So at this point, let's go and let's import our motion capture and we're gonna use the same motion capture that we did before this retargeting demo armature, make sure that automatic bone orientation is checked, import fbx, and at this point, okay, so here we've got our mocap and let's again, let's expand out the timeline to six 40 and play through. And so here, and I'm gonna kind of move all these guys out so you can see them. So here's our motion capture. Here's the copy we made of our characters before we added the control rig. And here's our character. So the Mixamo control rig system, it's great, it creates this control rig automatically. And then under animation, you can see if we select our Mixamo character with this control rig, we could load in animation from Mixamo, which comes when you download animation from Maximo. It comes on a Mixamo skeleton. Well, unfortunately we can't feed in directly the Rokoko motion capture, but what we can do is retarget our Rokoko motion capture right here. We'll retarget that onto the copy we made of our Mixamo character, and then we can feed that into the Mixamo control rig. And then we'll have animation on an IK control rig. So I'm gonna undo moving these guys around cuz I want to keep 'em in the same place. And I'm gonna do just that. I'm gonna go to my Rococo plugin under Retargeting source is gonna be our motion capture target is going to be the Mixamo character Copy Build Bone list. Again, this all does it automatically because it's a Mixamo character retarget animation. It'll think for a second.

‍
Now we have our mocap animation on our copy of our Mixamo character. Let's delete our, our Rokoko mocap outta the scene. And now if we just go to our Mixamo character and the Mixamo plugin and under animation for source skeleton, select that copy apply animation to control rig and it'll thank, and then I hit bake animation. I I don't think it does anything. And now let's delete our copy and we just have animation on a control rig. And what's amazing about this is if we want to go in and make some edits, what we can do is just go into pose mode and you can see that there's key frames on all of these control rig, right? So I'm going to use the animation layer plugin that I was talking about at the beginning of the video. This is just a way better version of the non-linear native blender, uh, system, the non-linear animation system that makes animation layers. So you can do non-destructive additional animation on top of your motion capture. This is just a better kind of layout and it's easier to use. So, so I'd like this plugin a lot for that.

‍
And we turn on animation layers right here and we just add hit plus. You can see under base layer we have all the key frames on our character, but on anime layer, all those key frames are gone, which means now we can edit some of this mocap really easily. So let's go back and, and fix this again. And this time again, we'll go into pose mode. We'll make, we'll make sure that we're on this top anim layer, not on the base layer, right? We'll select this control null for the wrist here. I'm gonna add a key frame right here. Boom. When my, my head, when my hand hits my head, now I can just use

This IK control rig and look at how much easier it is to animate it. And you could imagine how going through and doing a bunch of edits using, you know, IK control rig instead of doing each bone individually, boop boop is so much easier. And if we want this animation to end and go back to how it was, you know, before we touched it, right? We would add a key frame right here when the animation is kind of ending. And then if I go and I select this initial key frame, right, copy and paste it, it goes back to how it was in the beginning. And there we go. We've just fixed that. So you, we can go through and we could add more layers and we could fix, you know, the clapping

Pretty easily just by going in fixing a couple of these things. And, and again, hopefully you can see just how powerful this tool is, um, when it comes to editing your motion capture as opposed to going in and, and doing each bone individually. You know, you've got this amazing control rig that kind of does it all for you. And so that is, you know, our, our retargeting workflow for Blender. Um, hopefully this, uh, helped you and uh, but if you have any questions, please put them in the comments below and we'll try to get to them. We're gonna be having more tutorials coming out soon, going over some face capture stuff and also, and so stay tuned for those sorts of things and, uh, you know, join our discord if you have more questions. And other than that, we'll see you in the next video and I hope, uh, you go out and make some awesome mocap out there. Okay, see you everyone. Bye-bye.

‍