The model and workflow

0:00 To demonstrate how all these tools can work together, we filmed an entire short film on a phone and used this AI model to transform me into a man who one day wakes up to find himself transformed into a monstrous vermin. And in this video, we're going to share all the techniques and workflows we built to create this short film with you. So whether you want to level up your AI filmmaking skills or turn yourself into a cute anime woman, we got you covered.

0:23 Oh, and make sure to watch to the end for the full short film. As a base for our workflows, we're going to use the free open-source 12.1 video model. This is not a model that can do everything, but rather it is like the code foundation for a lot of other AI models that are based on it that can do all the things that you would want from an AI video model like for example text to video, image to video or even frame interpolation.

0:49 But my favorite version of this model is the fun variation which allows you to use control nets to guide video generation. For example, we can extract the poses of our character from our input video and then generate a new video where the character follows the same exact movement. And the advantage of using a pose control net is that the model has a lot of freedom to change the character since we're only providing the basic bone structure.

1:12 Compare that to the line art control net where we extract outlines from the original video. Here we give the model much more detail about the structure so it has less freedom to make changes. A good middle ground is often the depth control net where we extract the estimated depth information from the original video. It's really good at maintaining the composition and camera movement, but you just give it a bit more freedom compared to a line art control net.

1:35 Let's look at an example from our short film. Since we want to transform me into a beetle, we should give one 2.1 as much freedom as possible to change my body structure. And that's why we are using the pose control net. This works okay, but we face three problems. The character looks different in every shot. The camera movement doesn't match the original input video and the surrounding scene is completely different. And since we want to put the transformed character back on top of the original footage, the scene and the camera movement has to match as closely as possible.

2:21 So, let's solve these problems one by one. First, let's fix the camera movement. I've created an automatic workflow that combines the post control net with a line art control net, but fades out the line art control net where the character is. And this way, one 2.1 can freely change the character, but it still has enough information about the surrounding scene geometry so it can match the camera movement. Now, the character is still not really consistent from shot to shot.

2:44 There are a few ways to fix this. The first one would be to train a 1 2.1 Laura, but this is very resource inensive and we would need to rent some GPUs for that. The second option would be to use the vase video model which is also based on W 2.1. This one lets you input a reference image of your character or style while still allowing you to use control nets. And while this works really well for a lot of use cases, for our beetle it was just not good enough.

3:07 I think it has a hard time understanding the anatomy of the beetle and it also changes the environment which is something that we don't want. So we went back to that fun model because this one also lets you use a reference image as the first frame of your video. This way we can automatically extract the first frame of our video. Use a simple inpainting workflow to change our character and then generate the rest of the video based on that frame.

3:30 And just look at these amazing results. Look at how well this worked. When we saw this, we knew we could shoot the entire short film this way. So now let me walk you through the process step by step. Starting with the installation. All my workflows are