Hello everyone~! How has your day been?
Today, we’re going to talk about a really hot technique!
It’s about using Automatic1111 to transform an existing cool video into AI Animation featuring your favorite character, creating the same video with Img2Img.
If in a previous post we used Automatic1111 to create a video that was close to pure creation, today we will introduce how to change an existing video into an AI art style. Shall we get started?
Img2Img AI Animation
Img2Img (Image-to-Image) in the context of stable diffusion models refers to a process where an existing image is taken as input and the model is used to modify or extend that image based on additional text prompts or instructions. Unlike text-to-image generation, which starts with a blank canvas, Img2Img begins with a given image, using it as a base to maintain certain aspects such as style, color scheme, or subject matter.
1. Preparation
Before you start working, you need to make sure that you have the following items prepared:
- Automatic1111
- ControlNet extension installed
- Original video data that you want to transform with AI
- Base Model
1) Automatic1111 Installation
Please read the previous post I uploaded about how to install Automatic1111, as it contains important information.
–> Go to Post (How to install Automatic1111)
2) Install ControlNet Extension
If you have read the previous article, you should now know how to install extensions. However, since the content is not very long, I will guide you through it again. The best way to install is to enter the link I share, copy the code, or copy the link itself, then go to the Extension tab in Automatic1111 and click on ‘Url Install’.
Paste it into the blank space to install. Since the installation is done via URL, it automatically installs all the necessary data for the program, which is really convenient, isn’t it?
Copy Code: https://github.com/Mikubill/sd-webui-controlnet
After clicking the install button, click the button to check for updates to make sure it is installed correctly, then press ‘Restart UI’.
3) Original Video Data to Change with AI
This might be the most important part of today’s post: preparing the original video data. One tip is to avoid videos where the body turns left and right. Although image creation is possible, joints or faces may not be properly generated momentarily, so it would be better to look for videos that are mostly front-facing.
Best Video Info for optimize
- Best Size for Video : 512 x 960
- Fps : 25~30
- Duration : 10’s
I decided to use a video of a woman dancing from YouTube Shorts. After downloading the video, we will now convert the video into JPG files.
First, create Input and Output folders in the desired location, and then use video software to save the video as a JPEG Sequence inside the Input folder for a duration of 10 seconds. Be sure to check the Frame Rate and render to save inside the Input folder.
4) Base Model
Now, prepare the Checkpoint model of your choice and, if you wish, the LoRA model, and place them in the appropriate folder inside Automatic1111.
2. Creating the Desired Model
It looks like all the preparations are done. Let’s start working on the project. Before creating the video, go to the img2img tab and drag the first image from the first frame of the images in the Input folder you previously extracted. Creating the model you want based on this image will be our first mission.
I will show you the model and prompts that I have designated.
- Base Model: ToonYou beta6
- Vae: vae-ft-mse-840000-ema-pruned.safetensors
- Positive Prompt: (masterpiece, best quality), 1girl, (oversized white leather jacket and black tank top underneath), navel, thick thighs, shorts, intricate, long hair, wavy hair, blonde, wide hips, black eyes, (simple black background)
- Negative Prompt: (worst quality, low quality, letterboxed, extra hands, extra fingers, headphones)
- Sampling method : DPM++SDE Karras
- Sampling Steps : 30
- Size : 512 X 960
- CFG Scale : 8
- Denoising Strength : 0.75
- Seed : 3718722656
Now that you have set the basic prompts and configurations, you need to unfold the ControlNet tab below and make some settings in that tab. Please follow along with the image below.
As in the picture above, activate the Enable button in the first tab of ControlNet and then check the Pixel Perfect on the right. Then, among the various options below, select Openpose, choose “dw_openpose_full” from Preprocessor, and select control_v11p_sd15_openpose as the Model. Finally, in the Control Mode below, please check the option that says “ControlNet is more important.”
Similarly, enable the second tab and also select Pixel Perfect, then select Lineart. In the Preprocessor, choose “Lineart_anime_desnoise” and for the Model, select “control_v11p_sd15s2_lineart_anime.” Finally, in the Control Mode, choose “Balanced” to complete all the settings.
Now, after pressing the Generate button, let’s wait for the image to be created!
I have tested using a Shorts video from the famous YouTuber @HelenPeng, which is filled with dancing. Can you see the image from the first frame of the video on the left and the generated image on the right?
It seems that the prompt for the “simple black background” was applied very appropriately, resulting in a very fitting backdrop for the content. It’s great to hear that the blonde and cartoonish visual elements came out to your satisfaction. It sounds like you’re ready to proceed with your work using this image.
Feel free to follow along with me or create your own amazing results using the prompts and base model of your choice.
3. Generate
Have you crafted the character of your dreams? If so, it’s time to unveil the final, almost magical step. You’ve diligently worked frame by frame with ControlNet to model your vision, and you’ve probably landed on an image that captures everything perfectly. At this moment, remember to safeguard the precious Seed value. It’s easy to lose amidst the multiple tweaks and changes, so copy and save it for the next part I’m about to introduce.
The final phase is surprisingly straightforward. Just open the Batch tab, as shown in the image below, and enter the paths to the Input and Output folders you’ve prepared earlier. Once that’s done, the AI will take over the heavy lifting.
When all is said and done, you’ll have a stunning video starring your very own character.
How exciting is that? Are you ready to get started?
(Absolutely, double-checking your settings before you begin any process is crucial to ensure you get the desired outcome, especially in complex tasks such as AI image generation where every parameter can significantly alter the results. Saving or documenting your seed values and settings can also be very helpful if you want to recreate a particular result or continue experimenting from a certain point.)
Isn’t it incredible? Imagine the image files in the Input folder being reborn one by one, all based on the settings you specified.
Now, it’s a matter of patience to see how it all comes together. As previously explained, the result will be stored as a PNG Sequence. Keep in mind that it won’t automatically save as a GIF, so the only file you can review at the end is the sequence of PNG images.
4. PNG Sequence to VIdeo
Ready to see your final product in action? If you have video editing software, you can load the sequence into programs like Adobe After Effects or Adobe Premiere Pro.
For those without access to video software, I suggest trying out an online converter you can find at the link below.
Online Converting Site
There are numerous other options, but for starters, I will be working with After Effects.
Once all the images are generated in the output folder, select them all and rename them so that they are automatically numbered from 0 until the end. Then, open Adobe After Effects and import one of the images. Make sure to check the ‘PNG Sequence‘ box. Similarly, in the import window, ensure that ‘Create Composition‘ is also checked before hitting OK. This will arrange the images into a sequence with each image being one frame of the video.
Shall we take a look at the video now? I’ve trimmed it down to about 3 seconds instead of 10 for brevity, so please take that into account.
Absolutely mind-blowing! The meticulous frame-by-frame transformation unleashes a dance of pixels that’s nothing short of wizardry. When the prompts are left open-ended, the animation takes on a life of its own, morphing in ways that dazzle the imagination. And yes, there’s flickering, but it’s part of the raw charm of this frontier technology. It’s like watching magic unfold in real time.
Stay tuned, because I’m going to take it up a notch by diving into crystal-clear prompts and the alchemy of image post-processing. The results? Expect to be spellbound.
So, what are you waiting for? Unleash your creative spirit and dive into the world of AI-powered animation. It’s a wild ride, and you’re in the driver’s seat!