What is Style Transfer in AI?
Style transfer in AI is a way to take the look or style of one image and apply it to another. For example, imagine you have a photo of a city's skyline and a famous painting like Van Gogh's "Starry Night." With AI style transfer, you can change the photo so it looks like Van Gogh painted it, with his special swirls and colors. This technology mixes the details of your photo with the style of the painting to create something new and beautiful.
In this guide, we will explain some basic ideas behind this method and show you how to do it yourself in ComfyUI. We'll be transforming a dancing video of a human into a dancing noodle dish.
Big fat special shout out to the original creator of this concept, the talented James Gerde! Please check out his incredible work here.
The purpose of style transfer is to generate a new image that has both the semantic content of a content image and the style of a reference style image.
Why use ComfyUI?
- User-Friendly Workflow Sharing: Download workflows with preset settings so you can get straight to work.
- Creative Applications: Ideal for artists, designers and marketers who want to create unique visuals and engaging content.
- Democratized Creativity: ComfyUI uses powerful open source AI, allowing anyone to create stunning, style-rich images and videos quickly.
One-Time Setup
Step 1: Load the ComfyUI workflow into ThinkDiffusion
Download the workflow and drag & drop or 'Load' it into your ComfyUI window, whether locally or on ThinkDiffusion. If you're using ThinkDiffusion, use the ComfyUI machine. It's necessary to use at minimum the Turbo 24gb machine, but we do recommend the Ultra 48gb machine.
Step 2: Install Custom Nodes
If there are red nodes in the workflow, it means that the workflow lacks the certain required nodes. Install the custom nodes in order for the workflow to work.
- Go to ComfyUI Manager > Click Install Missing Custom Nodes
- Check the list below if there's a list of custom nodes that needs to be installed and click the install.
Step 3: Install Models
Download the recommended models (see list below) using the ComfyUI manager and go to Install models. Refresh or restart the machine after the files have downloaded.
- Go to ComfyUI Manager > Click Install Models
- When you find the exact model that you're looking for, click install and make sure to press refresh when you are finished.
Model Path Source
The easier way to install the models, is to 'Copy Path' from the table below and paste the URL into ThinkDiffusion MyFiles using the 'upload' option. Use the 'Guide Table' to find the directory for each model.
| Model Name | Model Link Address |
|---|---|
| dreamshaper_8LCM.safetensors | |
| vae-ft-mse-840000-ema-pruned.safetensors | |
| Ghibli_v6.safetensors | |
| add_detail.safetensors | |
| Ip-adapter-plus_sd15.safetensors PLUS (High Strength) | |
| aid-RUN-Motion_Lora.safetensors | |
| AnimateLCM_sd15_t2v.ckpt | |
| control_v1p_sd15_qrcode_monster.safetensors | |
| control_v11p_sd15_lineart.pth | |
| BiRefNet-DIS_ep580.pth | |
| BiRefNet-ep480.pth | |
| Swin_base_patch4_window12_384_22kto1k.pth | |
| swin_large_patch4_window12_384_22kto1k.pth | |
| 4x_NMKD-Siax_200k.pth |
Guide Table for Upload
Tips
If you prefer to upload from your Google Drive, follow the instructions here UPLOAD HELP
Step 4: Run the workflow
Now that the hard work is out of the way, let's get creative. You need to follow the steps from top to bottom. The workflow is a one-click process after everything has been set up.
| Steps | Description / Impact | Default / Recommended Values | Required Change |
|---|---|---|---|
| Load a Dance Video or Dance Movement | Upload a video which shows a dance style or body movement. It will create a video mask using this video. Set your desired limit for frame_load_cap. Default is 0. |  |
YES |
| Load a 1 image for the background and 1 image for the foreground. | The Load Image node can be used to to load an image. Images can be uploaded by starting the file dialog or by dropping an image onto the node. Once the image has been uploaded they can be selected inside the node. I needs 2 images; 1 for background and 1 for the subject or character. |  |
YES |
| Check the video settings for dance mask | This is an area of nodes where you can set the dimension of upcale image before it will process for generation. Recommended upscale methods are lanczos and crop at the center. Otherwise you can disable for a wide angle view of output. |  |
|
| Check the Efficient Loader | A collection of ComfyUI custom nodes to help streamline workflows and reduce total node count. Set it to the recommended default value for the checkpoint, vae and lora. When writing a prompt, put descriptive words on what should be the appearance of the subject or how it appeared while moving. |  |
|
| Check the IPAdapter Plus settings and adjust the weight when necessary | These are the groups for ipadapter, a novel approach for enhancing text-to-image diffusion models with the capability to use image prompts in image generation tasks. IP-Adapter aims to address the shortfalls of text prompts which often require complex prompts to generate desired images. |  |
YES |
| Check the AnimateDiff settings and you may adjust the strength of animation | These group of nodes enhances the generation by integrating improved motion models. The only change that you can adjust is the strength of motion lora. |  |
|
| Check and adjust the controlnet strength while testing the prompt. | These groups of nodes can be used to provide further visual guidance to a diffusion model. This is an essential area because it will control the appearance of your output. |  |
YES |
| Check the KSampler and Latent Image Size and set it to preferred size | This is where your images are generated. The KSampler uses the provided model and positive and negative conditioning to generate a new version of the given latent. |  |
|
| Check the Video Combine which shows your video | This node merges a sequence of images into a cohesive video file. |  |
|
| (OPTIONAL: for High RAM/VRAM workflow) Check the settings of additional workflow for video refinement | These groups of nodes are settings for your upscale and video interpolation. See the recommended values such upscale model, steps, cfg, denoise, etc |  |
|
| (OPTIONAL: for High RAM/VRAM workflow) Check the preview of the video combine | These groups of nodes are preview for your upscale and interpolated video. |  |
Tips
Examples
You can check the examples below together with its prompt settings.
Click the toggle below for you view the recommended settings.
Cactus Dance Settings
Prompt: Translucent cactus, glistening with spikes, spiraling outwards in an green color dance, cactus made of scary thorns, glowing with an inner azure light, surrounded by a faint smoke
Steps - 10, cfg 1.5, lcm, sgm_uniform
qrcode - 0.4 , lineart - 0.4
Ice Cream Dance Settings
Prompt: delicious ice cream, dripping with delicious treat, spiraling outwards in an chilling dance, ice cream made of sweet flavor, glowing with an inner azure light, surrounded by frozen effect
Steps - 10, cfg 1.55, lcm, sgm_uniform
qrcode - 0.4, lineart - 0.5
Fire Dance Settings
Prompt: burning flame, blazing with fire aura, spiraling outwards in an intricate dance, made of delicate flame burst, glowing with an inner fire and smoke, surrounded by a smoldering amber, background is snow landscape
Steps - 10, cfg 1.55, lcm, sgm_uniform
qrcode - 0.4, lineart - 0.4
Resources
Download the Input and Output Files Here
It contains of images and videos for you to test in the workflow.
Frequently Ask Questions
How can we define Style Transfer in AI?
This process creates a new image that preserves the key elements of the original photo but mimics the artistic appearance of the second image. This technology is commonly used in digital art and photo/video editing to generate unique and striking visuals.
How to use ComfyUI with Civitai on Mac?
We recommend using ThinkDiffusion so you don't have to install locally on your mac, but here are some quick steps on installing on a Mac computer.
To use ComfyUI with Civitai on a Mac, first install Python, clone the ComfyUI repository, and set up a virtual environment. Install dependencies using pip, download models from Civitai, and configure ComfyUI to recognize them. Launch ComfyUI, access it via your browser, and load the Civitai models. Upload your content, apply the desired style transfer, then save and export your final images or videos.
How to create a dancing noodles video with AI?
Follow this guide and you can create your own dancing noodle video with ThinkDiffusion. To produce a video of dancing noodles using artificial intelligence, begin by generating a noodle dance using the picture of a noodle and background of ramen bowl within the ComfyUI interface. Subsequently, employ an AI-driven animatediff, ipadapter and controlnet nodes to imbue the noodles with dancing movements. Assemble the animations using a video editing software like Adobe, which allows you to include backgrounds, synchronize music and finetune your result.
If you’re having issues with installation or slow hardware, you can try any of these workflows on a more powerful GPU in your browser with ThinkDiffusion.
If you enjoy ComfyUI and you want to test out creating awesome animations, then feel free to check out this AnimateDiff tutorial here. And have fun out there with your noodles!
Member discussion