Black Forest Labs' Flux model became a rising star in the exciting field of generative art in August 2024 and during the time of release Flux AI image generation performed better than other models in image generation. Flux can easily generate excellent visuals with more control than many models. If you've missed our Introduction of Flux, check that one out first.
Update: Well just as we were about to publish this guide, Black Forest labs just came out with it's own "Structural Conditioning" using Canny and Depth, similar to ControlNet. We'll be sure to do a deeper dive on this as well. AI innovation never sleeps!
What is Flux AI and ControlNet?
With Flux, you can create impressive images from text prompts, descriptions or other inputs such as images. It also works well with ComfyUI, a powerful and straightforward workspace. Both experts and newbies, who may not have a technical background, can use it.
ControlNet is a tool for controlling the precise composition of AI images. Using its model, you can provide an additional control image to condition and control your image generation. It can generate detailed photos, illustrations and assets or change existing images.
Intro to Flux AI ControlNet Model
Recently XLab, InstantX, Shakker Labs and MistoAI have released ControlNets for Flux. In this guide, we'll use XLab's ControlNet models because of their quality and generation speed. XLab's collection supports 3 models for now: Canny, Depth and HED. Each ControlNet is trained on 1024x1024 resolution and works optimally for this resolution.
How to Use Flux AI with ControlNet model
Installation guide
Custom Nodes
If there are red nodes in the workflow, it means that the workflow lacks the certain required nodes. Install the custom nodes in order for the workflow to work.
- Go to ComfyUI Manager > Click Install Missing Custom Nodes
- Check the list below if there's a list of custom nodes that needs to be installed and click install.
Models
Download the recommended models (see list below) using the ComfyUI manager and go to Install models. Refresh or restart the machine after the files have downloaded.
- Go to ComfyUI Manager > Click Install Models
- When you find the exact model that you're looking for, click install and make sure to press refresh when you are finished.
Model Path Source
Use the model path source if you prefer to install the models using model's link address and paste into ThinkDiffusion MyFiles using upload URL.
| Model Name | Model Link Address |
|---|---|
| flux1-dev.safetensors | |
| flux1-schnell.safetensors | |
| flux1-dev-fp8.safetensors | |
| flux1-schnell-fp8.safetensors | |
| ae.safetensors | |
| t5xxl_fp8_e4m3fn.safetensors | |
| t5xxl_fp16.safetensors | |
| clip_l.safetensors | |
| flux-canny-controlnet-v3.safetensors | |
| flux-depth-controlnet-v3.safetensors | |
| flux-hed-controlnet-v3.safetensors |
Guide Table for Upload
Reminder
Step-by-step Flux Controlnet image generation
Now that the hard work is out of the way, let's get creative. Follow the steps from top to bottom.
| Steps | Default Nodes |
|---|---|
| Load an Image |  |
| Set your desired image size |  |
| Set the correct models as seen on the image. |  |
| Write a good prompt which describes what the outcome will be. |  |
| Set a ControlNet and equivalent preprocessor model as seen in the image. |  |
| Set the desired settings as seen on the image. A preview after generation porcess will appear in this area. Finally, run the queue prompt. |  |
Quick Tips
Flux AI ControlNet Examples
Click the toggle arrow to view the generation settings below.
Man in the Boat
Prompt: A solitary man navigates a small wooden boat down a winding river, surrounded by dense, mist-covered forests. The morning light filters through the trees, casting dappled shadows on the calm water as he paddles quietly, the sounds of nature his only companion.
Seed - 520434187747470 , controlnet > depth
steps - 30, cfg - 1, denoise - 1
diffusion model - dev-fp8, clip model - t5xxl_fp8
controlnet strength - 0.80
Simple House
Prompt: A sprawling, luxurious mansion stands majestically against the backdrop of rolling grasslands. The house boasts floor-to-ceiling windows that offer panoramic views of the serene landscape, elegant stonework, and manicured gardens with winding pathways leading to an infinity pool overlooking the endless green
Seed - 1100703369842817 , controlnet > canny
steps - 30, cfg - 1, denoise - 1
diffusion model - dev-fp8, clip model - t5xxl_fp8
controlnet strength - 0.80
Mountain Hell
Prompt: A towering, jagged mountain looms ominously under a blood-red sky, its peaks shrouded in swirling, sulfurous clouds. Rivers of molten lava snake down its scorched, blackened slopes, and the air is thick with the acrid stench of brimstone as eerie, flickering flames dance around the base, creating an otherworldly, infernal landscape.
Seed - 699854010079247 , controlnet > depth
steps - 30, cfg - 1, denoise - 1
diffusion model - dev-fp8, clip model - t5xxl_fp8
controlnet strenght - 0.80
Man from Future in Office
Prompt: A man dressed in sleek, futuristic attire sits incongruously inside a rustic, old wooden office. The room is filled with the scent of aged wood and the creak of old floorboards, with antique books lining dusty shelves. Advanced technological devices and holographic screens are set up amidst the vintage furniture, their glowing interfaces starkly contrasting with the dim, warm light from an oil lamp on the worn wooden desk. The man surveys the room with a blend of curiosity and purpose, his advanced gadgets reflecting in the polished brass fixtures.
Seed - 494503909866766, controlnet > HED
steps - 20, cfg - 1, denoise - 1
diffusion model - dev-fp8, clip model - t5xxl_fp8
controlnet strenght - 0.50
If you’re having issues with installation or slow hardware, you can try any of these workflows on a more powerful GPU in your browser with ThinkDiffusion.
If you enjoy ComfyUI and you want to test out creating awesome animations, then feel free to check out this AnimateDiff tutorial here. Happy creating!
Member discussion