An Introduction to ControlNet and OpenPose

Introducing ControlNet, a revolutionary Stable Diffusion model designed to facilitate the replication of compositions or human poses from a reference image.

With ControlNet, the frustrations of Stable Diffusion users are alleviated, as it provides a precise level of control over subject placement and appearance.

In this article, we will delve into the comprehensive details of ControlNet OpenPose, covering:

  • An overview of ControlNet OpenPose and its functionalities
  • Key settings and practical applications for various use cases.

Prepare to unlock the full potential of ControlNet OpenPose and take your creative endeavours to new heights.

What is ControlNet OpenPose?

ControlNet OpenPose refers to a specific component or feature that combines the capabilities of ControlNet with OpenPose, an advanced computer vision library for human pose estimation. By integrating ControlNet with OpenPose, users gain the ability to control and manipulate human poses within the Stable Diffusion framework.

By leveraging the combined power of ControlNet and OpenPose, Stable Diffusion users can achieve more controlled and targeted results when generating or manipulating compositions involving human subjects.

What OpenPose Pre-processors are available?

The OpenPose pre-processors are:

  • OpenPose: eyes, nose, eyes, neck, shoulder, elbow, wrist, knees, and ankles.

  • OpenPose_face: OpenPose + facial details

  • OpenPose_hand: OpenPose + hands and fingers

  • OpenPose_faceonly: facial details only

  • OpenPose_full: All of the above

How to use ControlNet and OpenPose

  • (1) On the text to image tab...
  • (2) upload your image to the ControlNet single image section as shown below
  • (3) Enable the ControlNet extension by checking the Enable checkbox.
  • (4) Select OpenPose as the control type
  • (5) Select "openpose" as the Pre-processor. OpenPose detects human key points like the positions of the head, arms, etc.
  • (6) Choose "control_sd15_openpose" as the ControlNet model, which is compatible with OpenPose.
  • You can then type in your positive and negative prompts and click the generate button to start generating images using ControlNet.
  • You will see the generated images following the pose of the input image, with the last image showing the detected keypoints.

Disabling ControlNet:

  • When you're done with the ControlNet extension, uncheck the Enable checkbox to disable it.

ControlNet Settings

Pre-processors:

  • ControlNet provides various pre-processors for different tasks.
  • To enable the preview of the pre-processor's output, select Allow Preview.
  • Optionally, choose Pixel Perfect to generate the pre-processed image with the specified height and width from text-to-image.

Pre-processors and Models:

  • ControlNet requires selecting a pre-processor and a compatible model.
  • The pre-processor modifies the original image, and only the pre-processed image is used for ControlNet.

To choose the pre-processor and model:

  • Click on the explosion icon next to the Pre-processor dropdown menu.
  • Select the desired pre-processor from the available options.
  • Ensure the corresponding model is selected in the ControlNet's Model dropdown menu.

Now that we have covered the features and settings for using ControlNet, let's go through each model to see what can be done!

Examples of using OpenPose

  • OpenPose detects human key point positions of the eyes, nose, eyes, neck, shoulder, elbow, wrist, knees and ankles
  • It is suitable for copying human poses while excluding other details like outfits, hairstyles, and backgrounds.

For example, if we upload a picture of a man in a certain pose, we can select the control type to be OpenPose, the pre-processor to openpose and control_sd15_openpose as the model.

Positive Prompts: (a photograph of a beautiful girl jumping in the air wearing trainers), (ultra detailed face:1.3), makeup, shot on a Sony mirrorless camera, DSLR, 50mm lens f/2.8, ultra detailed, 8k, morning golden hour

Negative Prompts: deformed, disfigured, nudity, nsfw, blurry, lowres, cartoon, anime, multiple people, warped face

OpenPose Full

  • OpenPose_full detects everything openPose face and openPose_hand do.
  • OpenPose_full combines all the key points detected by OpenPose, including facial details and hands/fingers.

For example, if we upload a picture of a man waving, we can select the pre-processor to openpose_full and control_sd15_openpose as the model.

Positive Prompts: Photograph of a man sat in a chair, waving, shot on a Sony mirrorless camera, DSLR, 50mm lens f/2.8, ultra detailed, 8k, [ at night / morning golden hour ]

Negative Prompts: Nudity, nsfw, blurry, lowres, cartoon, anime

OpenPose_face

  • OpenPose_face includes all the key points detected by OpenPose, along with additional facial details.
  • It is useful for copying facial expressions along with the body pose.

For example, if we upload a picture of a man waving, we can select the pre-processor to openpose_face and control_sd15_openpose as the model.

Positive Prompts: Photograph of a lady sat in a chair, waving, shot on a Sony mirrorless camera, DSLR, 50mm lens f/2.8, ultra detailed, 8k, [ at night / morning golden hour ]

Negative Prompts: Nudity, nsfw, blurry, lowres, cartoon, anime

OpenPose_faceonly

  • OpenPose_faceonly focuses solely on detecting facial details and excludes other key points.
  • This is helpful for copying only the face without including other body key points.

For example, if we upload a picture of a man laughing, we can select the pre-processor to openpose_faceonly and control_sd15_openpose as the model.

Positive Prompts: (a photograph of a man laughing:1.2), shot on a Sony mirrorless camera, DSLR, 50mm lens f/2.8, ultra detailed, 8k, morning golden hour

Negative Prompts: Nudity, nsfw, blurry, lowres, cartoon, anime

OpenPose_hand

  • OpenPose_hand detects key points using OpenPose, including the hands and fingers. This can be useful as stable diffusion can sometimes really struggle to generate realistic hand poses.
  • It is beneficial for copying hand poses along with the body posture.

For example, if we upload a picture of a man doing the thumbs up, we can select the pre-processor to openpose_hand and control_sd15_openpose as the model.

Positive Prompts: (a photograph of a beautiful girl doing the thumbs up), shot on a Sony mirrorless camera, DSLR, 50mm lens f/2.8, ultra detailed, 8k, morning golden hour

Negative Prompts: deformed, disfigured, Nudity, nsfw, blurry, lowres, cartoon, anime, multiple people

That's everything covered for using ControlNet with OpenPose! Experiment with different pre-processors and models to generate images based on human poses and key points.

And if you’re having issues with installation or slow hardware, you can try any of these workflows on a more powerful GPU in your browser with ThinkDiffusion.

Try with ThinkDiffusion

If you'd like to turn any of your masterpieces into a QR code, check out this post here. Happy posing!

Let's see your moves!