2023-09-12 12:25:10
ControlNet’s OpenPose is the constraint type that attracted everyone’s attention at the beginning. By extracting the action pose skeleton diagram of the character in the original image, we can more accurately control the posture of the imaged character. Even some action poses that cannot be described by text description alone can be easily achieved. OpenPose Preprocessor & Example OpenPose currently has 6 preprocessors: openpose, openpose_full, openpose_hand, openpose_face, openpose_face_only, dw_openpose_full. Let’s look directly at imaging examples to compare the differences in sensitivity (accuracy) of each preprocessor in different action postures, feature extraction, and final imaging. First of all, let’s test with the easiest-to-recognize front-facing portrait of a person. (The prompt words are all blank) openpose: only extracts the approximate skeleton position of the character. Since openpose only extracts the position of the character’s whole body skeleton and lacks hand joint information, the final image may be as shown below. The left hand of the character does not have a pocket on the waist, but the entire body is The palm is missing (not drawn). openpose_hand: Extract the approximate full body skeleton position of the character + hand joints openpose_face: Extract the approximate full body skeleton position of the character + facial contour/facial features position. Although the hand joints were not extracted, this time the SD left hand brain made it very natural and reasonable. openpose_face_only : Extract only the facial outline/facial features position openpose_full : Extract the approximate whole body skeleton position of the character + hand joints + facial outline / facial features position dw_openpose_full : Extract the approximate full body skeleton position of the character + hand joints + facial outline / facial features position, the same as openpose_full , the skeleton, face and hand joints are all extracted, but in this picture dw_openpose_full, only a small part of the back of the right hand is also detected. At present, most of the full-body portraits have been tested, and the sensitivity (accuracy) of dw_openpose_full is the best. The action postures of the characters in the original picture above are relatively simple and easy to identify, and the pictures produced by each preprocessor are not very different (all can produce normal and reasonable pictures). But when the movements and postures of other characters are more complex later (depth or overlapping limbs need to be considered), the gap between the preprocessors can be clearly seen. Next, let’s test a more difficult action posture. In the original picture below, the character’s left and right feet stagger forward and backward. (The prompt words are all blank.) As can be seen from the preprocessing diagram below, openpose_full failed to extract the joints of the right hand, while dw_openpose_full successfully extracted all action posture features (body skeleton, joints of both hands, and face). But as mentioned before in the depth constraint article, the character skeleton image information of Openpose is flat. For the spatial position of the left and right feet staggered forward and backward, the final imaging image cannot fully restore the effect of the original image. (Although the right foot is staggered behind, it is slightly leaning out rather than completely tucked in behind). Maybe the difference in the depth of the character’s action poses in the original picture above is not obvious, so let’s test a more obvious example to see if the Openpose preprocessor will overturn. (This picture needs a hint: a girl taking selfies with smart phone) Like the previous example, the extraction effect of dw_openpose_full is better than that of openpose_full (although dw_openpose_full does not extract the hand joints completely correctly, at least both hands are recognized) . However, in addition to the hand problem in the two final images, the character’s head-to-body ratio and front-to-back depth are still very weird/unnatural. Now that we think there is a problem with the depth perspective, let’s try adding a depth constraint to assist imaging. After adding the depth constraint, the overall feeling of the picture becomes more natural and reasonable. Or actually, taking the actions of the characters in this picture, to restore the action postures of the characters in the picture, just use a depth control constraint. openpose is not practical. Finally, let’s take a look at the effects of feature extraction on different types of character images through openpose. A group photo of many people in the picture: (Prompt word: man and women looking at a tablet cheerfully). This SD naturally automatically draws the tallest skeleton as a man, and the other skeletons as women. Although there is a slight error in the final imaging of dw_openpose_full (the front and back order of the overlapping arms of the man and woman in the red box is slightly different from the original image), there is no big problem overall. Openpose_full loses more action posture information when multiple people appear in the screen at the same time and are very close to each other. From the previous case of a group photo, we found that it seems that it is particularly easy to misidentify the parts where the characters’ limbs overlap in the picture, so let’s try another photo that finally overturned all the preprocessors: When we got to this picture, we were very surprised. dw_openpose_full, which had always outperformed openpose_full, overturned more seriously. Neither of these two preprocessed images can be used directly. But can we only be limited by the limitations of the existing preprocessor effects, and cannot generate a character skeleton diagram that fully meets our needs regardless of any action posture? Of course not~ When all the preprocessors in Openpose cannot meet our needs, we can also use various plug-in extensions to adjust the skeleton diagram to correct the parts where the preprocessor judgment/extraction fails, or we can generate the skeleton diagram ourselves. OpenPose’s Model reads and generates images. OpenPose skeleton diagram modification/adjustment plug-in tool Search “openpose” under the extensions page. You can see that there are currently 4 related plug-in extensions available. Regarding how to install plug-in extensions, please refer to the previous article: → AI Drawing-Stable Diffusion 007- Installation, update, removal and backup of plug-in extensions. Here we first install the first one “sd-webui-openpose-editor-editing” “To test. After the installation is complete and the SD is restarted, click “Edit” in the lower right corner of the Openpose preprocessing image window to enter the page for modifying/adjustment of the preprocessing image: This editing screen feels straight to use, and the canvas area on the right slides. Use the mouse wheel to zoom in and out, and use the space bar + left mouse button to move the canvas. In the lower left corner, you can add or remove character groups by yourself, which is the same as the layer operation in Photoshop. The only thing to note is that when you want to zoom in, out, rotate and move the entire part group (the right hand in the picture below), when the drop-down menu is not expanded, just click on the part position and the outline of the deformation operation will appear. But if the drop-down menu is expanded, you have to use the drag box to select all the joints of the hand. If you want to make adjustments to individual nodes in the group (such as adjusting the position of the right hand joint as shown below), you must first expand the group menu, and then click on each node to operate. After spending some time, the skeletons and hand joints of the father and daughter were finally completed. The skeleton diagram is then transferred back to ControlNet. First, only use a single OpenPose to constrain the imaging. (Prompt word: father holding little girl, except that the canvas is changed to have the same length and width as the original image. All other parameter settings are directly using the default) The skeleton diagram restores the “flat” action posture, but the interlaced front and back positions of the character’s limbs cannot be controlled. . ~ The little girl’s left hand should cover her right hand, and the man’s right hand should be behind the little girl’s right foot to hold her butt ~ Maybe there is a chance that the plane skeleton diagram alone can produce the same front and back position as the original picture (but I have drawn dozens of them Zhang Cai has appeared in 1 or 2 pictures @@), but if we just rely on luck to brush the pictures, that is not the purpose of spending more time learning SD. Therefore, let’s try it with other constraint types. Add one more depth constraint. Just looking at the depth preprocessing picture, at least the little girl’s right foot should be in front of the man’s right hand, but no matter how you try (even if you increase the weight of the depth constraint + depth’s Control Mode, use ControlNet More importantly), everything is still the same, and the front and back positions that the original image should have cannot be displayed. Since the depth constraint does not work, let’s try to match it with normal. Intuitively, the effect of the normal preprocessing map should be more accurate than depth for the positional relationship that is close to the front and back. But in the final drawing, only the part where the little girl’s left hand covered her right hand was corrected, but the man’s right hand was still stubbornly drawn in front of the little girl’s right foot. It’s really hard to understand here. The pre-processed pictures clearly indicate that the little girl’s right foot is more protruding than the man’s right hand (even if it is more important to increase the weight of the normal constraint + normal’s Control Mode and use ControlNet), Why do you SD always have to draw your right hand forward~ This way of holding is not ergonomic, okay? Your right hand should be dragged on the little girl’s butt to hold her firmly~~~Depth is not good, nor is normal. Finally, I will Try one, especially add line constraints (Lineart) to the part where the right hand is placed behind the little girl. With Lineart (Lineart constraint weight preset 1 + Control Mode with Balanced), there is finally almost half a chance of correctly drawing the man’s right hand behind the little girl’s right foot, and the little girl’s left hand is also covering the right hand. Moreover, since I have modified the Lineart preprocessing image here to remove lines other than non-action poses (in order to prevent the characters from looking too similar to the original image), at this time, try not to set the weight of the Lineart constraint too high or It is more important to use the Control Mode of Lineart and not to set ControlNet, because there is a high chance that the third and fourth heads will be drawn inexplicably. This is because you let the SD place too much emphasis on the Lineart pre-processed image, but in the image There are too many places where no line instructions are given, so SD has to use its imaginative skills to draw some weird and deformed pictures for you. Finally, I want to increase the probability of correct action postures in the pictures, and open three ControlNet types at a time to constrain the imaging. When the three ControlNets are used together, the action poses of the pictures produced each time are almost exactly what I want. However, for some reason, the pictures produced become more and more like retro old photos. It seems that this is another issue that needs to be dealt with later… For this action, I spent an entire followingnoon brushing pictures. I sighed in my heart, give me a camera. Sometimes it is quite tiring to guess at pictures for a long time @ @ ~ Why not go out and take real photos~ After taking the photos you want, you can also relax your mind and body~ (PS. Use SD to brush pictures, especially if you have been using ControlNet. Sometimes following using it for a long time I feel that the picture I painted is weird (the picture below shows that a certain parameter is abnormally adjusted too high or too low), but it is obviously not that the parameter is not adjusted correctly, but it is more like that when the SD is exposed, just close the SD and restart it~ A lot After that, all the problems that had been bothering me disappeared~) As for the other three plug-in extensions for modifying and adjusting the skeleton diagram, I only used the 3D Openpose Editor tab. If you think the other two are easier to use, you can try them out. For me, the first two (“sd-webui-openpose-editor-editing” and 3D Openpose Editor tab) are enough (and easier to operate). The operating interface of the 3D Openpose Editor tab may not be easy to control for people who have not used 3D software. However, because it has more parameters to adjust, action postures can be adjusted more finely. Those who are interested can first go to its online web version to try out the operation, and then install it on the SD if they find it necessary (easy to use). →
1694531163
#DrawingStable #Diffusion #ControlNet #Action #Posture #Constraints
© 2024 All rights reserved