🐼 stable diffusion角色设计 用Controlnet分别在SD WEBUI和ComfyUI中绘制多角度人像的方法
TLDRThe video tutorial introduces a method for character design using AI painting, specifically discussing the challenges of maintaining consistency in character appearance from different angles. It highlights a technique involving the use of a diffusion algorithm-based tool called SD, and the training of a model called Lora with images generated entirely from SD. The process involves creating a sequence of multi-angle character images, training Lora with these images, and refining the results. The tutorial provides实操 tips, addresses potential issues with AI-generated images, and demonstrates how to use COMFY UI for efficient deployment of the character design system.
Takeaways
- 🎨 The video discusses a method for character design using AI painting, specifically focusing on achieving consistency in character portrayal across different angles.
- 🤖 The method is based on an article by a foreign expert, which addresses the challenge of training Lora models with 100% virtual characters generated by SD (Stable Diffusion).
- 📸 The process involves creating a sequence of multi-angle character images using text-to-image generation, which are then used to train the Lora model.
- 📈 The article provides a template with 15 angles for generating faces in open pose and a 15-grid image for segmentation during the image generation process.
- 🖼️ Images generated should be 256x256 pixels in size, and when combined, the total size should be 1328x800 pixels, considered optimal for current hardware conditions.
- 💡 The video emphasizes the importance of the image sequence for training Lora, as it allows for image segmentation and subsequent training.
- 🔍 The video acknowledges the unpredictability of SD-generated images, which may result in image degradation or inconsistencies in the 15-grid layout.
- 🛠️ The presenter shares personal solutions for potential issues, such as adjusting the weight of the open pose control and using tile models and upscaling algorithms for image repair.
- 🎓 The video provides a practical demonstration of the process using the Deliberate model and the latest version of SD, including downloading templates and setting up CONTROLNET units.
- 🌐 The presenter's approach to character design can be adapted to create different characters by changing the prompt words while maintaining consistency in the design.
- 🔗 The video mentions the availability of the节点 system for character design on the presenter's ko-fi store and offers a way for viewers to receive it for free by following the presenter's Twitter account.
Q & A
What is the main topic of the 28th episode of the SD series tutorial?
-The main topic of the 28th episode of the SD series tutorial is character design using AI painting.
What is the significance of consistency in character design in AI painting?
-Consistency in character design is crucial as it ensures that the character appears uniform across different angles and perspectives, which is currently a hot topic and considered the 'holy grail' in the AI community.
What is the traditional solution for maintaining character consistency in AI painting?
-The traditional solution for maintaining character consistency in AI painting is through 3D modeling.
What is the new method introduced in the tutorial for character design using SD?
-The new method introduced for character design using SD involves training Lora with a sequence of multi-angle images of a character created entirely through AI painting.
How many angles are included in the template provided by the article's author for generating different angles of a face?
-The template provided by the article's author includes 15 angles for generating different angles of a face.
What is the minimum number of images required to train Lora?
-The minimum number of images required to train Lora is 15.
What are the two main CONTROLNET models used in the character design process?
-The two main CONTROLNET models used in the character design process are the open pose skeletal model and the lineart line稿 model.
What is the purpose of using the open pose skeletal model in the CONTROLNET?
-The purpose of using the open pose skeletal model in the CONTROLNET is to control the direction of the face and to generate different angles of the face based on the skeletal structure.
What is the role of the lineart model in the CONTROLNET?
-The role of the lineart model in the CONTROLNET is to perform a grid split on the generated image, which helps in creating separate images for training Lora.
How can one improve the quality of images generated with Lora?
-One can improve the quality of images generated with Lora by using high-quality source images from the AI painting process, optimizing the CONTROLNET settings, and applying image enlargement and repair techniques using models like tile and ESRGAN 4X.
What is the issue that may arise when using the new method for character design, and how can it be addressed?
-The issue that may arise when using the new method for character design is the potential for image degradation or 'image崩坏'. This can be addressed by using image enlargement and repair techniques in the图生图 phase, and by adjusting the settings and权重 in the CONTROLNET models to achieve better results.
How can the new character design method be applied in different scenarios?
-The new character design method can be applied in different scenarios by changing the prompt words to generate characters with varying features, emotions, and accessories, as demonstrated in the tutorial with the 'one girl blue hair, smile white background, red head wear glasses' prompt.
Outlines
🎨 Introduction to Character Design with AI Painting
The video begins with an introduction to the 28th episode of the SD tutorial series, focusing on character design. The host discusses the widespread interest in AI painting since its inception and shares their experience with the topic. They mention a recent article by an international expert on using SD for character design, which they found useful and decided to share in a tutorial. The host emphasizes the importance of consistency in character design across different angles, traditionally achieved through 3D modeling. However, the article introduces a novel method using SD and Lora training, specifically for characters created entirely through AI-generated images. The process involves creating a sequence of multi-angle character images, training Lora with these images, and refining the results to achieve a consistent character design. The host also provides a link to the article in the video description and outlines the three main stages of the process: creating a sequence of character images, training Lora, and refining the results.
📚 Understanding the Role of Lora and Training Process
This paragraph delves into the specifics of Lora training and the challenges associated with it. The host explains that Lora training typically relies on real-life人物 or existing素材, which may not align with AI-generated characters. The article's author addresses this by proposing a method to train Lora using 100% virtual characters. The process is divided into three stages: generating a sequence of multi-angle character images, training Lora with these images, and refining the model. The host emphasizes the importance of the image sequence, as it allows for image segmentation and Lora training. They also discuss the practical aspects of training, such as the minimum number of images required and the recommended image size for Lora training. The host shares their实操 experience, including the use of腾讯云16G显存 for image generation and the potential issues with lower VRAM.
🖌️ Practical Demonstration of Character Design with SD
The host transitions to a practical demonstration of character design using SD. They guide the audience through the process of using the templates provided in the article, including a 15-angle facial template and a 15-grid image for SD's understanding of individual images. The host explains the use of CONTROLNET for image segmentation and the selection of open pose and lineart models for facial direction and detail control. They also discuss the use of tile models and upscaling algorithms to enhance image quality. The host demonstrates the generation of a simple image using the 'one girl' prompt and shows how to放大 the image using tile models and upscaling algorithms. They then provide a detailed explanation of the CONTROLNET models' roles and how they contribute to the final image quality.
🔄 Deployment and Optimization of Character Design System
The host introduces the deployment of the character design system using COMFY UI, which allows for a more streamlined and batch processing of images. They explain the system's capability to output multiple sets of images and discuss the use of CONTROLNET models for image control. The host also shares their拓展用法 of the 15-grid template to a 4-grid template and the rationale behind using two CONTROLNET open pose models to avoid facial duplication in the grid. They provide insights into the impact of prompt words on the final image, emphasizing the importance of consistent seed values and sampling steps. The host also discusses the use of different models and their impact on image quality, offering suggestions for optimizing the training process and addressing common issues such as facial recognition inaccuracies and image degradation.
Mindmap
Keywords
💡SD系列教程
💡角色设计
💡扩散算法
💡Lora
💡多角度角色图片序列
💡open pose骨骼图
💡tile模型
💡放大算法
💡COMFY UI
💡提示词
💡Web UI
💡节点系统
Highlights
The introduction of a method for character design using AI painting, which has been widely discussed since the emergence of AI painting.
The reliance on traditional 3D modeling to ensure consistency of characters from different angles, and the exploration of new methods using SD (Stable Diffusion).
The article by a foreign expert that proposes a novel approach to training Lora with 100% virtual characters created by SD, instead of real-life figures.
The three-stage process involving creating a sequence of multi-angle character images, training Lora with these images, and refining the results.
The use of a 15-angle face template for generating open pose images and a 15-grid image for SD to recognize individual images.
The minimum requirement of 15 images for Lora training and the recommended image size of 256x256.
The practical application of tile models and upscaling algorithms to enhance image quality and size for Lora training material.
The acknowledgment of the author's innovative approach and generous sharing of knowledge.
The unpredictability of SD-generated images and the potential for image degradation, which is characteristic of SD rather than a flaw in the method.
The demonstration of the实操 (practical operation) process using the Deliberate model and SD version 1.6.
The utilization of CONTROLNET to upload and process the template images for open pose recognition and grid segmentation.
The testing of the method with a simple prompt 'one girl' and the observation of the generated 15-angle facial images.
The repair of minor image flaws using tile models and upscaling techniques to achieve higher quality images for Lora training.
The explanation of the role of the two CONTROLNET models in the image generation process and the importance of controlling facial direction and details.
The deployment of the character design system using COMFY UI, which allows for streamlined operations and batch image output.
The拓展 (expansion) of the method to a 4-grid image output by using two CONTROLNET open pose models to control facial orientation.
The emphasis on maintaining consistent prompt words to ensure uniformity in the final images and the impact of emotional prompts on the tone of the generated images.
The provision of solutions for common issues encountered during the image generation process, such as facial recognition inaccuracies and prompt word optimization.
The availability of the COMFY UI node system for purchase and the offer to receive it for free by following the creator's social media and providing an email address.