ComfyUI教程 5分钟制作丝滑的AI视频-AnimateDiff

1. AnimateDiff的技术原理

AnimateDiff可以搭配扩散模型算法（Stable Diffusion）[来生成高质量的动态视频](https://blog.youkuaiyun.com/weixin_66363548/article/details/134036591)，其中动态模型（Motion Models）用来实时跟踪人物的动作以及画面的改变。

2. 环境搭建

这里我们使用ComfyUI来搭配AnimateDiff做视频转视频的工作流。

我预设ComfyUI的环境已经搭建好了，这里就只介绍如何安装AnimateDiff插件。

3. ComfyUI AnimateDiff视频转视频工作流

找到一个真人跳舞视频

提取视频帧，并用ControlNet Openpose提取人物动作信息

根据视频帧里面的动作信息，使用SD重新绘制每一帧视频

组合出完整视频

3.1 读取ComfyUI工作流

直接把下面这张图拖入ComfyUI界面，它会自动载入工作流，或者下载这个工作流的JSON文件，在ComfyUI里面载入文件信息。

工作流参考
https://pan.quark.cn/s/c828ca989603

3.2 安装缺失的node组件

第一次载入这个工作流之后，ComfyUI可能会提示有node组件未被发现，我们需要通过ComfyUI manager安装，它会自动找到缺失的组件并下载安装（！！需要网络通畅）。

点击ComfyUI Manager按钮打开组件管理

再点击Install Missing Custom Nodes安装组件

安装完成后要重启ComfyUI才能完成载入刚才缺失的组件。

！如果网络状态不好，可能会导致下载组件失败，需要多试几次

3.3 下载一个AI绘画模型

这里比较推荐C站上的Dreamshaper 8这个模型，实测动画人物效果比较好，当然大家也可以自己去测试其他的模型，有好的效果，欢迎评论留言。

下载好的模型放到ComfyUI > models > checkpoints目录。

然后刷新页面

在Load Checkpoint这个组件里可以选择不同的模型

3.4 再下载一个VAE模型

下载VAE，放到ComfyUI > models > vae这个目录里。

然后刷新页面

在Load VAE里可以选择不同的编码器模型

3.5 下载AnimateDiff动态特征模型

下载mm_sd_v15_v2.ckpt，放到ComfyUI > custom_nodes > ComfyUI-AnimateDiff-Evolved > models文件夹。

刷新页面

在AnimateDiff Loader里，可以选择我们需要的动态特征模型

3.6 下载Openpose ControlNet模型

下载control_v11p_sd15_openpose.pth，放到ComfyUI > models > controlnet文件夹。

刷新页面

在 Load ControlNet Model (Advanced)里，选择对应模型

下载lora

https://huggingface.co/guoyww/animatediff/tree/main

3.7 上传原视频

以这个视频（点击下载）作为示例

在Load Video（Upload）选择视频素材

3.8 生成视频

然后就可以点击Queue Prompt开始生成视频了

视频生成的大部分时间都集中在KSampler这个组件里，在生成过程中上面有一个进度条

另外在打开ComfyUI的windows终端里也可以看到进度以及错误信息

生成结束后，视频会在AnimateDiff Combine这个组件里合成

下面是一个结果的对比示例

Ai跳舞教学案例视频

3.9 文生动画案例

链接: https://pan.baidu.com/s/1zrNOKMYO9_C3tdOUW7L0EQ?pwd=7zjd
提取码: 7zjd 复制这段内容后打开百度网盘手机App，操作更方便哦

comfyui 文生动画

"0" :"Portrait,Elsa of Arendelle,cute,seductive,innocent,light smile,plump lips,slender body,(dark cape),(dark dress),fur trim,single braid,snowflake pattern,snowy mountain background ",
"15" :"girl,cute,seductive,innocent,light smile,plump lips,slender body,high detailed gothic dress,(red|white|black),vibrant color,mistery forest background , (closed eyes:1.8)",
"30" :"Full Body Shot, girl, cute, seductive, innocent, light smile:0.3, plump lips, slender body, trees, intricate, gold leaf accents, decorative surfaces, Klimt's Art Nouveau style, forest background,  ",
"45" :"18 year old, solo, 1girl, long hair, brown hair, blue eyes, long length hair, medium size breasts, petite,,  (white  sleeve short shirt, fitted shirt), beige executive pants, public, seductive smile,rabbit pose,(smileing:1.8)",
"60" :"Cowboy Shot, girl, cute, seductive, innocent, light smile, plump lips, slender body, solo, dark red dress, fog atmosphere, fashion photography, cinematic photography, vibrant colors, cinematic lighting, abandoned old train station, fog background"

宋江的一生

“0” : “a new born baby boy lying on a crib crying, top view, In Songjia Village, Yuncheng County, Shandong”,
“10” : “a new born baby boy lying on a crib crying, top view, In Songjia Village, Yuncheng County, Shandong”,
“20” : “a 3 years old, a young boy named Song Jiang grows up under the care of his family, his eyes sparkling with a thirst for knowledge and curiosity about the world,”,
“30” : “a 3 years old, a young boy named Song Jiang grows up under the care of his family, his eyes sparkling with a thirst for knowledge and curiosity about the world,”,
“40” : “a 5 years old, In the private school, Song Jiang is intently reading classics and histories, with his teachers and classmates praising his intelligence and leadership abilities”,
“50” : “a 5 years old, In the private school, Song Jiang is intently reading classics and histories, with his teachers and classmates praising his intelligence and leadership abilities”,
“60” : “20 years old, As a bailiff in Yuncheng County, Song Jiang is known for his generosity and sense of justice, often seen helping the people in need.”,
“70” : “20 years old, As a bailiff in Yuncheng County, Song Jiang is known for his generosity and sense of justice, often seen helping the people in need.”,
“80” : “30 years old, In an unexpected turn of events, Song Jiang is forced to leave his hometown, disappearing into the night on a perilous journey of exile.”,
“90” : “30 years old, In an unexpected turn of events, Song Jiang is forced to leave his hometown, disappearing into the night on a perilous journey of exile.”,
“100” : “40, years old, On Liangshan Marsh, Song Jiang stands among the heroes, his eyes resolute as he commands one battle after another against the imperial forces, with the Liangshan banner fluttering in the wind.”,
“110” : “40, years old, On Liangshan Marsh, Song Jiang stands among the heroes, his eyes resolute as he commands one battle after another against the imperial forces, with the Liangshan banner fluttering in the wind.”,
“120” : “50, years old, After accepting the amnesty from the court, Song Jiang and his brothers don the uniforms of the imperial army, bravely fighting in the battle against Fang La, despite the complex emotions in their hearts.”,
“130” : “50, years old, After accepting the amnesty from the court, Song Jiang and his brothers don the uniforms of the imperial army, bravely fighting in the battle against Fang La, despite the complex emotions in their hearts.”,
“140” : “60, years old, With the court’s bestowal, Song Jiang becomes the Duke of Martial Virtue, but his brows are furrowed, as he knows the glory hides endless conspiracies and betrayals.”,
“150” : “70 years old, In a dimly lit room, Song Jiang sits quietly at a table, holding a cup of poisoned wine. His eyes reflect a review of the past and helplessness about the future, and finally, he drinks the bitter wine of fate.”,
“160” : “70 years old, In a dimly lit room, Song Jiang sits quietly at a table, holding a cup of poisoned wine. His eyes reflect a review of the past and helplessness about the future, and finally, he drinks the bitter wine of fate.”,
“170” : “70 year old man, top view, lying at a table, eyes closed”

4. Debug

其中有三个组件会储存中间信息帮助分析错误

视频帧

ControlNet OpenPose处理结果

结果视频帧

5. 其他设置

5.1 通过改变图片种子值（seed）来修改图片内容

5.2 修改人物提示词

prompt prefix: 图片主体

prompt travel: 变化的背景内容，会根据不同的frames添加进图片

在这个例子中：

第0-23帧，画面提示词是：

High detail, girl, short pant, t-shirt, sneaker, a modern living room

第24-59帧：

High detail, girl, short pant, t-shirt, sneaker, beach and sun

第60帧之后：

High detail, girl, short pant, t-shirt, on the moon

5.3 输入视频参数

frame_load_cap : 载入视频的最大帧数

select_every_nth : 每隔～帧载入1帧画面，这样会导致跳过一些帧，导致视频不连贯，但是由于减少了需要处理的帧数，这样可以提高处理速度