【Paddle笔记】使用PaddleGan工具包实现表情迁移(First Order Motion)

原创 已于 2023-04-30 23:39:43 修改 · 1.6k 阅读 · 6 ·
CC 4.0 BY-SA版权
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
文章标签：

#笔记 #深度学习 #pytorch #人工智能

于 2023-04-29 21:57:47 首次发布
本文介绍了如何使用PaddleGan工具包进行表情迁移，详细讲解了从环境安装、数据集获取到模型训练的全过程。在安装中涉及Conda、PyTorch、Tensorflow和Paddle框架，数据集部分涵盖了VoxCeleb和Fashion，最后通过First Order Motion Model实现了表情迁移。
部署运行你感兴趣的模型镜像

Paddle笔记 使用PaddleGan工具包实现表情迁移 FOM

PaddleGan工具包提供了Motion Driving DEMO，实现用一个视频驱动图片成动画，让图片做出与驱动视频中相似的动作。

  • 使用S3FD人脸检测模型将照片中的每张人脸检测出来并抠出
  • 使用First Order Motion模型对抠出的每张人脸进行脸部表情迁移
  • 将完成表情迁移的人脸进行适当剪裁后贴回原照片位置
  • 人脸增强特效使得驱动后的大大提升视频中人脸清晰度

PaddleGAN针对人脸的相关处理提供faceutil工具，包括人脸检测、五官分割、关键点检测等能力。
在这里插入图片描述

1、环境安装

1.1 运行环境

1.1.1 Conda虚拟环境

创建ppai
conda create -n ppai python=3.7
激活环境
conda activate ppai

1.1.2 PyTorch

Python3.7推荐版本:
pytorch1.13.1 + torchvision0.14.1 + torchaudio0.13.1 + cuda11.7
conda install pytorch==1.13.1 torchvision==0.14.1 torchaudio==0.13.1 pytorch-cuda=11.7 -c pytorch -c nvidia

安装遇到疑难参考 这里

1.1.3 Tensorflow

Python3.7推荐版本:
tensorflow-gpu 1.15
pip install tensorflow-gpu==1.15 -i https://pypi.tuna.tsinghua.edu.cn/simple

安装遇到疑难参考 这里

1.2 Paddle核心框架

1.2.1 安装Paddle框架

  • CPU 版本
    pip install paddlepaddle
  • GPU 版本
    pip install paddlepaddle-gpu
(ppai) xf@VP01:~/ai/face$ pip install paddlepaddle-gpu
Looking in indexes: https://pypi.tuna.tsinghua.edu.cn/simple, https://pypi.ngc.nvidia.com
Collecting paddlepaddle-gpu
  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/48/6d/66e8d97902ce232d07efd0741325ad3b7e45d7b9ccf7140e7a637db8893d/paddlepaddle_gpu-2.4.2-cp37-cp37m-manylinux1_x86_64.whl (584.8 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 584.8/584.8 MB 6.8 MB/s eta 0:00:00
Requirement already satisfied: opt-einsum==3.3.0 in /home/xf/anaconda3/envs/ppai/lib/python3.7/site-packages (from paddlepaddle-gpu) (3.3.0)
Requirement already satisfied: requests>=2.20.0 in /home/xf/anaconda3/envs/ppai/lib/python3.7/site-packages (from paddlepaddle-gpu) (2.28.1)
Requirement already satisfied: six in /home/xf/anaconda3/envs/ppai/lib/python3.7/site-packages (from paddlepaddle-gpu) (1.16.0)
Requirement already satisfied: Pillow in /home/xf/anaconda3/envs/ppai/lib/python3.7/site-packages (from paddlepaddle-gpu) (9.4.0)
Requirement already satisfied: decorator in /home/xf/anaconda3/envs/ppai/lib/python3.7/site-packages (from paddlepaddle-gpu) (5.1.1)
Requirement already satisfied: protobuf<=3.20.0,>=3.1.0 in /home/xf/anaconda3/envs/ppai/lib/python3.7/site-packages (from paddlepaddle-gpu) (3.20.0)
Requirement already satisfied: astor in /home/xf/anaconda3/envs/ppai/lib/python3.7/site-packages (from paddlepaddle-gpu) (0.8.1)
Requirement already satisfied: numpy>=1.13 in /home/xf/anaconda3/envs/ppai/lib/python3.7/site-packages (from paddlepaddle-gpu) (1.21.5)
Requirement already satisfied: paddle-bfloat==0.1.7 in /home/xf/anaconda3/envs/ppai/lib/python3.7/site-packages (from paddlepaddle-gpu) (0.1.7)
Requirement already satisfied: charset-normalizer<3,>=2 in /home/xf/anaconda3/envs/ppai/lib/python3.7/site-packages (from requests>=2.20.0->paddlepaddle-gpu) (2.0.4)
Requirement already satisfied: urllib3<1.27,>=1.21.1 in /home/xf/anaconda3/envs/ppai/lib/python3.7/site-packages (from requests>=2.20.0->paddlepaddle-gpu) (1.26.14)
Requirement already satisfied: certifi>=2017.4.17 in /home/xf/anaconda3/envs/ppai/lib/python3.7/site-packages (from requests>=2.20.0->paddlepaddle-gpu) (2022.12.7)
Requirement already satisfied: idna<4,>=2.5 in /home/xf/anaconda3/envs/ppai/lib/python3.7/site-packages (from requests>=2.20.0->paddlepaddle-gpu) (3.4)
Installing collected packages: paddlepaddle-gpu
Successfully installed paddlepaddle-gpu-2.4.2

1.2.2 验证框架是否安装成功

python -c "import paddle; print(paddle.__version__)"

(ppai) xf@VP01:~/ai/face$ python -c "import paddle; print(paddle.__version__)"
2.4.2

python -c "import paddle;paddle.utils.run_check()"

(ppai) xf@VP01:~/ai/face$ python -c "import paddle;paddle.utils.run_check()"
Running verify PaddlePaddle program ...
W0428 03:41:27.101418  1187 gpu_resources.cc:61] Please NOTE: device: 0, GPU Compute Capability: 8.9, Driver API Version: 12.1, Runtime API Version: 10.2
W0428 03:41:27.775382  1187 gpu_resources.cc:91] device: 0, cuDNN Version: 7.4.
PaddlePaddle works well on 1 GPU.
PaddlePaddle works well on 1 GPUs.
PaddlePaddle is installed successfully! Let's start deep learning with PaddlePaddle now.

1.3 PaddleGAN 生成对抗网络

飞桨生成对抗网络开发套件–PaddleGAN，为开发者提供经典及前沿的生成对抗网络高性能实现，并支撑开发者快速构建、训练及部署生成对抗网络，以供学术、娱乐及产业应用。

1.3.1 安装ppgan

pip install --upgrade ppgan

1.3.2 安装其他依赖

pip install -r requirements.txt

1.3.3 下载示例代码、配置文件、中英文档

git clone https://github.com/PaddlePaddle/PaddleGAN
cd PaddleGAN

(ppai) xf@VP01:~/ai/face$ git clone https://github.com/PaddlePaddle/PaddleGAN
Cloning into 'PaddleGAN'...
remote: Enumerating objects: 5606, done.
remote: Counting objects: 100% (408/408), done.
remote: Compressing objects: 100% (222/222), done.
remote: Total 5606 (delta 238), reused 319 (delta 182), pack-reused 5198
Receiving objects: 100% (5606/5606), 163.63 MiB | 17.37 MiB/s, done.
Resolving deltas: 100% (3612/3612), done.
Updating files: 100% (565/565), done.
(ppai) xf@VP01:~/ai/face$ cd PaddleGAN
(ppai) xf@VP01:~/ai/face/PaddleGAN$ ll
total 56
drwxrwxrwx 1 xf xf   512 Apr 28 05:02 ./
drwxrwxrwx 1 xf xf   512 Apr 28 05:02 ../
drwxrwxrwx 1 xf xf   512 Apr 28 05:02 .git/
-rwxrwxrwx 1 xf xf  1242 Apr 28 05:02 .gitignore*
-rwxrwxrwx 1 xf xf  1040 Apr 28 05:02 .pre-commit-config.yaml*
-rwxrwxrwx 1 xf xf    47 Apr 28 05:02 .style.yapf*
-rwxrwxrwx 1 xf xf 11438 Apr 28 05:02 LICENSE*
-rwxrwxrwx 1 xf xf 13335 Apr 28 05:02 README.md*
-rwxrwxrwx 1 xf xf 14430 Apr 28 05:02 README_cn.md*
drwxrwxrwx 1 xf xf   512 Apr 28 05:02 applications/
drwxrwxrwx 1 xf xf   512 Apr 28 05:02 benchmark/
drwxrwxrwx 1 xf xf   512 Apr 28 05:02 configs/
drwxrwxrwx 1 xf xf   512 Apr 28 05:02 data/
drwxrwxrwx 1 xf xf   512 Apr 28 05:02 deploy/
drwxrwxrwx 1 xf xf   512 Apr 28 05:02 docs/
drwxrwxrwx 1 xf xf   512 Apr 28 05:02 education/
drwxrwxrwx 1 xf xf   512 Apr 28 05:02 paddlegan-wechaty-demo/
drwxrwxrwx 1 xf xf   512 Apr 28 05:02 ppgan/
-rwxrwxrwx 1 xf xf     0 Apr 28 05:02 python*
-rwxrwxrwx 1 xf xf   160 Apr 28 05:02 requirements.txt*
-rwxrwxrwx 1 xf xf  1900 Apr 28 05:02 setup.py*
drwxrwxrwx 1 xf xf   512 Apr 28 05:02 test/
drwxrwxrwx 1 xf xf   512 Apr 28 05:02 test_tipc/
drwxrwxrwx 1 xf xf   512 Apr 28 05:02 tools/

1.4 Paddle VisualDL 可视化训练工具

python -m pip install visualdl -i https://mirror.baidu.com/pypi/simple

1.5 错误

1.5.1 ImportError: libcudart.so.10.2: cannot open shared object file: No such file or directory

创建so软链接：

cd /usr/local/cuda/lib64
sudo ln -s libcudart.so.11.0 libcudart.so.10.2
sudo ldconfig

libcudart.so.11.0 这个版本根据本地实际安装为准

(ppai) xf@VP01:/usr/local/cuda/lib64$ ll
...
lrwxrwxrwx 1 root root        17 Sep 22  2022 libcudart.so -> libcudart.so.11.0
lrwxrwxrwx 1 root root        17 Apr 17 13:12 libcudart.so.10.0 -> libcudart.so.11.0
lrwxrwxrwx 1 root root        20 Sep 22  2022 libcudart.so.11.0 -> libcudart.so.11.8.89
-rw-r--r-- 1 root root    679264 Sep 22  2022 libcudart.so.11.8.89
...

1.5.2 ImportError: /usr/local/cuda/lib64/libcudart.so.10.2: version `libcudart.so.10.2’ not found

本机安装的CUDA是11.8版本，上小节1.5.1中创建的软链接是11.8，正常是CUDA向下兼容，但这里程序明确请求libcudart.so库版本要10.2，但我不想对CUDA11.8降级，其他软件需要用。我们只需要找个libcudart.so.10.2文件放到lib64，让Paddle能调用就行了。

  File "/home/xf/anaconda3/envs/ppai/lib/python3.7/site-packages/paddle/fluid/core.py", line 274, in <module>
    from . import libpaddle
ImportError: /usr/local/cuda/lib64/libcudart.so.10.2: version `libcudart.so.10.2' not found (required by /home/xf/anaconda3/envs/ppai/lib/python3.7/site-packages/paddle/fluid/libpaddle.so)

解决方法：

下载libcudart.so.10.2文件，复制到/usr/local/cuda/lib64/

2、表情迁移

Meta00147 数字人表情迁移

通过First Order Motion Model 实现表情迁移，我们使用PaddleGan工具包提供的API，可以快速完成表情迁移。

2.1 创建表情迁移shell

2.1.1 motion_driving表情迁移脚本

进入applications目录
cd applications
创建脚本 motion_driving.sh
sudo vim motion_driving.sh

# 表情迁移
#!bash

python -u tools/first-order-demo.py  \
     --driving_video ../docs/imgs/fom_dv.mp4 \
     --source_image ../docs/imgs/fom_source_image.png \
     --output ./output \
     --ratio 0.4 \
     --relative \
     --adapt_scale \
     --image_size 512 \
     --face_enhancement

# 音频迁移
python md_addwav.py

参数使用说明

  • driving_video
    驱动视频，视频中人物的表情动作作为待迁移的对象。
  • source_image
    原始图片，支持单人图片和多人图片，视频中人物的表情动作将迁移到该原始图片中的人物上。
  • relative
    指示程序中使用视频和图片中人物关键点的相对坐标还是绝对坐标，建议使用相对坐标，
    若使用绝对坐标，会导致迁移后人物扭曲变形。
  • adapt_scale
    根据关键点凸包自适应运动尺度。
  • ratio
    贴回驱动生成的人脸区域占原图的比例, 用户需要根据生成的效果调整该参数，
    尤其对于多人脸距离比较近的情况下需要调整改参数, 默认为0.4，调整范围是[0.4, 0.5]。
  • image_size
    图片人脸大小，默认为256，可设置为512.
  • face_enhancement
    添加人脸增强，不添加参数默认为不使用增强功能
  • multi_person
    当图片中有多张人脸请添加此参数，不加则默认为单人脸

2.1.2 md_addwav.py迁移源视频的声音

applications目录下创建md_addwav.py，用于表情迁移完成后，从源视频中提取声音合成到这个表情视频中。

from moviepy.editor import *

audio_clip = VideoFileClip("./assets/mayiyahei.mp4")
video_clip = VideoFileClip("./output/result.mp4")
audio_obj = audio_clip.audio

comp_clip = video_clip.set_audio(audio_obj)
comp_clip.write_videofile("./output/result_aac.mp4", audio_codec="aac")

2.2 运行shell

. motion_driving.sh

  • 注意别漏点了 .
    前面的点和空格是shell的执行命令，等价source，在当前shell进程环境中执行
  • ./
    基于当前窗口shell进程，创建一个子进程，运行完成返回当前shell

2.2.1 GPU 用时10秒

(ppai) xf@VP01:~/ai/face/PaddleGAN/applications$ . motion_driving.sh
[04/29 02:23:00] ppgan INFO: Found /home/xf/.cache/ppgan/vox-cpk-512.pdparams
W0429 02:23:00.647293  9165 gpu_resources.cc:61] Please NOTE: device: 0, GPU Compute Capability: 8.9, Driver API Version: 12.1, Runtime API Version: 11.7
W0429 02:23:00.651757  9165 gpu_resources.cc:91] device: 0, cuDNN Version: 8.8.
[04/29 02:23:01] ppgan INFO: Found /home/xf/.cache/ppgan/GPEN-512.pdparams
/home/xf/anaconda3/envs/ppai/lib/python3.7/site-packages/paddle/fluid/dygraph/layers.py:1652: UserWarning: Skip loading for generator.noises.noise_0. generator.noises.noise_0 is not found in the provided dict.
  warnings.warn(("Skip loading for {}. ".format(key) + str(err)))
/home/xf/anaconda3/envs/ppai/lib/python3.7/site-packages/paddle/fluid/dygraph/layers.py:1652: UserWarning: Skip loading for generator.noises.noise_1. generator.noises.noise_1 is not found in the provided dict.
  warnings.warn(("Skip loading for {}. ".format(key) + str(err)))
/home/xf/anaconda3/envs/ppai/lib/python3.7/site-packages/paddle/fluid/dygraph/layers.py:1652: UserWarning: Skip loading for generator.noises.noise_2. generator.noises.noise_2 is not found in the provided dict.
  warnings.warn(("Skip loading for {}. ".format(key) + str(err)))
/home/xf/anaconda3/envs/ppai/lib/python3.7/site-packages/paddle/fluid/dygraph/layers.py:1652: UserWarning: Skip loading for generator.noises.noise_3. generator.noises.noise_3 is not found in the provided dict.
  warnings.warn(("Skip loading for {}. ".format(key) + str(err)))
/home/xf/anaconda3/envs/ppai/lib/python3.7/site-packages/paddle/fluid/dygraph/layers.py:1652: UserWarning: Skip loading for generator.noises.noise_4. generator.noises.noise_4 is not found in the provided dict.
  warnings.warn(("Skip loading for {}. ".format(key) + str(err)))
/home/xf/anaconda3/envs/ppai/lib/python3.7/site-packages/paddle/fluid/dygraph/layers.py:1652: UserWarning: Skip loading for generator.noises.noise_5. generator.noises.noise_5 is not found in the provided dict.
  warnings.warn(("Skip loading for {}. ".format(key) + str(err)))
/home/xf/anaconda3/envs/ppai/lib/python3.7/site-packages/paddle/fluid/dygraph/layers.py:1652: UserWarning: Skip loading for generator.noises.noise_6. generator.noises.noise_6 is not found in the provided dict.
  warnings.warn(("Skip loading for {}. ".format(key) + str(err)))
/home/xf/anaconda3/envs/ppai/lib/python3.7/site-packages/paddle/fluid/dygraph/layers.py:1652: UserWarning: Skip loading for generator.noises.noise_7. generator.noises.noise_7 is not found in the provided dict.
  warnings.warn(("Skip loading for {}. ".format(key) + str(err)))
/home/xf/anaconda3/envs/ppai/lib/python3.7/site-packages/paddle/fluid/dygraph/layers.py:1652: UserWarning: Skip loading for generator.noises.noise_8. generator.noises.noise_8 is not found in the provided dict.
  warnings.warn(("Skip loading for {}. ".format(key) + str(err)))
/home/xf/anaconda3/envs/ppai/lib/python3.7/site-packages/paddle/fluid/dygraph/layers.py:1652: UserWarning: Skip loading for generator.noises.noise_9. generator.noises.noise_9 is not found in the provided dict.
  warnings.warn(("Skip loading for {}. ".format(key) + str(err)))
/home/xf/anaconda3/envs/ppai/lib/python3.7/site-packages/paddle/fluid/dygraph/layers.py:1652: UserWarning: Skip loading for generator.noises.noise_10. generator.noises.noise_10 is not found in the provided dict.
  warnings.warn(("Skip loading for {}. ".format(key) + str(err)))
/home/xf/anaconda3/envs/ppai/lib/python3.7/site-packages/paddle/fluid/dygraph/layers.py:1652: UserWarning: Skip loading for generator.noises.noise_11. generator.noises.noise_11 is not found in the provided dict.
  warnings.warn(("Skip loading for {}. ".format(key) + str(err)))
/home/xf/anaconda3/envs/ppai/lib/python3.7/site-packages/paddle/fluid/dygraph/layers.py:1652: UserWarning: Skip loading for generator.noises.noise_12. generator.noises.noise_12 is not found in the provided dict.
  warnings.warn(("Skip loading for {}. ".format(key) + str(err)))
/home/xf/anaconda3/envs/ppai/lib/python3.7/site-packages/paddle/fluid/dygraph/layers.py:1652: UserWarning: Skip loading for generator.noises.noise_13. generator.noises.noise_13 is not found in the provided dict.
  warnings.warn(("Skip loading for {}. ".format(key) + str(err)))
/home/xf/anaconda3/envs/ppai/lib/python3.7/site-packages/paddle/fluid/dygraph/layers.py:1652: UserWarning: Skip loading for generator.noises.noise_14. generator.noises.noise_14 is not found in the provided dict.
  warnings.warn(("Skip loading for {}. ".format(key) + str(err)))
1 persons have been detected
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 222/222 [00:10<00:00, 21.79it/s]
IMAGEIO FFMPEG_WRITER WARNING: input image is not divisible by macro_block_size=16, resizing from (1224, 1225) to (1232, 1232) to ensure video compatibility with most codecs and players. To prevent resizing, make your input image divisible by the macro_block_size or set the macro_block_size to 1 (risking incompatibility).
[swscaler @ 0x66edcc0] Warning: data is not aligned! This can lead to a speed loss
Moviepy - Building video ./output/result_aac.mp4.
MoviePy - Writing audio in result_aacTEMP_MPY_wvf_snd.mp4
MoviePy - Done.
Moviepy - Writing video ./output/result_aac.mp4

Moviepy - Done !
Moviepy - video ready ./output/result_aac.mp4
  • GPU 硬件信息
(base) xf@VP01:~$ nvidia-smi
Sat Apr 29 02:56:43 2023
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 530.41.03              Driver Version: 531.41       CUDA Version: 12.1     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                  Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf            Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  NVIDIA GeForce RTX 4090         On | 00000000:01:00.0  On |                  Off |
|  0%   49C    P8               28W / 450W|   1028MiB / 24564MiB |      0%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+

+---------------------------------------------------------------------------------------+
| Processes:                                                                            |
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
|        ID   ID                                                             Usage      |
|=======================================================================================|
|    0   N/A  N/A        13      G   /Xwayland                                 N/A      |
+---------------------------------------------------------------------------------------+

watch -n 1 nvidia-smi 可动态显示每秒显卡温度、显存用量等信息

2.2.2 CPU 用时44分35秒

(ppai) xf@VP01:~/ai/face/PaddleGAN/applications$ . motion_driving.sh
[04/29 02:30:51] ppgan INFO: Found /home/xf/.cache/ppgan/vox-cpk-512.pdparams
[04/29 02:30:52] ppgan INFO: Found /home/xf/.cache/ppgan/GPEN-512.pdparams
/home/xf/anaconda3/envs/ppai/lib/python3.7/site-packages/paddle/fluid/dygraph/layers.py:1652: UserWarning: Skip loading for generator.noises.noise_0. generator.noises.noise_0 is not found in the provided dict.
  warnings.warn(("Skip loading for {}. ".format(key) + str(err)))
/home/xf/anaconda3/envs/ppai/lib/python3.7/site-packages/paddle/fluid/dygraph/layers.py:1652: UserWarning: Skip loading for generator.noises.noise_1. generator.noises.noise_1 is not found in the provided dict.
  warnings.warn(("Skip loading for {}. ".format(key) + str(err)))
/home/xf/anaconda3/envs/ppai/lib/python3.7/site-packages/paddle/fluid/dygraph/layers.py:1652: UserWarning: Skip loading for generator.noises.noise_2. generator.noises.noise_2 is not found in the provided dict.
  warnings.warn(("Skip loading for {}. ".format(key) + str(err)))
/home/xf/anaconda3/envs/ppai/lib/python3.7/site-packages/paddle/fluid/dygraph/layers.py:1652: UserWarning: Skip loading for generator.noises.noise_3. generator.noises.noise_3 is not found in the provided dict.
  warnings.warn(("Skip loading for {}. ".format(key) + str(err)))
/home/xf/anaconda3/envs/ppai/lib/python3.7/site-packages/paddle/fluid/dygraph/layers.py:1652: UserWarning: Skip loading for generator.noises.noise_4. generator.noises.noise_4 is not found in the provided dict.
  warnings.warn(("Skip loading for {}. ".format(key) + str(err)))
/home/xf/anaconda3/envs/ppai/lib/python3.7/site-packages/paddle/fluid/dygraph/layers.py:1652: UserWarning: Skip loading for generator.noises.noise_5. generator.noises.noise_5 is not found in the provided dict.
  warnings.warn(("Skip loading for {}. ".format(key) + str(err)))
/home/xf/anaconda3/envs/ppai/lib/python3.7/site-packages/paddle/fluid/dygraph/layers.py:1652: UserWarning: Skip loading for generator.noises.noise_6. generator.noises.noise_6 is not found in the provided dict.
  warnings.warn(("Skip loading for {}. ".format(key) + str(err)))
/home/xf/anaconda3/envs/ppai/lib/python3.7/site-packages/paddle/fluid/dygraph/layers.py:1652: UserWarning: Skip loading for generator.noises.noise_7. generator.noises.noise_7 is not found in the provided dict.
  warnings.warn(("Skip loading for {}. ".format(key) + str(err)))
/home/xf/anaconda3/envs/ppai/lib/python3.7/site-packages/paddle/fluid/dygraph/layers.py:1652: UserWarning: Skip loading for generator.noises.noise_8. generator.noises.noise_8 is not found in the provided dict.
  warnings.warn(("Skip loading for {}. ".format(key) + str(err)))
/home/xf/anaconda3/envs/ppai/lib/python3.7/site-packages/paddle/fluid/dygraph/layers.py:1652: UserWarning: Skip loading for generator.noises.noise_9. generator.noises.noise_9 is not found in the provided dict.
  warnings.warn(("Skip loading for {}. ".format(key) + str(err)))
/home/xf/anaconda3/envs/ppai/lib/python3.7/site-packages/paddle/fluid/dygraph/layers.py:1652: UserWarning: Skip loading for generator.noises.noise_10. generator.noises.noise_10 is not found in the provided dict.
  warnings.warn(("Skip loading for {}. ".format(key) + str(err)))
/home/xf/anaconda3/envs/ppai/lib/python3.7/site-packages/paddle/fluid/dygraph/layers.py:1652: UserWarning: Skip loading for generator.noises.noise_11. generator.noises.noise_11 is not found in the provided dict.
  warnings.warn(("Skip loading for {}. ".format(key) + str(err)))
/home/xf/anaconda3/envs/ppai/lib/python3.7/site-packages/paddle/fluid/dygraph/layers.py:1652: UserWarning: Skip loading for generator.noises.noise_12. generator.noises.noise_12 is not found in the provided dict.
  warnings.warn(("Skip loading for {}. ".format(key) + str(err)))
/home/xf/anaconda3/envs/ppai/lib/python3.7/site-packages/paddle/fluid/dygraph/layers.py:1652: UserWarning: Skip loading for generator.noises.noise_13. generator.noises.noise_13 is not found in the provided dict.
  warnings.warn(("Skip loading for {}. ".format(key) + str(err)))
/home/xf/anaconda3/envs/ppai/lib/python3.7/site-packages/paddle/fluid/dygraph/layers.py:1652: UserWarning: Skip loading for generator.noises.noise_14. generator.noises.noise_14 is not found in the provided dict.
  warnings.warn(("Skip loading for {}. ".format(key) + str(err)))
1 persons have been detected
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 222/222 [44:35<00:00, 12.05s/it]
IMAGEIO FFMPEG_WRITER WARNING: input image is not divisible by macro_block_size=16, resizing from (1224, 1225) to (1232, 1232) to ensure video compatibility with most codecs and players. To prevent resizing, make your input image divisible by the macro_block_size or set the macro_block_size to 1 (risking incompatibility).
[swscaler @ 0x6126cc0] Warning: data is not aligned! This can lead to a speed loss
Moviepy - Building video ./output/result_aac.mp4.
MoviePy - Writing audio in result_aacTEMP_MPY_wvf_snd.mp4
MoviePy - Done.
Moviepy - Writing video ./output/result_aac.mp4

Moviepy - Done !
Moviepy - video ready ./output/result_aac.mp4
(ppai) xf@VP01:~/ai/face/PaddleGAN/applications$
  • CPU 硬件信息
(base) xf@VP01:~$ lscpu
Architecture:            x86_64
  CPU op-mode(s):        32-bit, 64-bit
  Address sizes:         39 bits physical, 48 bits virtual
  Byte Order:            Little Endian
CPU(s):                  24
  On-line CPU(s) list:   0-23
Vendor ID:               GenuineIntel
  Model name:            12th Gen Intel(R) Core(TM) i9-12900KF
    CPU family:          6
    Model:               151
    Thread(s) per core:  2
    Core(s) per socket:  12
    Socket(s):           1
    Stepping:            2
    BogoMIPS:            6374.40
    Flags:               fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp
                          lm constant_tsc rep_good nopl xtopology tsc_reliable nonstop_tsc cpuid pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 movbe pop
                         cnt aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch invpcid_single ssbd ibrs ibpb stibp ibrs_enhanced fsgsbase bmi
                         1 avx2 smep bmi2 erms invpcid rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 xsaves umip gfni vaes vpclmulqdq rdp
                         id fsrm flush_l1d arch_capabilities
Virtualization features:
  Hypervisor vendor:     Microsoft
  Virtualization type:   full
Caches (sum of all):
  L1d:                   576 KiB (12 instances)
  L1i:                   384 KiB (12 instances)
  L2:                    15 MiB (12 instances)
  L3:                    30 MiB (1 instance)
Vulnerabilities:
  Itlb multihit:         Not affected
  L1tf:                  Not affected
  Mds:                   Not affected
  Meltdown:              Not affected
  Mmio stale data:       Not affected
  Retbleed:              Mitigation; Enhanced IBRS
  Spec store bypass:     Mitigation; Speculative Store Bypass disabled via prctl and seccomp
  Spectre v1:            Mitigation; usercopy/swapgs barriers and __user pointer sanitization
  Spectre v2:            Mitigation; Enhanced IBRS, IBPB conditional, RSB filling, PBRSB-eIBRS SW sequence
  Srbds:                 Not affected
  Tsx async abort:       Not affected

3、数据集

3.1 VoxCeleb Dataset

参考这里 https://github.com/AliaksandrSiarohin/video-preprocessing

3.1.1 创建数据集目录

mkdir -p data/first_order
cd data/first_order

3.1.2 下载数据集清单与脚本

git clone https://github.com/AliaksandrSiarohin/video-preprocessing.git

(ppai) xf@VP01:~/ai/face/paddlegan/data/first_order$ git clone https://github.com/AliaksandrSiarohin/video-preprocessing.git
Cloning into 'video-preprocessing'...
remote: Enumerating objects: 171, done.
remote: Counting objects: 100% (65/65), done.
remote: Compressing objects: 100% (8/8), done.
remote: Total 171 (delta 58), reused 57 (delta 57), pack-reused 106
Receiving objects: 100% (171/171), 608.42 KiB | 1.61 MiB/s, done.
Resolving deltas: 100% (103/103), done.

下载必要的youtube-dl
wget https://yt-dl.org/downloads/latest/youtube-dl -O youtube-dl
chmod a+rx youtube-dl

(ppai) xf@VP01:~/ai/face/paddlegan/data/first_order/video-preprocessing$ wget https://yt-dl.org/downloads/latest/youtube-dl -O youtube-dl
--2023-04-27 07:40:26--  https://yt-dl.org/downloads/latest/youtube-dl
Resolving yt-dl.org (yt-dl.org)... 95.143.172.170, 185.26.156.71, 95.143.172.73, ...
Connecting to yt-dl.org (yt-dl.org)|95.143.172.170|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://yt-dl.org/downloads/2021.12.17/youtube-dl [following]
--2023-04-27 07:40:27--  https://yt-dl.org/downloads/2021.12.17/youtube-dl
Reusing existing connection to yt-dl.org:443.
HTTP request sent, awaiting response... 302 Found
Location: https://github.com/ytdl-org/youtube-dl/releases/download/2021.12.17/youtube-dl [following]
--2023-04-27 07:40:27--  https://github.com/ytdl-org/youtube-dl/releases/download/2021.12.17/youtube-dl
Resolving github.com (github.com)... 20.205.243.166, 198.51.44.8, 198.51.45.8, ...
Connecting to github.com (github.com)|20.205.243.166|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://objects.githubusercontent.com/github-production-release-asset-2e65be/1039520/65476747-919e-4c56-a57f-ded64bed461f?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20230426%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20230426T234030Z&X-Amz-Expires=300&X-Amz-Signature=cb2de721f9f09993715b6f2dfb28c43adedc5c1bbe71c9e0bd24fbb7016b1f85&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=1039520&response-content-disposition=attachment%3B%20filename%3Dyoutube-dl&response-content-type=application%2Foctet-stream [following]
--2023-04-27 07:40:28--  https://objects.githubusercontent.com/github-production-release-asset-2e65be/1039520/65476747-919e-4c56-a57f-ded64bed461f?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20230426%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20230426T234030Z&X-Amz-Expires=300&X-Amz-Signature=cb2de721f9f09993715b6f2dfb28c43adedc5c1bbe71c9e0bd24fbb7016b1f85&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=1039520&response-content-disposition=attachment%3B%20filename%3Dyoutube-dl&response-content-type=application%2Foctet-stream
Resolving objects.githubusercontent.com (objects.githubusercontent.com)... 185.199.110.133, 185.199.108.133, 185.199.111.133, ...
Connecting to objects.githubusercontent.com (objects.githubusercontent.com)|185.199.110.133|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 1837828 (1.8M) [application/octet-stream]
Saving to: ‘youtube-dl’

youtube-dl                             100%[==========================================================================>]   1.75M  27.7KB/s    in 56s

2023-04-27 07:41:24 (32.2 KB/s) - ‘youtube-dl’ saved [1837828/1837828]

3.1.3 优化数据集下载脚本

cd video-preprocessing
sudo vim load_vox.sh

python load_videos.py --metadata vox-metadata.csv --format .mp4 --out_folder ../Voxceleb --workers 8

(ppai) xf@VP01:~/ai/face/paddlegan/data/first_order/video-preprocessing$ ll
total 1501
drwxrwxrwx 1 xf xf     512 Apr 27 07:27 ./
drwxrwxrwx 1 xf xf     512 Apr 27 07:28 ../
drwxrwxrwx 1 xf xf     512 Apr 27 07:26 .git/
-rwxrwxrwx 1 xf xf    4696 Apr 27 07:26 README.md*
-rwxrwxrwx 1 xf xf    3506 Apr 27 07:26 crop_nemo.py*
-rwxrwxrwx 1 xf xf    9893 Apr 27 07:26 crop_taichi.py*
-rwxrwxrwx 1 xf xf   13918 Apr 27 07:26 crop_vox.py*
-rwxrwxrwx 1 xf xf    4414 Apr 27 07:26 load_videos.py*
-rwxrwxrwx 1 xf xf     101 Apr 27 07:26 load_vox.sh*
-rwxrwxrwx 1 xf xf   15180 Apr 27 07:26 maskrcnn.py*
-rwxrwxrwx 1 xf xf     562 Apr 27 07:26 requirements.txt*
-rwxrwxrwx 1 xf xf  196418 Apr 27 07:26 taichi-metadata.csv*
-rwxrwxrwx 1 xf xf    4300 Apr 27 07:26 util.py*
-rwxrwxrwx 1 xf xf 1265155 Apr 27 07:26 vox-metadata.csv*
(ppai) xf@VP01:~/ai/face/paddlegan/data/first_order/video-preprocessing$

3.1.4 下载数据集VoxCeleb

First Order Motion Model for Image Animation，提供每个YouTube视频的URL和发言的时间戳。 提供的帧数假定视频以25fps保存。
VoxCeleb是一个视听数据集，由从上传到YouTube的采访视频中提取的人类语音短片组成。
VoxCeleb1包含超过100,000个针对1,251个名人的话语，这些话语是从上传到YouTube的视频中提取的。
发音人数量
1251
视频数量
21,245
音频数量
145265

3.1.4.1 用脚本从Youtube抓数据 失败

运行脚本下载视频，有 2 种格式可用于存储视频:
一种是 .mp4，另一种是带有 .png 图像的文件夹。
虽然 .png 图像占用的空间要多得多，但这种格式是无损的，并且在训练时具有更好的 i/o 性能。

使用原始下载python脚本
python load_videos.py --metadata vox-metadata.csv --format .mp4 --out_folder ../Voxceleb --workers 8
或
使用优化下载的shell脚本
./load_vox.sh

xf@VP01:~/ai/face/paddlegan/data/first_order/video-preprocessing$ ./load_vox.sh
0it [00:00, ?it/s]Can not load video 2eMX5X0WzkM, broken link
1it [00:07,  7.41s/it]Can not load video J9lHsKG98U8, broken link
Can not load video mt5E9pPyxls, broken link
Can not load video 1p1Ds52eevo, broken link
Can not load video oL2J19duVLE, broken link
Can not load video e93OOf47Qak, broken link
Can not load video DW2umNQrQU0, broken link
7it [00:07,  5.19s/it]Can not load video Ywy9JEBbyuI, broken link
Can not load video o2z3Lq16KTw, broken link

无法连接到youtube数据，只能换方法了 失败

3.1.4.2 从官网下载 失败

  • VoxCeleb 视听数据集官网 点这里 https://www.robots.ox.ac.uk/~vgg/data/voxceleb/
    已经不提供下载 失败
    请添加图片描述

  • 申请谷歌授权下载 这里 https://docs.google.com/forms/d/e/1FAIpQLSdQhpq2Be2CktaPhuadUMU7ZDJoQuRlFlzNO45xO-drWQ0AXA/viewform?fbzx=7440236747203254000
    请添加图片描述申请成功，获得用户名和密码
    请添加图片描述
    wget -c --http-user=**** --http-passwd=*** http://www.robots.ox.ac.uk/~vgg/data/voxceleb/vox1a/vox1_dev_mp4_partaa > log_vox_download 2>&1 &
    发现还是通过官网，官网的数据已经下架 失败

3.1.4.3 从kaist下载 放弃

在这里https://mm.kaist.ac.kr/datasets/voxceleb/申请 下载voxceleb音视频文件
由于需要机构邮箱 放弃

3.1.4.4 从网盘分享下载 成功

感谢网友的网盘分享 参考这里

实测日期：2023-4-27

3.2 Fashion Dataset

参考这里 https://vision.cs.ubc.ca/datasets/fashion/

3.2.1 创建数据集目录

mkdir -p data/first_order/fashion
cd data/first_order/fashion

(ppai) xf@VP01:~/ai/face/paddlegan$ mkdir -p data/first_order/fashion
(ppai) xf@VP01:~/ai/face/paddlegan$ cd data/first_order/fashion
(ppai) xf@VP01:~/ai/face/paddlegan/data/first_order/fashion$ ll
total 0
drwxrwxrwx 1 xf xf 512 Apr 27 06:20 ./
drwxrwxrwx 1 xf xf 512 Apr 27 06:20 ../

3.2.2 下载数据清单和脚本

wget https://vision.cs.ubc.ca/datasets/fashion/resources/fashion_dataset/fashion_train.txt
wget https://vision.cs.ubc.ca/datasets/fashion/resources/fashion_dataset/fashion_test.txt
wget https://vision.cs.ubc.ca/datasets/fashion/resources/fashion_dataset/data_crowler.py

(ppai) xf@VP01:~/ai/face/paddlegan/data/first_order/fashion$ wget https://vision.cs.ubc.ca/datasets/fashion/resources/fashion_dataset/fashion_train.txt
--2023-04-27 06:21:14--  https://vision.cs.ubc.ca/datasets/fashion/resources/fashion_dataset/fashion_train.txt
Resolving vision.cs.ubc.ca (vision.cs.ubc.ca)... 142.103.89.140, 198.162.35.1, 142.103.6.6, ...
Connecting to vision.cs.ubc.ca (vision.cs.ubc.ca)|142.103.89.140|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 26000 (25K) [text/plain]
Saving to: ‘fashion_train.txt’

fashion_train.txt                      100%[==========================================================================>]  25.39K  --.-KB/s    in 0.003s

2023-04-27 06:21:15 (8.22 MB/s) - ‘fashion_train.txt’ saved [26000/26000]

(ppai) xf@VP01:~/ai/face/paddlegan/data/first_order/fashion$ wget https://vision.cs.ubc.ca/datasets/fashion/resources/fashion_dataset/fashion_test.txt
--2023-04-27 06:21:22--  https://vision.cs.ubc.ca/datasets/fashion/resources/fashion_dataset/fashion_test.txt
Resolving vision.cs.ubc.ca (vision.cs.ubc.ca)... 142.103.89.140, 198.162.35.1, 142.103.6.6, ...
Connecting to vision.cs.ubc.ca (vision.cs.ubc.ca)|142.103.89.140|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 5200 (5.1K) [text/plain]
Saving to: ‘fashion_test.txt’

fashion_test.txt                       100%[==========================================================================>]   5.08K  --.-KB/s    in 0s

2023-04-27 06:21:23 (596 MB/s) - ‘fashion_test.txt’ saved [5200/5200]

(ppai) xf@VP01:~/ai/face/paddlegan/data/first_order/fashion$ ll
total 36
drwxrwxrwx 1 xf xf   512 Apr 27 06:21 ./
drwxrwxrwx 1 xf xf   512 Apr 27 06:20 ../
-rwxrwxrwx 1 xf xf  5200 Oct 13  2022 fashion_test.txt*
-rwxrwxrwx 1 xf xf 26000 Oct 13  2022 fashion_train.txt*
(ppai) xf@VP01:~/ai/face/paddlegan/data/first_order/fashion$ wget https://vision.cs.ubc.ca/datasets/fashion/resources/fashion_dataset/data_crowler.py
--2023-04-27 06:27:34--  https://vision.cs.ubc.ca/datasets/fashion/resources/fashion_dataset/data_crowler.py
Resolving vision.cs.ubc.ca (vision.cs.ubc.ca)... 142.103.89.140, 198.162.35.1, 142.103.6.6, ...
Connecting to vision.cs.ubc.ca (vision.cs.ubc.ca)|142.103.89.140|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 580 [application/octet-stream]
Saving to: ‘data_crowler.py’

data_crowler.py                        100%[==========================================================================>]     580  --.-KB/s    in 0s

2023-04-27 06:27:35 (140 MB/s) - ‘data_crowler.py’ saved [580/580]

3.2.3 优化数据集下载脚本

sudo vim fashion_dataset.py

### download fashion dataset for first order

import requests
import os

f_train = open('fashion_train.txt', "r")
train_files = f_train.readlines()
if not os.path.exists('train'):
	os.mkdir('train')
	i = len(train_files)
	for video_url in train_files:
		file_name = video_url[:-1].split("/")[-1]
		print('download...'+ str(i) + ': ' + file_name)
		i = i-1
		r = requests.get(video_url[:-1])
		with open("train/"+file_name,'wb') as f:
			f.write(r.content)

f_test = open('fashion_test.txt', "r")
test_files = f_test.readlines()
if not os.path.exists('test'):
	os.mkdir('test')
	i = len(test_files)
	for video_url in test_files:
		file_name = video_url[:-1].split("/")[-1]
		print('download...'+ str(i) + ': ' + file_name)
		i = i-1
		r = requests.get(video_url[:-1])
		with open("test/"+file_name,'wb') as f:
			f.write(r.content)

3.2.3 下载数据集fashion

使用原始下载脚本，没有进度显示
python data_crowler.py
或
使用优化下载脚本，实时了解下载进度
python fashion_dataset.py

(ppai) xf@VP01:~/ai/face/paddlegan/data/first_order/fashion$ python fashion_dataset.py
download...500: 91D23ZVV6NS.mp4
download...499: 91hECpvrRSS.mp4
download...498: A1P+0ZhMAiS.mp4
download...497: 91WIsDoNP2S.mp4
download...496: A1PY+v5m7oS.mp4
download...495: 91t1iUyA-DS.mp4
download...494: 91awragicLS.mp4
download...493: A1Y8HsouVnS.mp4
download...492: 91O0DKws+gS.mp4
download...491: A1aF4I7s-hS.mp4
download...490: 91i7wHVECAS.mp4
download...489: 91qaGfTESXS.mp4
download...488: 91t-lS-8VkS.mp4
download...487: 91FQY9XTyQS.mp4
download...486: A1X3koQIOoS.mp4
download...485: A11fCqkXWUS.mp4
download...484: 91elFllk8oS.mp4
download...483: 91FxsyKa6cS.mp4
download...482: A1QvutYxzXS.mp4
download...481: 91VR+4O2H6S.mp4
download...480: A1E-bnKT5iS.mp4
download...479: 91UveTFmA3S.mp4
download...478: A1iFJaDDBKS.mp4
download...477: A12qCU8nimS.mp4
download...476: 91d3Lx1uCYS.mp4
download...475: 91KbTM-Qs5S.mp4
download...474: A1PyMMv4i2S.mp4
download...473: 91rr7Fj3wmS.mp4
download...472: 91LVKVKed0S.mp4
download...471: 91-kqBbzDIS.mp4
download...470: A1f94LlYyIS.mp4
download...469: 91IiZsMxf9S.mp4
download...468: 91TDp5Z08AS.mp4
download...467: 91dVyEFeFXS.mp4
...

4、训练模型

由于没有获得完整的数据集，训练章节只按说明摘录备用，未实测，后续有机会训练后补上详情

4.1 GPU单卡训练

选择单个GPU
export CUDA_VISIBLE_DEVICES=0
需要指定训练的数据集配置
python tools/main.py --config-file configs/dataset_name.yaml

export CUDA_VISIBLE_DEVICES=0
python tools/main.py --config-file configs/firstorder_fashion.yaml

4.2 GPU多卡训练

指定多个GPU
export CUDA_VISIBLE_DEVICES=0,1,2,3
需要指定训练的数据集配置
python -m paddle.distributed.launch tools/main.py --config-file configs/dataset_name.yaml

export CUDA_VISIBLE_DEVICES=0,1,2,3
python -m paddle.distributed.launch \
    tools/main.py \
    --config-file configs/firstorder_fashion.yaml \

4.3 导出FOM模型

使用tools/fom_export.py脚本导出模型已经部署时使用的配置文件，
配置文件名字为firstorder_vox_mobile_256.yml

python tools/export_model.py \
    --config-file configs/firstorder_vox_mobile_256.yaml \
    --load /root/.cache/ppgan/vox_mobile.pdparams \
    --inputs_size "1,3,256,256;1,3,256,256;1,10,2;1,10,2,2" \
    --export_model output_inference/

预测模型会导出到output_inference/fom_dy2st/目录下

model.pdiparams
model.pdiparams.info
model.pdmodel

《完》

您可能感兴趣的与本文相关的镜像

PyTorch 2.5

PyTorch 2.5

PyTorch
Cuda

PyTorch 是一个开源的 Python 机器学习库，基于 Torch 库，底层由 C++ 实现，应用于人工智能领域，如计算机视觉和自然语言处理

评论
成就一亿技术人!
拼手气红包6.0元
还能输入1000个字符
 
红包 添加红包
表情包 插入表情
表情包 代码片
 条评论被折叠 查看
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值