论文阅读笔记：Denoising Diffusion Implicit Models （1）

0、快速访问

论文阅读笔记：Denoising Diffusion Implicit Models （1）
论文阅读笔记：Denoising Diffusion Implicit Models （2）
论文阅读笔记：Denoising Diffusion Implicit Models （3）
论文阅读笔记：Denoising Diffusion Implicit Models （4）
论文阅读笔记：Denoising Diffusion Implicit Models （5）

1、参考来源

论文《Denoising Diffusion Implicit Models》
来源：ICLR2021
https://iclr.cc/virtual/2021/poster/2804
论文链接：https://arxiv.org/abs/2010.02502
代码链接：https://github.com/ermongroup/ddim

2、论文DDIM中符号不同给一些公式带来的一些变化

在论文DDPM《Denoising Diffusion Implicit Models》当中，前向传播过程的$q(x_{t-1}|x_t,x_0)\sim N(x_{t-1};{\tilde{\mu }}_t(x_t,x_0),{\sigma }_t)$。并且${\tilde{\mu }}_t(x_t,x_0)和{\sigma }_t$分别如公式（1）所示。
$$\begin{array}{c}\begin{array}{rl}{\sigma }_t& =\sqrt{\frac{{\beta }_t\cdot (1-\overline{{\alpha }_{t-1}})}{(1-\overline{{\alpha }_t})}}\\ {\tilde{\mu }}_t(x_t,x_0)& =\frac{\sqrt{{\alpha }_t}\cdot (1-\overline{{\alpha }_{t-1}})}{1-\overline{{\alpha }_t}}\cdot x_t+\frac{{\beta }_t\cdot \sqrt{\overline{{\alpha }_{t-1}}}}{1-\overline{{\alpha }_t}}\cdot x_0\end{array}\end{array}$$
在DDIM《Denoising Diffusion Implicit Models》中对符号进行了重新定义。具体来说使用${\alpha }_t$替换掉了${\bar{\alpha }}_t$，而在DDPM当中
$$\begin{array}{c}\begin{array}{r}{\bar{\alpha }}_t=\prod _0^t{\alpha }_i\end{array}\end{array}$$
因此，在DDIM中会发生一些变化，例如${\beta }_t$的改变如公式（3）所示。
$$\begin{array}{c}\begin{array}{rl}{\beta }_t& =1-{\alpha }_t(DDPM)\\ & =1-\frac{{\alpha }_t}{{\alpha }_{t-1}}(DDIM)\end{array}\end{array}$$
前向加噪过程中的$q(x_{t-1}|x_t,x_0)$分布的方差和均值分别如公式（4）和（5）所示。
$$\begin{array}{c}\begin{array}{rl}{\sigma }_t^2& =\frac{1-{\bar{\alpha }}_{t-1}}{1-\overline{{\alpha }_t}}\cdot {\beta }_t(DDPM)\\ & =\frac{1-{\alpha }_{t-1}}{1-{\alpha }_t}\cdot (1-\frac{{\alpha }_t}{{\alpha }_{t-1}})(DDIM)\end{array}\end{array}$$
$$\begin{array}{c}\begin{array}{rl}{\tilde{\mu }}_t(x_t,x_0)& =\frac{\sqrt{{\alpha }_t}\cdot (1-{\bar{\alpha }}_{t-1})}{1-\overline{{\alpha }_t}}\cdot x_t+\frac{{\beta }_t\cdot \sqrt{{\bar{\alpha }}_{t-1}}}{1-\overline{{\alpha }_t}}\cdot x_0（DDPM）\\ & =\frac{\sqrt{{\alpha }_t}\cdot (1-{\alpha }_{t-1})}{\sqrt{{\alpha }_{t-1}}\cdot (1-{\alpha }_t)}\cdot x_t+(1-\frac{{\alpha }_t}{{\alpha }_{t-1}})\cdot \frac{\sqrt{{\alpha }_{t-1}}}{1-{\alpha }_t}\cdot x_0(DDIM)\\ & =\sqrt{\frac{{\alpha }_t\cdot (1-{\alpha }_{t-1}{)}^2}{{\alpha }_{t-1}\cdot (1-{\alpha }_t{)}^2}}\cdot x_t+\frac{{\alpha }_{t-1}-{\alpha }_t}{{\alpha }_{t-1}}\cdot \frac{\sqrt{{\alpha }_{t-1}}}{1-{\alpha }_t}\cdot x_0\\ & =\sqrt{\frac{1-{\alpha }_{t-1}}{1-{\alpha }_t}\cdot \frac{{\alpha }_t-{\alpha }_t\cdot {\alpha }_{t-1}}{{\alpha }_{t-1}-{\alpha }_{t-1}\cdot {\alpha }_t}}\cdot x_t+\frac{{\alpha }_{t-1}-{\alpha }_t}{\sqrt{{\alpha }_{t-1}}\cdot (1-{\alpha }_t)}\cdot x_0\\ & =\sqrt{\frac{1-{\alpha }_{t-1}}{1-{\alpha }_t}\cdot \frac{{\alpha }_t+{\alpha }_{t-1}-{\alpha }_{t-1}-{\alpha }_t\cdot {\alpha }_{t-1}}{{\alpha }_{t-1}-{\alpha }_{t-1}\cdot {\alpha }_t}}\cdot x_t+\frac{{\alpha }_{t-1}-{\alpha }_t\cdot {\alpha }_{t-1}+{\alpha }_t\cdot {\alpha }_{t-1}-{\alpha }_t}{\sqrt{{\alpha }_{t-1}}\cdot (1-{\alpha }_t)}\cdot x_0\\ & =\sqrt{\frac{1-{\alpha }_{t-1}}{1-{\alpha }_t}\cdot (1+\frac{{\alpha }_t-{\alpha }_{t-1}}{{\alpha }_{t-1}-{\alpha }_{t-1}\cdot {\alpha }_t})}\cdot x_t+\frac{{\alpha }_{t-1}\cdot (1-{\alpha }_t)-{\alpha }_t\cdot (1-{\alpha }_{t-1})}{\sqrt{{\alpha }_{t-1}}\cdot (1-{\alpha }_t)}\cdot x_0\\ & =\sqrt{\frac{1-{\alpha }_{t-1}}{1-{\alpha }_t}\cdot (1-\frac{{\alpha }_{t-1}-{\alpha }_t}{{\alpha }_{t-1}-{\alpha }_{t-1}\cdot {\alpha }_t})}\cdot x_t+[\sqrt{{\alpha }_{t-1}}-\frac{{\alpha }_t\cdot (1-{\alpha }_{t-1})}{\sqrt{{\alpha }_{t-1}}\cdot (1-{\alpha }_t)}]\cdot x_0\\ & =\sqrt{\frac{1}{1-{\alpha }_t}\cdot (1-{\alpha }_{t-1}-\frac{({\alpha }_{t-1}-{\alpha }_t)\cdot (1-{\alpha }_{t-1})}{{\alpha }_{t-1}-{\alpha }_{t-1}\cdot {\alpha }_t})}\cdot x_t+[\sqrt{{\alpha }_{t-1}}-\frac{\sqrt{{\alpha }_t^2\cdot (1-{\alpha }_{t-1}{)}^2}}{\sqrt{{\alpha }_{t-1}}\cdot (1-{\alpha }_t)}]\cdot x_0\\ & =\sqrt{\frac{1}{1-{\alpha }_t}\cdot (1-{\alpha }_{t-1}-\underset{={\sigma }_t^2}{\underbrace{\frac{({\alpha }_{t-1}-{\alpha }_t)\cdot (1-{\alpha }_{t-1})}{{\alpha }_{t-1}\cdot (1-{\alpha }_t)}}})}\cdot x_t+[\sqrt{{\alpha }_{t-1}}-\frac{\sqrt{{\alpha }_t\cdot (1-{\alpha }_{t-1})\cdot ({\alpha }_t-{\alpha }_t\cdot {\alpha }_{t-1})}}{\sqrt{{\alpha }_{t-1}}\cdot (1-{\alpha }_t)}]\cdot x_0\\ & =\sqrt{\frac{1}{1-{\alpha }_t}\cdot (1-{\alpha }_{t-1}-{\sigma }_t^2)}\cdot x_t+[\sqrt{{\alpha }_{t-1}}-\frac{\sqrt{{\alpha }_t\cdot (1-{\alpha }_{t-1})\cdot ({\alpha }_t+{\alpha }_{t-1}-{\alpha }_{t-1}-{\alpha }_t\cdot {\alpha }_{t-1})}}{\sqrt{{\alpha }_{t-1}\cdot (1-{\alpha }_t)}\cdot (\sqrt{1-{\alpha }_t})}]\cdot x_0\\ & =\sqrt{\frac{1-{\alpha }_{t-1}-{\sigma }_t^2}{1-{\alpha }_t}}\cdot x_t+[\sqrt{{\alpha }_{t-1}}-\frac{\sqrt{1-{\alpha }_{t-1}}}{\sqrt{1-{\alpha }_t}}\cdot \frac{\sqrt{{\alpha }_t\cdot ({\alpha }_t-{\alpha }_{t-1}+{\alpha }_{t-1}\cdot (1-{\alpha }_t))}}{\sqrt{{\alpha }_{t-1}\cdot (1-{\alpha }_t)}}]\cdot x_0\\ & =\sqrt{\frac{1-{\alpha }_{t-1}-{\sigma }_t^2}{1-{\alpha }_t}}\cdot x_t+[\sqrt{{\alpha }_{t-1}}-\frac{\sqrt{1-{\alpha }_{t-1}}}{\sqrt{1-{\alpha }_t}}\cdot \sqrt{\frac{{\alpha }_t\cdot ({\alpha }_t-{\alpha }_{t-1}+{\alpha }_{t-1}\cdot (1-{\alpha }_t))}{{\alpha }_{t-1}\cdot (1-{\alpha }_t)}}]\cdot x_0\\ & =\sqrt{\frac{1-{\alpha }_{t-1}-{\sigma }_t^2}{1-{\alpha }_t}}\cdot x_t+[\sqrt{{\alpha }_{t-1}}-\frac{\sqrt{1-{\alpha }_{t-1}}}{\sqrt{1-{\alpha }_t}}\cdot \sqrt{{\alpha }_t\cdot (1+\frac{{\alpha }_t-{\alpha }_{t-1}}{{\alpha }_{t-1}\cdot (1-{\alpha }_t)}})]\cdot x_0\\ & =\sqrt{\frac{1-{\alpha }_{t-1}-{\sigma }_t^2}{1-{\alpha }_t}}\cdot x_t+[\sqrt{{\alpha }_{t-1}}-\frac{1}{\sqrt{1-{\alpha }_t}}\cdot \sqrt{(1-{\alpha }_{t-1})\cdot {\alpha }_t\cdot (1-\frac{{\alpha }_{t-1}-{\alpha }_t}{{\alpha }_{t-1}\cdot (1-{\alpha }_t)}})]\cdot x_0\\ & =\sqrt{\frac{1-{\alpha }_{t-1}-{\sigma }_t^2}{1-{\alpha }_t}}\cdot x_t+[\sqrt{{\alpha }_{t-1}}-\frac{1}{\sqrt{1-{\alpha }_t}}\cdot \sqrt{{\alpha }_t\cdot (1-{\alpha }_{t-1}-\underset{{\sigma }_t^2}{\underbrace{\frac{({\alpha }_{t-1}-{\alpha }_t)\cdot (1-{\alpha }_{t-1})}{{\alpha }_{t-1}\cdot (1-{\alpha }_t)}}}})]\cdot x_0\\ & =\sqrt{\frac{1-{\alpha }_{t-1}-{\sigma }_t^2}{1-{\alpha }_t}}\cdot x_t+[\sqrt{{\alpha }_{t-1}}-\frac{1}{\sqrt{1-{\alpha }_t}}\cdot \sqrt{{\alpha }_t\cdot (1-{\alpha }_{t-1}-{\sigma }_t^2})]\cdot x_0\\ & =\sqrt{\frac{1-{\alpha }_{t-1}-{\sigma }_t^2}{1-{\alpha }_t}}\cdot x_t+[\sqrt{{\alpha }_{t-1}}-\frac{\sqrt{{\alpha }_t\cdot (1-{\alpha }_{t-1}-{\sigma }_t^2})}{\sqrt{1-{\alpha }_t}}]\cdot x_0\end{array}\end{array}$$

因此，前向传播过程中的$q(x_{t-1}|x_t,x_0)\sim N(x_{t-1};\sqrt{\frac{1-{\alpha }_{t-1}-{\sigma }_t^2}{1-{\alpha }_t}}\cdot x_t+[\sqrt{{\alpha }_{t-1}}-\frac{\sqrt{{\alpha }_t\cdot (1-{\alpha }_{t-1}-{\sigma }_t^2})}{\sqrt{1-{\alpha }_t}}]\cdot x_0,{\sigma }_t^{2}I)$

由于$x_0=\frac{x_t-\sqrt{1-{\alpha }_t}\cdot z_t}{{\alpha }_t}$，将此式替换掉公式（5）中的$x_0$。得到

$$\begin{array}{c}\begin{array}{rl}{\tilde{\mu }}_t(x_t,x_0)& =\sqrt{\frac{1-{\alpha }_{t-1}-{\sigma }_t^2}{1-{\alpha }_t}}\cdot x_t+[\sqrt{{\alpha }_{t-1}}-\frac{\sqrt{{\alpha }_t\cdot (1-{\alpha }_{t-1}-{\sigma }_t^2})}{\sqrt{1-{\alpha }_t}}]\cdot \underset{\frac{x_t-\sqrt{1-{\alpha }_t}\cdot z_t}{\sqrt{{\alpha }_t}}}{\underbrace{x_0}}\\ & =\sqrt{\frac{1-{\alpha }_{t-1}-{\sigma }_t^2}{1-{\alpha }_t}}\cdot x_t+[\sqrt{{\alpha }_{t-1}}-\frac{\sqrt{{\alpha }_t\cdot (1-{\alpha }_{t-1}-{\sigma }_t^2})}{\sqrt{1-{\alpha }_t}}]\cdot \frac{x_t-\sqrt{1-{\alpha }_t}\cdot z_t}{\sqrt{{\alpha }_t}}\\ & =\sqrt{\frac{1-{\alpha }_{t-1}-{\sigma }_t^2}{1-{\alpha }_t}}\cdot x_t+\sqrt{{\alpha }_{t-1}}\cdot \frac{x_t-\sqrt{1-{\alpha }_t}\cdot z_t}{\sqrt{{\alpha }_t}}-\frac{\sqrt{{\alpha }_t\cdot (1-{\alpha }_{t-1}-{\sigma }_t^2})}{\sqrt{1-{\alpha }_t}}\cdot \frac{x_t-\sqrt{1-{\alpha }_t}\cdot z_t}{\sqrt{{\alpha }_t}}\\ & =\sqrt{{\alpha }_{t-1}}\cdot \frac{x_t-\sqrt{1-{\alpha }_t}\cdot z_t}{\sqrt{{\alpha }_t}}+\sqrt{\frac{1-{\alpha }_{t-1}-{\sigma }_t^2}{1-{\alpha }_t}}\cdot x_t-\frac{\sqrt{1-{\alpha }_{t-1}-{\sigma }_t^2}\cdot (x_t-\sqrt{1-{\alpha }_t}\cdot z_t)}{\sqrt{1-{\alpha }_t}}\\ & =\sqrt{{\alpha }_{t-1}}\cdot \frac{x_t-\sqrt{1-{\alpha }_t}\cdot z_t}{\sqrt{{\alpha }_t}}+\frac{\sqrt{1-{\alpha }_{t-1}-{\sigma }_t^2}\cdot x_t-\sqrt{1-{\alpha }_{t-1}-{\sigma }_t^2}\cdot x_t+\sqrt{1-{\alpha }_{t-1}-{\sigma }_t^2}\cdot \sqrt{1-{\alpha }_t}\cdot z_t}{\sqrt{1-{\alpha }_t}}\\ & =\sqrt{{\alpha }_{t-1}}\cdot \frac{x_t-\sqrt{1-{\alpha }_t}\cdot z_t}{\sqrt{{\alpha }_t}}+\sqrt{1-{\alpha }_{t-1}-{\sigma }_t^2}\cdot z_t\end{array}\end{array}$$
结合公式（6）和公式（4），总结出后向去噪过程$p(x_{t-1}|x_t,x_0)$的均值和方差如公式 （7）所示
$$\begin{array}{c}\begin{array}{rl}{\tilde{\mu }}_t(x_t,x_0)& =\sqrt{{\alpha }_{t-1}}\cdot \frac{x_t-\sqrt{1-{\alpha }_t}\cdot z_t}{\sqrt{{\alpha }_t}}+\sqrt{1-{\alpha }_{t-1}-{\sigma }_t^2}\cdot z_t\\ {\sigma }_t^2& =\frac{1-{\alpha }_{t-1}}{1-{\alpha }_t}\cdot (1-\frac{{\alpha }_t}{{\alpha }_{t-1}})\end{array}\end{array}$$

因此，
$$\begin{array}{c}\begin{array}{rl}{x}_{t-1}& ={\tilde{\mu }}_t(x_t,x_0)+{\sigma }_t\cdot ϵ\\ & =\sqrt{{\alpha }_{t-1}}\cdot \frac{x_t-\sqrt{1-{\alpha }_t}\cdot z_t}{\sqrt{{\alpha }_t}}+\sqrt{1-{\alpha }_{t-1}-{\sigma }_t^2}\cdot z_t+{\sigma }_t\cdot ϵ\end{array}\end{array}$$
式中，$z_t$为在由$x_0\to x_t$过程中添加的噪音，为标准高斯分布，这个分布是带参$\theta$的模型需要预测的。由于$x_{t-1}$是一个高斯分布（非标准高斯分布），因此其表达式中需要添加$ϵ$，而$ϵ$为标准高斯分布。值得注意的是，这个公式（8）就是DDIM论文中的公式 （12）。