GBDT、XGBoost、LightGBM算法公式推导

一、GBDT公式推导

1、第一个基函数：
$$F_0(X)=\frac{1}{2}log\frac{1+\overline{y}}{1-\overline{y}}\text{\hspace{1em}(1.1)}$$
对于损失函数L(y,F)=log(1+e−2yF),y∈{ −1,1}L(y, F)=log(1+e^{-2yF}), y\in \left\{-1, 1\right\}L(y,F)=log(1+e−2yF),y∈{ −1,1}，求损失函数最小对应的$F$值。求一阶导数：
$$L^{{}^{\prime }}=\frac{-2y{e}^{-2yF}}{1+e^{-2yF}}\text{\hspace{1em}(1.2)}$$
设
$$\{\begin{array}{ll}& count(y=+1)=m1\\ & count(y=-1)=m2\end{array}\text{\hspace{1em}(1.3)}$$
则有
$$\{\begin{array}{l}m1+m2=n\\ \frac{m1-m2}{n}=\overline{y}\end{array}\text{\hspace{1em}(1.4)}$$
即
$$\{\begin{array}{l}m1=\frac{n}{2}(1+\overline{y})\\ m2=\frac{n}{2}(1-\overline{y})\end{array}\text{\hspace{1em}(1.5)}$$
令$\sum L^{{}^{\prime }}=0$得到,
$$\sum L^{{}^{\prime }}=\sum _{y=1}{L}^{{}^{\prime }}+\sum _{y=-1}{L}^{{}^{\prime }}=0\text{\hspace{1em}(1.6)}$$
将$(1.5)$带入$(1.6)$，得到，
$$\begin{array}{rl}{L}^{{}^{\prime }}& =\sum _{y=1}{L}^{{}^{\prime }}+\sum _{y=-1}{L}^{{}^{\prime }}\\ & \\ & =\frac{n}{2}(1+\overline{y})\ast \frac{-2e^{-2F}}{1+e^{-2F}}+\frac{n}{2}(1-\overline{y})\ast \frac{2e^{2F}}{1+e^{2F}}\\ & \\ & =\frac{n}{2}(1+\overline{y})\ast \frac{-2}{1+e^{2F}}+\frac{n}{2}(1-\overline{y})\ast \frac{2e^{2F}}{1+e^{2F}}\\ & \\ & =\frac{n}{1+e^{2F}}[-(1+\overline{y})+e^{2F}(1-\overline{y})]\end{array}\text{\hspace{1em}(1.7)}$$
由式$(1.7)=0$解得F0=12ln1+y‾1−y‾F_0=\frac{1}{2}ln\frac{1+\overline{y}}{1-\overline{y}}F0​=