朴素贝叶斯分类器（Naive Bayes）

1. 贝叶斯定理

假设有两个事件，事件$A$和事件$B$，已知事件$A$发生的概率为$p(A)$，事件$B$发生的概率为$P(B)$，事件$A$发生的前提下，事件$B$发生的概率为$p(B|A)$，事件$B$发生的前提下，事件$A$发生的概率为$p(A|B)$，事件$A$和事件$B$同时发生的概率是$p(AB)$，则有

$$p(AB)=p(A)p(B|A)=p(B)p(A|B)\tag{1}$$
根据式(1)可以推出贝叶斯定理为
$$p(B|A)=\frac{p(B)p(A|B)}{p(A)}\tag{2}$$
给定一个全集 $\{B_{1},B_{1},\dots,B_{n}\}$，其中 $B_{i}$与 $B_{j}$是不相交的，即 $B_{i}B_{j}=\varnothing$，则根据全概率公式，对于一个事件 $A$，会有
$$p(A)=\sum_{i=1}^{n}{p(B_{i})p(A|B_{i})}\tag{3}$$
则广义的贝叶斯定理有
$$p(B_{i}|A)=\frac{p(B_{i})p(A|B_{i})}{\sum_{i=1}^{n}{p(B_{i})p(A|B_{i})}}\tag{4}$$

2. 朴素贝叶斯基本原理

给定一组训练数据集$\{(\mathbf{X}_{1},y_{1}),(\mathbf{X}_{2},y_{2}),(\mathbf{X}_{3},y_{3}),\dots,(\mathbf{X}_{m},y_{m})\}$，其中，$m$是样本的个数，每个数据集包含着$n$个特征，即$\mathbf{X}_{i}=(x_{i1},x_{i2},\dots,x_{in})$。类标记集合为$\{y_{1},y_{2},\dots,y_{k}\}$。设$p(y=y_{i}|X=x)$表示输入的$X$样本为$x$时，输出的$y$为$y_{k}$的概率。
假设现在给定一个新的样本$\mathbf{x}$，要判断其属于哪一类，可分别求解$p(y=y_{1}|\mathbf{x})$，$p(y=y_{2}|\mathbf{x})$，$p(y=y_{3}|\mathbf{x})$，…，$p(y=y_{k}|\mathbf{x})$的值，哪一个值最大，就属于那一类。即，求解最大的后验概率 $arg max p(y|\mathbf{x})$。
那如何求解出这些后验概率呢？根据贝叶斯定理，有

$$p(y=y_{i}|\mathbf{x})=\frac{p(y_{i})p(\mathbf{x}|y_{i})}{p(x)}\tag{5}$$
一般地，朴素贝叶斯方法假设各个特征之间是相互独立的，则式(5)可以写成：
$$p(y=y_{i}|\mathbf{x})=\frac{p(y_{i})p(\mathbf{x}|y_{i})}{p(x)}=\frac{p(y_{i})\prod_{j=1}^{n}{p(x_{j}|y_{i})}}{\prod_{j=1}^{n}{p(x_{j})}}\tag{6}$$
由于(6)式的分母，对于每一个 $p(y=y_{i}|\mathbf{x})$求解都是一样的，所以，在实际操作中，可以省略掉。最终，朴素贝叶斯分类器的判别公式变成如下的形式：
$$y=arg \quad \max\limits_{y_{i}} p(y_{i})p(\mathbf{x}|y_{i})=arg \quad \max\limits_{y_{i}} p(y_{i})\prod_{j=1}^{n}{p(x_{j}|y_{i})} \tag{7}$$

下面，是如何通过样本对 $p(y)$ 和 $p(\mathbf{x}|y)$ 进行概率估计。

3. 朴素贝叶斯法的参数估计

3.1 极大似然估计

在朴素贝叶斯法中，学习就是意味着估计先验概率$p(y)$ 和 条件概率 $p(\mathbf{x}|y)$，然后根据先验概率和条件概率，去计算新的样本的后验概率 $p(y|\mathbf{x})$。其中，估计先验概率和条件概率的方法有很多，比如极大似然估计，多项式，高斯，伯努利等。
其中，在极大似然估计中，先验概率$p(y)$的极大似然估计如下：

$$p(y=y_{i})=\frac{样本中y_{i}标签的个数}{总样本的个数}\tag{8}$$
假设输入样本的第 $j$的特征中所有可能取值的集合是 $\{a_{j1},a_{j2},\dots,a_{js_{j}}\}$，则条件概率 $p(\mathbf{x}^{(j)}|y=y_{i})$的极大似然估计如下：
$$p(\mathbf{x}^{(j)}=a_{jl}|y=y_{i})=\frac{在属于y_{i}标签的样本中，第j个特征值等于a_{jl}的个数}{样本中属于y_{i}标签的个数}\tag{9}$$

例子1
该例子来自李航的《统计学习方法》。
表中$X^{(1)}$和$X^{(2)}$为特征，取值的集合分别是$A_{1}=\{1,2,3\}$，$A_{2}=\{S,M,L\}$，$Y$为类标记，$Y={1,-1}$。试求，$x=(2,S)$的类标记。数据如下所示，其中，特征$X^{(2)}$的取值$\{S,M,L\}$分别表示成$\{0,1,2\}$。

import numpy as np
import pandas as pd

x1 = np.array([1,1,1,1,1,2,2,2,2,2,3,3,3,3,3])
x2 = np.array([0,1,1,0,0,0,1,1,2,2,2,1,1,2,2])
y = np.array([-1,-1,1,1,-1,-1,-1,1,1,1,1,1,1,1,-1])

dataSet = np.concatenate((x1[:,None],x2[:,None],y[:,None]),axis=1)

df = pd.DataFrame(dataSet,index=np.arange(1,16,1),columns=['X1','X2','y'])

df.T

	1	2	3	4	5	6	7	8	9	10	11	12	13	14	15
X1	1	1	1	1	1	2	2	2	2	2	3	3	3	3	3
X2	0	1	1	0	0	0	1	1	2	2	2	1	1	2	2
y	-1	-1	1	1	-1	-1	-1	1	1	1	1	1	1	1	-1

求解

step1: 求解先验概率
$p(y=-1)=\frac{6}{15}$，$p(y=1)=\frac{9}{15}$

step2 求解条件概率
(2.1) 特征$X_{1}$
$p(X_{1}=1|y=-1)=\frac{3}{6}=\frac{1}{2}$，$p(X_{1}=2|y=-1)=\frac{2}{6}=\frac{1}{3}$， $p(X_{1}=3|y=-1)=\frac{1}{6}$
$p(X_{1}=1|y=1)=\frac{2}{9}$，$p(X_{1}=2|y=1)=\frac{3}{9}=\frac{1}{3}$， $p(X_{1}=3|y=1)=\frac{4}{9}$

(2.2) 特征$X_{1}$
$p(X_{2}=0|y=-1)=\frac{3}{6}=\frac{1}{2}$，$p(X_{2}=1|y=-1)=\frac{2}{6}=\frac{1}{3}$， $p(X_{2}=2|y=-1)=\frac{1}{6}$
$p(X_{2}=0|y=1)=\frac{1}{9}$，$p(X_{2}=1|y=1)=\frac{4}{9}=$， $p(X_{2}=2|y=1)=\frac{4}{9}$

step3 求解后验概率
$p(y=-1)p(\mathbf{X}=(2,S)|y=-1)=p(y=-1)p(X_{1}=2|y=-1)p(X_{2}=S|y=-1)=\frac{6}{15}\frac{1}{3}\frac{1}{2}=\frac{1}{15}$
$p(y=1)p(\mathbf{X}=(2,S)|y=1)=p(y=1)p(X_{1}=2|y=1)p(X_{2}=S|y=1)=\frac{9}{15}\frac{1}{3}\frac{1}{9}=\frac{1}{45}$

因为 $\frac{1}{15} > \frac{1}{45}$， 所以该样本的类标记为 $-1$

如下是python的极大似然估计的朴素贝叶斯代码，代码运行结果跟求解一致。

class MLENB:
    """
    Maximum likelihood estimation Naive Bayes

    Attributes
    ----------
    class_prior_ : array, shape (n_classes, )
        Smoothed empirical probability for each class.
    class_count_: array, shape (n_classes,)
        number of training samples observed in each class.
    MLE_: array, shape(n_classes, n_features)
        Maximum likelihood estimation of each feature per class, each of element is a dict
    """

    def __init__(self):
        pass

    def fit(self,X,y):
        """Fit maximum likelihood estimation Naive Bayes according to X, y

        Parameters
        ----------
        X : array-like, shape (n_samples, n_features)
            Training vectors, where n_samples is the number of samples
            and n_features is the number of features.
        y : array-like, shape (n_samples,)
            Target values.

        Returns
        -------
        self : object
            Returns self.
        """
        n_samples = X.shape[0]
        n_features = X.shape[1]
        n_classes = len(set(y))

        self.class_count_ = np.empty(n_classes)
        self.class_prior_ = np.empty(n_classes)
        self.MLE_ = np.empty((n_classes,n_features),dtype=dict)

        self.target_unique = np.unique(y)
        for i in range(n_classes):
            dataX_tu = X[y == self.target_unique[i]]
            self.class_prior_[i] = dataX_tu.shape[0] / float(len(y))
            self.class_count_[i] = dataX_tu.shape[0]

            for j in range(n_features):
                feature = dataX_tu[:,j]
                feature_unique = np.unique(feature)
                fp = {}
                for f_item in feature_unique:
                    fp[f_item] = list(feature).count(f_item) / float(len(feature))
                self.MLE_[i,j] = fp

        return self

    def __predict_likelihood(self,x):
        if x.ndim == 1:
            x = np.array([x])
        n_samples = x.shape[0]
        n_features = x.shape[1]
        n_classes = len(self.class_count_)

        likelihood = []
        for x_item in x:
            class_p = []
            for i in range(n_classes):
                p = self.class_prior_[i]
                for j in range(n_features):
                    if x_item[j] in self.MLE_[i,j]:
                        p *= self.MLE_[i,j][x_item[j]]
                    else:
                        p *= 0
                class_p.append(p)           
            likelihood.append(class_p)
        return np.array(likelihood)


    def predict(self,x):
        """Perform classification on an array of test vectors X.

       Parameters
        ----------
        X : array-like, shape = [n_samples, n_features]

        Returns
        -------
        C : array, shape = [n_samples]
            Predicted target values for X
        """

        likelihood = self.__predict_likelihood(x)
        max_index = np.argmax(likelihood, axis=1)
        return np.array([self.target_unique[i] for i in max_index])

    def predict_proba(self,x):
        """
        Return probability estimates for the test vector X.

        Parameters
        ----------
        X : array-like, shape = [n_samples, n_features]

        Returns
        -------
        C : array-like, shape = [n_samples, n_classes]
            Returns the probability of the samples for each class in
            the model. The columns correspond to the classes in sorted
            order, as they appear in the attribute `classes_`.
        """
        likelihood = self.__predict_likelihood(x)
        return np.array([lh / np.sum(lh) for lh in likelihood])    

# 测验结果
X = dataSet[:,0:-1]
y = dataSet[:,-1]

mlenb = MLENB()
mlenb.fit(X,y)
print(mlenb.predict(np.array([2,0])))
print(mlenb.predict_proba(np.array([2,0])))

[-1]
[[ 0.75  0.25]]

3.2 Multinomial Naive Bayes

用极大似然估计可能会出现所要估计的概率值为0的情况。这时会影响到后验概率的计算结果，使分类产生偏差。这时，可以采用多项式模型，对先验概率和条件概率做一些平滑处理。具体公式为：
先验概率$p(y)$的估计如下：

$$p(y=y_{i})=\frac{样本中y_{i}标签的个数+\alpha}{样本的总个数+总的类别个数\times\alpha}\tag{10}$$

假设输入样本的第$j$个特征的所有可能取值的集合是 $\{a_{j1},a_{j2},\dots,a_{js_{j}}\}$，则条件概率$p(x^{(j)}|y=y_{i})$的估计如下：

$$p(x^{(j)}|y=y_{i})=\frac{在属于y_{i}标签的样本中,第j个特征值等于a_{jl}的个数+\alpha}{属于y_{i}标签的个数+第j个特征的唯一值个数 \times \alpha}\tag{11}$$
其中， $\alpha$是平滑值。当 $\alpha=1$时，是拉普拉斯平滑（Laplace smoothing），当 $\alpha=0$时，退化到极大似然估计。当 $0<\alpha<1$时，称作Lidstone平滑。

有个疑问：多项式朴素贝叶斯与李航《统计学习方法》中说的贝叶斯估计有啥区别？本文的方法是参考李航的贝叶斯估计。

python的多项式朴素贝叶斯的参考代码如下：

class MultinomialNB:
    """Naive Bayes classifier for multinomial models
    Attributes
    ----------
    class_prior_ : array, shape (n_classes, )
        Smoothed empirical probability for each class.
    class_count_: array, shape (n_classes,)
        number of training samples observed in each class.
    bayes_estimation_: array, shape(n_classes, n_features)
        bayes estimations of each feature per class, each of element is a dict
    """
    def __init__(self, alpha=1.0):
        self.alpha_ = 1.0

    def fit(self,X,y):
        n_samples = X.shape[0]
        n_features = X.shape[1]
        n_classes = len(set(y))

        self.class_count_ = np.empty(n_classes)
        self.class_prior_ = np.empty(n_classes)
        self.bayes_estimation_ = np.empty((n_classes,n_features),dtype=dict)

        self.target_unique = np.unique(y)
        for i in range(n_classes):
            dataX_tu = X[y == self.target_unique[i]]
            self.class_prior_[i] = (dataX_tu.shape[0] + self.alpha_) / (float(len(y)) + n_classes * self.alpha_)
            self.class_count_[i] = dataX_tu.shape[0]

            for j in range(n_features):
                feature = dataX_tu[:,j]
                feature_unique = np.unique(feature)
                fp = {}
                for f_item in feature_unique:
                    fp[f_item] = (list(feature).count(f_item) + self.alpha_) / (float(len(feature)) + len(feature_unique) * self.alpha_)
                self.bayes_estimation_[i,j] = fp

        return self

    def __predict_likelihood(self,x):
        if x.ndim == 1:
            x = np.array([x])
        n_samples = x.shape[0]
        n_features = x.shape[1]
        n_classes = len(self.class_count_)

        likelihood = []
        for x_item in x:
            class_p = []
            for i in range(n_classes):
                p = self.class_prior_[i]
                for j in range(n_features):
                    if x_item[j] in self.bayes_estimation_[i,j]:
                        p *= self.bayes_estimation_[i,j][x_item[j]]
                    else:
                        p *= 0
                class_p.append(p)           
            likelihood.append(class_p)
        return np.array(likelihood)


    def predict(self,x):
        likelihood = self.__predict_likelihood(x)
        max_index = np.argmax(likelihood, axis=1)
        return np.array([self.target_unique[i] for i in max_index])

    def predict_proba(self,x):
        likelihood = self.__predict_likelihood(x)
        return np.array([lh / np.sum(lh) for lh in likelihood])

# 测验结果
X = dataSet[:,0:-1]
y = dataSet[:,-1]

mnb = MultinomialNB()
mnb.fit(X,y)
print(mnb.predict(np.array([2,0])))
print(mnb.predict_proba(np.array([2,0])))

[-1]
[[ 0.65116279  0.34883721]]

3.3 Gaussian Naive Bayes

当输入的特征是连续值的时候，我们无法用上面的方法来估计先验概率和条件概率，可以采用高斯模型。高斯模型假设特征服从高斯分布。
其特征的似然估计如下所示：

$$p(x_{i}|y)=\frac{1}{\sqrt{2\pi\sigma_{y}^{2}}}\exp(-\frac{(x_{i}-\mu_{y})^{2}}{2\sigma_{y}^{2}})\tag{12}$$
其中，
$\sigma_{y}^{2}$是第 $i$个特征的方差，$\mu_{y}$是第 i <script type="math/tex" id="MathJax-Element-102">i</script>个特征的均值。
其python代码如下：

class GaussianNB:
    """
    Attributes
    ----------
    class_prior_ : array, shape (n_classes,)
        probability of each class.
    class_count_ : array, shape (n_classes,)
        number of training samples observed in each class.
    theta_ : array, shape (n_classes, n_features)
        mean of each feature per class
    sigma_ : array, shape (n_classes, n_features)
        variance of each feature per class
    """

    def __init__(self):
        pass

    def fit(self, X, y):
        n_samples = X.shape[0]
        n_features = X.shape[1]
        n_classes = len(set(y))

        self.theta_ = np.zeros([n_classes,n_features]) 
        self.sigma_ = np.zeros([n_classes,n_features]) 
        self.class_prior = np.zeros(n_classes)   
        self.class_count = np.zeros(n_classes)   

        self.target_unique = np.unique(y)    
        for i in range(n_classes):
            dataX_tu = X[y == self.target_unique[i]]
            self.class_prior[i] = dataX_tu.shape[0] / float(len(y))
            self.class_count[i] = dataX_tu.shape[0]
            self.theta_[i,:] = np.mean(dataX_tu,axis=0)
            self.sigma_[i,:] = np.var(dataX_tu,axis=0)

        return self

    def __predict_likelihood(self,x):
        if x.ndim == 1:
            x = np.array([x])

        n_samples = x.shape[0]
        likelihood = []
        for x_item in x:
            gaussian = np.exp(-(x_item-self.theta_)**2 / (2 * self.sigma_)) / np.sqrt(2*np.pi*self.sigma_)
            p = np.exp(np.sum(np.log(gaussian),axis=1))
            likelihood.append(self.class_prior * p)
        return np.array(likelihood)

    def predict(self,x):
        likelihood = self.__predict_likelihood(x)
        max_index = np.argmax(likelihood, axis=1)
        return np.array([self.target_unique[i] for i in max_index])

    def predict_proba(self,x):
        likelihood = self.__predict_likelihood(x)
        return np.array([lh / np.sum(lh) for lh in likelihood])

# 测验结果
X = dataSet[:,0:-1]
y = dataSet[:,-1]

gnb = GaussianNB()
gnb.fit(X,y)
print(gnb.predict(np.array([2,0])))
print(gnb.predict_proba(np.array([2,0])))

[-1]
[[ 0.74566865  0.25433135]]

3.4 Bernoulli Naive Bayes

5. Naive Bayes 注意事项

1. Works only with categorical predictors, numerical predictors must be categorized or binned before use
2. Works with the assumption of predictor independence, and thus cannot detect or account for relationships between the predictors, unlike a decision tree for example.