Spearman's rank correlation coefficient 和 Pearson correlation coefficient详细

最新推荐文章于 2024-09-30 09:52:48 发布

原创最新推荐文章于 2024-09-30 09:52:48 发布 · 8.2k 阅读

5 ·

CC 4.0 BY-SA版权

Data Mining 专栏收录该内容

18 篇文章

订阅专栏

In statistics, Spearman's rank correlation coefficient or Spearman's rho, named after Charles Spearman and often denoted by the Greek letter $\rho$ (rho) or as $r_s$ , is a nonparametric measure of statistical dependence between two variables. It assesses how well the relationship between two variables can be described using a monotonic function. If there are no repeated data values, a perfect Spearman correlation of +1 or −1 occurs when each of the variables is a perfect monotone function of the other.

Spearman's coefficient, like any correlation calculation, is appropriate for both continuous and discrete variables, including ordinal variables.^[1]^[2] Spearman's $\rho$ and Kendall's $\tau$ can be formulated as special cases of a more general correlation coefficient.

Definition and calculation[edit]

The Spearman correlation coefficient is defined as the Pearson correlation coefficient between the ranked variables.^[3]

For a sample of size n, the n raw scores $X_i, Y_i$ are converted to ranks $\operatorname{rg} X_i, \operatorname{rg} Y_i$ , and $r_s$ is computed from:

r_s = \rho_{\operatorname{rg}_X,\operatorname{rg}_Y} = \frac {\operatorname{cov}(\operatorname{rg}_X,\operatorname{rg}_Y)} { \sigma_{\operatorname{rg}_X} \sigma_{\operatorname{rg}_Y} }

where

$\rho$ denotes the usual Pearson correlation coefficient, but applied to the rank variables.
$\operatorname{cov}(\operatorname{rg}_X, \operatorname{rg}_Y)$ is the covariance of the rank variables.
$\sigma_{rg_X}$ and $\sigma_{rg_Y}$ are the standard deviations of the rank variables.

Only if all n ranks are distinct integers, it can be computed using the popular formula

r_s = {1- \frac {6 \sum d_i^2}{n(n^2 - 1)}}.

where

$d_i = rg(X_i) - rg(Y_i)$ , is the difference between the two ranks of each observation.
n is the number of observations

Identical values are usually each assigned fractional ranks equal to the average of their positions in the ascending order of the values, which is equivalent to averaging over all possible permutations.

If ties are present in the data set, this equation yields incorrect results: Only if in both variables all ranks are distinct, then $\sigma_{\operatorname{rg}_X} \sigma_{\operatorname{rg}_Y} = \operatorname{Var}{\operatorname{rg}_X} = \operatorname{Var}{\operatorname{rg}_Y} = n(n^2 - 1)/6$ (cf. tetrahedral number $T_{n-1}$ ). The first equation—normalizing by the standard deviation—may even be used even when ranks are normalized to [0;1] ("relative ranks") because it is insensitive both to translation and linear scaling.

This method should also not be used in cases where the data set is truncated; that is, when the Spearman correlation coefficient is desired for the top X records (whether by pre-change rank or post-change rank, or both), the user should use the Pearson correlation coefficient formula given above.^{[citation needed]}

The standard error of the coefficient (σ) was determined by Pearson in 1907 and Gosset in 1920. It is

$\sigma_{r_s} = \frac{ 0.6325 }{ \sqrt{n-1} }$

Example[edit]

In this example, the raw data in the table below is used to calculate the correlation between the IQ of a person with the number of hours spent in front of TV per week.

IQ, $X_i$	Hours of TV per week, $Y_i$
106	7
86	0
100	27
101	50
99	28
103	29
97	20
113	12
112	6
110	17

Firstly, evaluate $d^2_i$ . To do so use the following steps, reflected in the table below.

Sort the data by the first column ( $X_i$ ). Create a new column $x_i$ and assign it the ranked values 1,2,3,...n.
Next, sort the data by the second column ( $Y_i$ ). Create a fourth column $y_i$ and similarly assign it the ranked values 1,2,3,...n.
Create a fifth column $d_i$ to hold the differences between the two rank columns ( $x_i$ and $y_i$ ).
Create one final column $d^2_i$ to hold the value of column $d_i$ squared.

IQ, $X_i$	Hours of TV per week, $Y_i$	rank $x_i$	rank $y_i$	$d_i$	$d^2_i$
86	0	1	1	0	0
97	20	2	6	−4	16
99	28	3	8	−5	25
100	27	4	7	−3	9
101	50	5	10	−5	25
103	29	6	9	−3	9
106	7	7	3	4	16
110	17	8	5	3	9
112	6	9	2	7	49
113	12	10	4	6	36

With $d^2_i$ found, add them to find $\sum d_i^2 = 194$ . The value of n is 10. These values can now be substituted back into the equation : $\rho = 1- {\frac {6 \sum d_i^2}{n(n^2 - 1)}}.$ to give

\rho = 1- {\frac {6\times194}{10(10^2 - 1)}}

which evaluates to ρ = -29/165 = −0.175757575... with a P-value = 0.627188 (using the t distribution)

Chart of the data presented. It can be seen that there might be a negative correlation, but that the relationship does not appear definitive.

This low value shows that the correlation between IQ and hours spent watching TV is very low, although the negative value suggests that the longer the time spent watching television the lower the IQ. In the case of ties in the original values, this formula should not be used; instead, the Pearson correlation coefficient should be calculated on the ranks (where ties are given ranks, as described above).

皮尔森相关系数

皮尔森相关系数（Pearson correlation coefficient）也叫皮尔森积差相关系数（Pearson product-moment correlation coefficient），是用来反应两个变量相似程度的统计量。或者说可以用来计算两个向量的相似度（在基于向量空间模型的文本分类、用户喜好推荐系统中都有应用）。

皮尔森相关系数计算公式如下：

ρX,Y=cov(X,Y)σXσY=E((X−μX)(Y−μY))σXσY=E(XY)−E(X)E(Y)E(X2)−E2(X)√E(Y2)−E2(Y)√

分子是协方差，分子是两个变量标准差的乘积。显然要求X和Y的标准差都不能为0。

当两个变量的线性关系增强时，相关系数趋于1或-1。正相关时趋于1，负相关时趋于-1。当两个变量独立时相关系统为0，但反之不成立。比如对于 y=x2 ，X服从[-1,1]上的均匀分布，此时E(XY)为0，E(X)也为0，所以 ρX,Y=0 ，但x和y明显不独立。所以“不相关”和“独立”是两回事。当Y 和X服从联合正态分布时，其相互独立和不相关是等价的。

对于居中的数据来说（何谓居中？也就是每个数据减去样本均值，居中后它们的平均值就为0），E(X)=E(Y)=0，此时有：

ρX,Y=E(XY)E(X2)√E(Y2)√=1N∑Ni=1XiYi1N∑Ni=1X2i√1N∑Ni=1Y2i√=∑Ni=1XiYi∑Ni=1X2i√∑Ni=1Y2i√=∑Ni=1XiYi||X||||Y||

即相关系数可以看作是两个随机变量中得到的样本集向量之间夹角的cosine函数。

进一步当X和Y向量归一化后，||X||=||Y||=1，相关系数即为两个向量的乘积 ρX,Y=X∙Y 。

Spearman秩相关系数

首先说明秩相关系数还有其他类型，比如kendal秩相关系数。

使用Pearson线性相关系数有2个局限：

必须假设数据是成对地从正态分布中取得的。
数据至少在逻辑范围内是等距的。

对于更一般的情况有其他的一些解决方案，Spearman秩相关系数就是其中一种。Spearman秩相关系数是一种无参数（与分布无关）检验方法，用于度量变量之间联系的强弱。在没有重复数据的情况下，如果一个变量是另外一个变量的严格单调函数，则Spearman秩相关系数就是+1或-1，称变量完全Spearman秩相关。注意这和Pearson完全相关的区别，只有当两变量存在线性关系时，Pearson相关系数才为+1或-1。

对原始数据x_i,y_i按从大到小排序，记x'_i,y'_i为原始x_i,y_i在排序后列表中的位置，x'_i,y'_i称为x_i,y_i的秩次，秩次差d_i=x'_i-y'_i。Spearman秩相关系数为：