数学基础 Probability Theory

基础知识

概率的定义 Axioms of Probability:

• Sample space $\Omega$: The set of all the outcomes of a random experiment.
• Set of events(or event space) $\mathcal{F}$: where each event $A \in \mathcal{F}$ is a set containing zero or more outcomes (i.e., $A \subseteq \Omega$ is a collection of possible outcomes of an experiment).
• Probability measure: A function $P: \mathcal{F} \longrightarrow \mathbb{R}$ that satisfies the following propeties:
  
  • $P(A) \ge 0$ for all $A \in \mathcal{F}$
  • $P(\Omega) = 1$
  • If $A_1,...,A_k$ are disjoint events (i.e., $A_i \cap A_j = \emptyset$ whereever $i \neq j$), then $P(\cup_iA_i) = \displaystyle\sum_iP(A_i)$

理解：以掷骰子为例，$\Omega$ 就是所有的可能出现的点数集{1, 2, 3, 4, 5, 6}，$\mathcal{F}$包括了各种事件，比如{1, 2, 3, 4}, {奇数点数}，{偶数点数}等等。

概率的基本属性:

1. $\text{If } A \subseteq B \Longrightarrow P(A) \leq P(B)$
2. $P(A \cap B) \leq \min(P(A), P(B))$
3. $P(A \cup B) \leq P(A) + P(B)$
4. $P(\Omega \backslash A) = P(\overline{A}) = 1 - P(A)$
5. If $A_1,...,A_k$ are a set of disjoint events such that $\textstyle\cup_{i=1}^kA_i=\Omega$, then $\textstyle\sum_{i=1}^kP(A_k)=1$.

条件概率：

$$P(A|B) = {P(A \cap B) \over P(B)}$$
$P(A|B)$ is the probability measure of the event A after observing the occurrence of event B. Two events are independent if and only if $P(A \cap B) = P(A)P(B)$ or $P(A|B) =P(A)$.

随机变量

扔10次硬币，出现的所有10个正反面(heads and tails)组合(考虑先后顺序)便是样本空间 $\Omega$，比如$\omega_0 = <H, H, T, H, T, H, H, T, T, T> \in \Omega$. 实际问题中，我们往往不关心出现某个特定正反面序列的概率，而更关心real-valued functions of outcomes，比如十次中出现正面的次数，或者连续反面的最长长度，这些函数便是random variables随机变量。所以随机变量是一个函数！
Random variable $X$ is a function $X: \Omega \longrightarrow \mathbb{R}$

• Discrete random variable: $P(X = k) := P(\{\omega: X(\omega) = k\})$

• Continuous random variable: $P(a \leq X \leq b) := P(\{\omega: a \leq X(\omega) \leq b\})$

CDFs, PDFs, PMFs

1.Cumulative distribution function (CDF) is a function $F_X: \mathbb{R} \longrightarrow [0,1]$ such that

$$F_X(x) \triangleq P(X \leq x)$$
By using this function one can calculate the probability of any event in $\mathcal{F}$.
Properties:

• $0 \leq F_X(x) \leq 1$.
• $\lim\nolimits_{x \rightarrow -\infty} F_X(x) = 0$.
• $\lim\nolimits_{x \rightarrow \infty} F_X(x) = 1$.
• $x \leq y \Longrightarrow F_X(x) \leq F_X(y)$.

2.Probability mass function (PMF) is a function $p_X: \Omega \longrightarrow \mathbb{R}$ such that

$$p_X(x) \triangleq P(X=x)$$
Properties:

• $0 \leq p_X(x) \leq 1$.
• $\textstyle\sum_{x \in Val(X)}p_X(x) = 1$, $Val(X)$ is the set of all possible values $X$ may assume.
• $\textstyle\sum_{x \in A}p_X(x) = P(X \in A) ?$.

3.Probability density functions (PDF) is the derivative of the CDF:

$$f_X(x) \triangleq {dF_X(x) \over dx}$$
PDF for a continuous random variable may not always exist and for very small