Probability
Calculus-based Probability
This is content covered in STAT410 and STAT700 at UMD.
Basics
Axioms of Probability
- \(\displaystyle 0 \leq P(E) \leq 1\)
- \(\displaystyle P(S) = 1\) where \(\displaystyle S\) is your sample space
- For mutually exclusive events \(\displaystyle E_1, E_2, ...\), \(\displaystyle P\left(\bigcup_i^\infty E_i\right) = \sum_i^\infty P(E_i)\)
Monotonicity
- For all events \(\displaystyle A\) and \(\displaystyle B\), \(\displaystyle A \subset B \implies P(A) \leq P(B)\)
Conditional Probability
\(\displaystyle P(A|B)\) is the probability of event A given event B.
Mathematically, this is defined as \(\displaystyle P(A|B) = P(A,B) / P(B)\).
Note that this can also be written as \(\displaystyle P(A|B)P(B) = P(A, B)\)
With some additional substitution, we get Baye's Theorem:
\(\displaystyle
P(A|B) = \frac{P(B|A)P(A)}{P(B)}
\)
Random Variables
A random variable is a variable which takes on a distribution rather than a value.
PMF, PDF, CDF
For discrete distributions, we call \(\displaystyle p_{X}(x)=P(X=x)\) the probability mass function (PMF).
For continuous distributions, we have the probability density function (PDF) \(\displaystyle f(x)\).
The comulative distribution function (CDF) is \(\displaystyle F(x) = P(X \leq x)\).
The CDF is the prefix sum of the PMF or the integral of the PDF. Likewise, the PDF is the derivative of the CDF.
Joint Random Variables
Two random variables are independant iff \(\displaystyle f_{X,Y}(x,y) = f_X(x) f_Y(y)\).
Otherwise, the marginal distribution is \(\displaystyle f_X(x) = \int f_{X,Y}(x,y) dy\).
Change of variables
Let \(\displaystyle g\) be a monotonic increasing function and \(\displaystyle Y = g(X)\).
Then \(\displaystyle F_Y(y) = P(Y \leq y) = P(X \leq g^{-1}(y)) = F_X(g^{-1}(y))\).
And \(\displaystyle f_Y(y) = \frac{d}{dy} F_Y(y) = \frac{d}{dy} F_X(g^{-1}(y)) = f_X(g^{-1}(y)) \frac{d}{dy}g^{-1}(y)\)
Hence:
\[
f_Y(y) = f_x(g^{-1}(y)) \frac{d}{dy} g^{-1}(y)
\]
Expectation and Variance
Some definitions and properties.
Definitions
Let \(\displaystyle X \sim D\) for some distribution \(\displaystyle D\). Let \(\displaystyle S\) be the support or domain of your distribution.
- \(\displaystyle E(X) = \sum_S xp(x)\) or \(\displaystyle \int_S xp(x)dx\)
- \(\displaystyle Var(X) = E[(X-E(X))^2] = E(X^2) - (E(X))^2\)
Total Expection
\(\displaystyle E_{X}(X) = E_{Y}(E_{X|Y}(X|Y))\)
Dr. Xu refers to this as the smooth property.
\(\displaystyle \begin{aligned} E(X) &= \int_S x p(x)dx \\ &= \int_x x \int_y p(x,y)dy dx \\ &= \int_x x \int_y p(x|y)p(y)dy dx \\ &= \int_y\int_x x p(x|y)dxp(y)dy \end{aligned} \)
Total Variance
\(\displaystyle Var(Y) = E(Var(Y|X)) + Var(E(Y | X))\)
This one is not used as often on tests as total expectation
\(\displaystyle \begin{aligned} Var(Y) &= E(Y^2) - E(Y)^2 \\ &= E(E(Y^2|X)) - E(E(Y|X))^2\\ &= E(Var(Y|X) + E(Y|X)^2) - E(E(Y|X))^2\\ &= E((Var(Y|X)) + E(E(Y|X)^2) - E(E(Y|X))^2\\ &= E((Var(Y|X)) + Var(E(Y|X))\\ \end{aligned} \)
Sample Mean and Variance
The sample mean is \(\displaystyle \bar{X} = \frac{1}{n}\sum_{i=1}^{n}X_i\).
The unbiased sample variance is \(\displaystyle S^2 = \frac{1}{n-1}\sum_{i=1}^{n}(X_i - \bar{X})^2\).
Student's Theorem
Let \(\displaystyle X_1,...,X_n\) be from \(\displaystyle N(\mu, \sigma^2)\).
Then the following results about the sample mean \(\displaystyle \bar{X}\)
and the unbiased sample variance \(\displaystyle S^2\) hold:
- \(\displaystyle \bar{X}\) and \(\displaystyle S^2\) are independent
- \(\displaystyle \bar{X} \sim N(\mu, \sigma^2 / n)\)
- \(\displaystyle (n-1)S^2 / \sigma^2 \sim \chi^2(n-1)\)
Jensen's Inequality
Let g be a convex function (i.e. second derivative is positive). Then \(\displaystyle g(E(x)) \leq E(g(x))\).
Moments and Moment Generating Functions
Definitions
- \(\displaystyle E(X^n)\) the n'th moment
- \(\displaystyle E((X-\mu)^n)\) the n'th central moment
- \(\displaystyle E(((X-\mu) / \sigma)^n)\) the n'th standardized moment
Expectation is the first moment and variance is the second central moment.
Additionally, skew is the third standardized moment and kurtosis is the fourth standardized moment.
Moment Generating Functions
To compute moments, we can use a moment generating function (MGF): \[M_X(t) = E(e^{tX})\] With the MGF, we can get any order moments by taking n derivatives and setting \(t=0\).
- Notes
- The MGF, if it exists, uniquely defines the distribution.
- The MGF of \(\displaystyle X+Y\) is \(\displaystyle MGF_{X+Y}(t) = E(e^{t(X+Y)})=E(e^{tX})E(e^{tY}) = MGF_X(t) * MGF_Y(t)\)
Characteristic function
Convergence
There are 4 common types of convergence.
Almost Surely
- \(\displaystyle P(\lim X_i = X) = 1\)
In Probability
For all \(\displaystyle \epsilon \gt 0\)
\(\displaystyle \lim P(|X_i - X| \geq \epsilon) = 0\)
- Implies Convergence in distribution
In Distribution
Pointwise convergence of the cdf
A sequence of random variables \(\displaystyle X_1,...\) converges to \(\displaystyle X\) in probability
if for all \(\displaystyle x \in S\),
\(\displaystyle \lim_{i \rightarrow \infty} F_i(x) = F(x)\)
- Equivalent to convergence in probability if it converges to a degenerate distribution (i.e. a number)
In Mean Squared
\(\displaystyle \lim_{i \rightarrow \infty} E(|X_i-X|^2)=0\)
Delta Method
Suppose \(\displaystyle \sqrt{n}(X_n - \theta) \xrightarrow{D} N(0, \sigma^2)\).
Let \(\displaystyle g\) be a function such that \(\displaystyle g'\) exists and \(\displaystyle g'(\theta) \neq 0\)
Then \(\displaystyle \sqrt{n}(g(X_n) - g(\theta)) \xrightarrow{D} N(0, \sigma^2 g'(\theta)^2)\)
Multivariate:
\(\displaystyle \sqrt{n}(B - \beta) \xrightarrow{D} N(0, \Sigma) \implies \sqrt{n}(h(B)-h(\beta)) \xrightarrow{D} N(0, h'(\theta)^T \Sigma h'(\theta))\)
- Notes
- You can think of this like the Mean Value theorem for random variables.
- \(\displaystyle (g(X_n) - g(\theta)) \approx g'(\theta)(X_n - \theta)\)
Order Statistics
Consider iid random variables \(\displaystyle X_1, ..., X_n\).
Then the order statistics are \(\displaystyle X_{(1)}, ..., X_{(n)}\) where \(\displaystyle X_{(i)}\) represents the i'th smallest number.
Min and Max
The easiest to reason about are the minimum and maximum order statistics: \(\displaystyle P(X_{(1)} \lt = x) = P(\text{min}(X_i) \lt = x) = 1 - P(X_1 \gt x, ..., X_n \gt x)\) \(\displaystyle P(X_{(n)} \lt = x) = P(\text{max}(X_i) \lt = x) = P(X_1 \lt = x, ..., X_n \lt = x)\)
Joint PDF
If \(\displaystyle X_i\) has pdf \(\displaystyle f\), the joint pdf of \(\displaystyle X_{(1)}, ..., X_{(n)}\) is: \(\displaystyle g(x_1, ...) = n!*f(x_1)*...*f(x_n) \) since there are n! ways perform a change of variables.
Individual PDF
\(\displaystyle f_{X(i)}(x) = \frac{n!}{(i-1)!(n-i)!} F(x)^{i-1} f(x) [1-F(x)]^{n-1} \)
Inequalities and Limit Theorems
Markov's Inequality
Let \(\displaystyle X\) be a non-negative random variable.
Then \(\displaystyle P(X \geq a) \leq \frac{E(X)}{a}\)
\(\displaystyle \begin{aligned} E(X) &= \int_{0}^{\infty}xf(x)dx \\ &= \int_{0}^{a}xf(x)dx + \int_{a}^{\infty}xf(x)dx\\ &\geq \int_{a}^{\infty}xf(x)dx\\ &\geq \int_{a}^{\infty}af(x)dx\\ &=a \int_{a}^{\infty}f(x)dx\\ &=a * P(X \geq a)\\ \implies& P(X \geq a) \leq \frac{E(X)}{a} \end{aligned} \)
Chebyshev's Inequality
- \(\displaystyle P(|X - \mu| \geq k \sigma) \leq \frac{1}{k^2}\)
- \(\displaystyle P(|X - \mu| \geq k) \leq \frac{\sigma^2}{k^2}\)
Apply Markov's inequality:
Let \(\displaystyle Y = |X - \mu|\)
Then:
\(\displaystyle
\begin{aligned}
P(|X - \mu| \geq k) &= P(Y \geq k) \\
&= P(Y^2 \geq k^2) \\
&\leq \frac{E(Y^2)}{k^2} \\
&= \frac{E((X - \mu)^2)}{k^2}
\end{aligned}
\)
- Usually used to prove convergence in probability
Central Limit Theorem
Very very important. Never forget this.
For any distribution, the sample mean converges in distribution to normal.
Let \(\displaystyle \mu = E(x)\) and \(\displaystyle \sigma^2 = Var(x)\)
Different ways of saying the same thing:
- \(\displaystyle \sqrt{n}(\bar{x} - \mu) \sim N(0, \sigma^2)\)
- \(\displaystyle \frac{\sqrt{n}}{\sigma}(\bar{x} - \mu) \sim N(0, 1)\)
- \(\displaystyle \bar{x} \sim N(\mu, \sigma^2/n)\)
Law of Large Numbers
The sample mean converges to the population mean in probability.
For all \(\displaystyle \epsilon \gt 0\),
\(\displaystyle \lim_{n \rightarrow \infty} P(|\bar{X}_n - E(X)| \geq \epsilon) = 0\)
- Notes
- The sample mean converges to the population mean almost surely.
Properties and Relationships between distributions
- This is important for exams.
Poisson Distribution
- If \(\displaystyle X_i \sim Poisson(\lambda_i)\) then \(\displaystyle \sum X_i \sim Poisson(\sum \lambda_i)\)
Normal Distribution
- If \(\displaystyle X_1 \sim N(\mu_1, \sigma_1^2)\) and \(\displaystyle X_2 \sim N(\mu_2, \sigma_2^2)\) then \(\displaystyle \lambda_1 X_1 + \lambda_2 X_2 \sim N(\lambda_1 \mu_1 + \lambda_2 X_2, \lambda_1^2 \sigma_1^2 + \lambda_2^2 + \sigma_2^2)\) for any \(\displaystyle \lambda_1, \lambda_2 \in \mathbb{R}\)
Exponential Distribution
- \(\displaystyle \operatorname{Exp}(\lambda)\) is equivalent to \(\displaystyle \Gamma(1, 1/\lambda)\)
- Note that some conventions flip the second parameter of gamma, so it would be \(\displaystyle \Gamma(1, \lambda)\)
- If \(\displaystyle \epsilon_1, ..., \epsilon_n\) are exponential distributions then \(\displaystyle \min\{\epsilon_i\} \sim \exp(\sum \lambda_i)\)
- Note that the maximum is not exponentially distributed
- However, if \(\displaystyle X_1, ..., X_n \sim \exp(1)\) then \(\displaystyle Z_n=n\exp(\max\{\epsilon_i\}) \rightarrow \exp(1)\)
Gamma Distribution
Note exponential distributions are also Gamma distrubitions
- If \(\displaystyle X \sim \Gamma(k, \theta)\) then \(\displaystyle \lambda X \sim \Gamma(k, c\theta)\).
- If \(\displaystyle X_1 \sim \Gamma(k_1, \theta)\) and \(\displaystyle X_2 \sim \Gamma(k_2, \theta)\) then \(\displaystyle X_2 + X_2 \sim \Gamma(k_1 + k_2, \theta)\).
- If \(\displaystyle X_1 \sim \Gamma(\alpha, \theta)\) and \(\displaystyle X_2 \sim \Gamma(\beta, \theta)\), then \(\displaystyle \frac{X_1}{X_1 + X_2} \sim B(\alpha, \beta)\).
T-distribution
- Ratio of standard normal and squared-root of Chi-sq distribution yields T-distribution.
- If \(\displaystyle Z \sim N(0,1)\) and \(\displaystyle V \sim \Chi^2(v)\) then \(\displaystyle \frac{Z}{\sqrt{V/v}} \sim \text{t-dist}(v)\)
Chi-Sq Distribution
- The ratio of two normalized Chi-sq is an F-distributions
- If \(\displaystyle X \sim \chi^2_{d1}\) and \(\displaystyle Y \sim \chi^2_{d2}\) then \(\displaystyle \frac{X/d1}{Y/d2} \sim F(d1,d2)\)
- If \(\displaystyle Z_1,...,Z_k \sim N(0,1)\) then \(\displaystyle Z_1^2 + ... + Z_k^2 \sim \Chi^2(k)\)
- If \(\displaystyle X_i \sim \Chi^2(k_i)\) then \(\displaystyle X_1 + ... + X_n \sim \Chi^2(k_1 +...+ k_n)\)
- \(\displaystyle \Chi^2(k)\) is equivalent to \(\displaystyle \Gamma(k/2, 2)\)
F Distribution
Too many to list. See Wikipedia: F-distribution.
Most important are Chi-sq and T distribution:
- If \(\displaystyle X \sim \chi^2_{d1}\) and \(\displaystyle Y \sim \chi^2_{d2}\) then \(\displaystyle \frac{X/d1}{Y/d2} \sim F(d1,d2)\)
- If \(\displaystyle X \sim t_{(n)}\) then \(\displaystyle X^2 \sim F(1, n)\) and \(\displaystyle X^{-2} \sim F(n, 1)\)