- 1 Estimation
- 2 Tests
- 3 Confidence Sets
- 4 Regression
- 5 Quadratic Forms
- 6 Bootstrapping
- 7 Textbooks
Method of Moments Estimator
Sometimes referred to as MME or MMO
- Calculate your population moments in terms of your parameters
- Then invert to get your parameters as a function of your moments
- Replace population moments with sample moments
Maximum Likelihood Estimator
(MLE) Maximum Likelihood Estimator
- Write out the likelihood function
- (Optional) Write out the log-likelihood function
- Take the derivative of the log-likelihood function w.r.t
- Find the maximum of the log-likelihood function by setting the first derivative to 0
- (Optional) Make sure it is the maximum by checking that the Hessian is positive definite
- Your MLE is the value which maximizes
- Note if the derivative is always 0, then any value is the MLE. If it is always positive, then take the largest possible value.
- If is the MLE for then the MLE for is
Uniformly Minimum Variance Unbiased Estimator (UMVUE)
UMVUE, sometimes called MVUE or UMVU.
See Wikipedia: Lehmann–Scheffé theorem
An unbiased estimator of a complete-sufficient statistics is a UMVUE.
In general, you should find a complete sufficient statistic using the property of exponential families.
Then make it unbiased with some factors to get the UMVUE.
An estimator is unbiased for if
- is unbiased for but is not consistent
An estimator is consistent for if it converges in probability to
- Example: is a consistent estimator
- for for but is not unbiased.
- or if is twice differentiable
- is the fisher information of the sample. Replace with your full likelihood.
Cramer-Rao Lower Bound
Given an estimator , let . Then
- If is unbiased then
- Our lower bound will be
The efficiency of an unbiased estimator is defined as
Used to test the mean.
Use to test the ratio of variances.
Likelihood Ratio Test
Uniformly Most Powerful Test
See Wikipedia: Neyman-Pearson Lemma
Relationship with Tests
Boostrapping is used to sample from your sample to get a measure of accuracy of your statistics.
In nonparametric bootstrapping, you resample from your sample with replacement.
In this scenario, you don't need to know the family of distributions that your sample comes from.
In parametric bootstrapping, you learn the distribution parameters of your sample, e.g. with MLE.
Then you can generate samples from that distribution on a computer.