Preface
Acknowledgments
Mathematical Preparation
Notation
1. Basic Probability Theory
1.1 Introduction
1.2 Outcomes and Events
1.3 Probability Function
1.4 Properties of the Probability Function
1.5 Equally Likely Outcomes
1.6 Joint Events
1.7 Conditional Probability
1.8 Independence
1.9 Law of Total Probability
1.10 Bayes Rule
1.11 Permutations and Combinations
1.12 Sampling with and without Replacement
1.13 Poker Hands
1.14 Sigma Fields
1.15 Technical Proofs
1.16 Exercises
2. Random Variables
2.1 Introduction
2.2 Random Variables
2.3 Discrete Random Variables
2.4 Transformations
2.5 Expectation
2.6 Finiteness of Expectations
2.7 Distribution Function
2.8 Continuous Random Variables
2.9 Quantiles
2.10 Density Functions
2.11 Transformations of Continuous Random Variables
2.12 Non-Monotonic Transformations
2.13 Expectation of Continuous Random Variables
2.14 Finiteness of Expectations
2.15 Unifying Notation
2.16 Mean and Variance
2.17 Moments
2.18 Jensen’s Inequality
2.19 Applications of Jensen’s Inequality
2.20 Symmetric Distributions
2.21 Truncated Distributions
2.22 Censored Distributions
2.23 Moment Generating Function
2.24 Cumulants
2.25 Characteristic Function
2.26 Expectation: Mathematical Details
2.27 Exercises
3. Parametric Distributions
3.1 Introduction
3.2 Bernoulli Distribution
3.3 Rademacher Distribution
3.4 Binomial Distribution
3.5 Multinomial Distribution
3.6 Poisson Distribution
3.7 Negative Binomial Distribution
3.8 Uniform Distribution
3.9 Exponential Distribution
3.10 Double Exponential Distribution
3.11 Generalized Exponential Distribution
3.12 Normal Distribution
3.13 Cauchy Distribution
3.14 Student t Distribution
3.15 Logistic Distribution
3.16 Chi-Square Distribution
3.17 Gamma Distribution
3.18 F Distribution
3.19 Non-Central Chi-Square
3.20 Beta Distribution
3.21 Pareto Distribution
3.22 Lognormal Distribution
3.23 Weibull Distribution
3.24 Extreme Value Distribution
3.25 Mixtures of Normals
3.26 Technical Proofs
3.27 Exercises
4. Multivariate Distributions
4.1 Introduction
4.2 Bivariate Random Variables
4.3 Bivariate Distribution Functions
4.4 Probability Mass Function
4.5 Probability Density Function
4.6 Marginal Distribution
4.7 Bivariate Expectation
4.8 Conditional Distribution for Discrete X
4.9 Conditional Distribution for Continuous X
4.10 Visualizing Conditional Densities
4.11 Independence
4.12 Covariance and Correlation
4.13 Cauchy-Schwarz Inequality
4.14 Conditional Expectation
4.15 Law of Iterated Expectations
4.16 Conditional Variance
4.17 Hölder’s and Minkowski’s Inequalities
4.18 Vector Notation
4.19 Triangle Inequalities
4.20 Multivariate Random Vectors
4.21 Pairs of Multivariate Vectors
4.22 Multivariate Transformations
4.23 Convolutions
4.24 Hierarchical Distributions
4.25 Existence and Uniqueness of the Conditional Expectation
4.26 Identification
4.27 Exercises
5. Normal and Related Distributions
5.1 Introduction
5.2 Univariate Normal
5.3 Moments of the Normal Distribution
5.4 Normal Cumulants
5.5 Normal Quantiles
5.6 Truncated and Censored Normal Distributions
5.7 Multivariate Normal
5.8 Properties of the Multivariate Normal
5.9 Chi-Square, t, F, and Cauchy Distributions
5.10 Hermite Polynomials
5.11 Technical Proofs
5.12 Exercises
6. Sampling
6.1 Introduction
6.2 Samples
6.3 Empirical Illustration
6.4 Statistics, Parameters, and Estimators
6.5 Sample Mean
6.6 Expected Value of Transformations
6.7 Functions of Parameters
6.8 Sampling Distribution
6.9 Estimation Bias
6.10 Estimation Variance
6.11 Mean Squared Error
6.12 Best Unbiased Estimator
6.13 Estimation of Variance
6.14 Standard Error
6.15 Multivariate Means
6.16 Order Statistics
6.17 Higher Moments of Sample Mean
6.18 Normal Sampling Model
6.19 Normal Residuals
6.20 Normal Variance Estimation
6.21 Studentized Ratio
6.22 Multivariate Normal Sampling
6.23 Exercises
7. Law of Large Numbers
7.1 Introduction
7.2 Asymptotic Limits
7.3 Convergence in Probability
7.4 Chebyshev’s Inequality
7.5 Weak Law of Large Numbers
7.6 Counterexamples
7.7 Examples
7.8 Illustrating Chebyshev’s Inequality
7.9 Vector-Valued Moments
7.10 Continuous Mapping Theorem
7.11 Examples
7.12 Uniformity Over Distributions
7.13 Almost Sure Convergence and the Strong Law
7.14 Technical Proofs
7.15 Exercises
8. Central Limit Theory
8.1 Introduction
8.2 Convergence in Distribution
8.3 Sample Mean
8.4 A Moment Investigation
8.5 Convergence of the Moment Generating Function
8.6 Central Limit Theorem
8.7 Applying the Central Limit Theorem
8.8 Multivariate Central Limit Theorem
8.9 Delta Method
8.10 Examples
8.11 Asymptotic Distribution for Plug-In Estimator
8.12 Covariance Matrix Estimation
8.13 t-Ratios
8.14 Stochastic Order Symbols
8.15 Technical Proofs
8.16 Exercises
9. Advanced Asymptotic Theory
9.1 Introduction
9.2 Heterogeneous Central Limit Theory
9.3 Multivariate Heterogeneous Central Limit Theory
9.4 Uniform Central Limit Theory
9.5 Uniform Integrability
9.6 Uniform Stochastic Bounds
9.7 Convergence of Moments
9.8 Edgeworth Expansion for the Sample Mean
9.9 Edgeworth Expansion for Smooth Function Model
9.10 Cornish-Fisher Expansions
9.11 Technical Proofs
10. Maximum Likelihood Estimation
10.1 Introduction
10.2 Parametric Model
10.3 Likelihood
10.4 Likelihood Analog Principle
10.5 Invariance Property
10.6 Examples
10.7 Score, Hessian, and Information
10.8 Examples
10.9 Cramér-Rao Lower Bound
10.10 Examples
10.11 Cramér-Rao Bound for Functions of Parameters
10.12 Consistent Estimation
10.13 Asymptotic Normality
10.14 Asymptotic Cramér-Rao Efficiency
10.15 Variance Estimation
10.16 Kullback-Leibler Divergence
10.17 Approximating Models
10.18 Distribution of the MLE under Misspecification
10.19 Variance Estimation under Misspecification
10.20 Technical Proofs
10.21 Exercises
11. Method of Moments
11.1 Introduction
11.2 Multivariate Means
11.3 Moments
11.4 Smooth Functions
11.5 Central Moments
11.6 Best Unbiased Estimation
11.7 Parametric Models
11.8 Examples of Parametric Models
11.9 Moment Equations
11.10 Asymptotic Distribution for Moment Equations
11.11 Example: Euler Equation
11.12 Empirical Distribution Function
11.13 Sample Quantiles
11.14 Robust Variance Estimation
11.15 Technical Proofs
11.16 Exercises
12. Numerical Optimization
12.1 Introduction
12.2 Numerical Function Evaluation and Differentiation
12.3 Root Finding
12.4 Minimization in One Dimension
12.5 Failures of Minimization
12.6 Minimization in Multiple Dimensions
12.7 Constrained Optimization
12.8 Nested Minimization
12.9 Tips and Tricks
12.10 Exercises
13. Hypothesis Testing
13.1 Introduction
13.2 Hypotheses
13.3 Acceptance and Rejection
13.4 Type I and Type II Errors
13.5 One-Sided Tests
13.6 Two-Sided Tests
13.7 What Does “Accept H0” Mean about H0?
13.8 t Testwith Normal Sampling
13.9 Asymptotic t Test
13.10 Likelihood Ratio Test for Simple Hypotheses
13.11 Neyman-Pearson Lemma
13.12 Likelihood Ratio Test against Composite Alternatives
13.13 Likelihood Ratio and t Tests
13.14 Statistical Significance
13.15 p-Value
13.16 Composite Null Hypothesis
13.17 Asymptotic Uniformity
13.18 Summary
13.19 Exercises
14. Confidence Intervals
14.1 Introduction
14.2 Definitions
14.3 Simple Confidence Intervals
14.4 Confidence Intervals for the Sample Mean under Normal Sampling
14.5 Confidence Intervals for the Sample Mean under Non-Normal Sampling
14.6 Confidence Intervals for Estimated Parameters
14.7 Confidence Interval for theVariance
14.8 Confidence Intervals by Test Inversion
14.9 Use of Confidence Intervals
14.10 Uniform Confidence Intervals
14.11 Exercises
15. Shrinkage Estimation
15.1 Introduction
15.2 Mean Squared Error
15.3 Shrinkage
15.4 James-Stein Shrinkage Estimator
15.5 Numerical Calculation
15.6 Interpretation of the Stein Effect
15.7 Positive-Part Estimator
15.8 Summary
15.9 Technical Proofs
15.10 Exercises
16. Bayesian Methods
16.1 Introduction
16.2 Bayesian Probability Model
16.3 Posterior Density
16.4 Bayesian Estimation
16.5 Parametric Priors
16.6 Normal-Gamma Distribution
16.7 Conjugate Prior
16.8 Bernoulli Sampling
16.9 Normal Sampling
16.10 Credible Sets
16.11 Bayesian Hypothesis Testing
16.12 Sampling Properties in the Normal Model
16.13 Asymptotic Distribution
16.14 Technical Proofs
16.15 Exercises
17. Nonparametric Density Estimation
17.1 Introduction
17.2 Histogram Density Estimation
17.3 Kernel Density Estimator
17.4 Bias of Density Estimator
17.5 Variance of Density Estimator
17.6 Variance Estimation and Standard Errors
17.7 Integrated Mean Squared Error of Density Estimator
17.8 Optimal Kernel
17.9 Reference Bandwidth
17.10 Sheather-Jones Bandwidth
17.11 Recommendations for Bandwidth Selection
17.12 Practical Issues in Density Estimation
17.13 Computation
17.14 Asymptotic Distribution
17.15 Undersmoothing
17.16 Technical Proofs
17.17 Exercises
18. Empirical Process Theory
18.1 Introduction
18.2 Framework
18.3 Glivenko-Cantelli Theorem
18.4 Packing, Covering, and Bracketing Numbers
18.5 Uniform Law of Large Numbers
18.6 Functional Central Limit Theory
18.7 Conditions for Asymptotic Equicontinuity
18.8 Donsker’s Theorem
18.9 Technical Proofs
18.10 Exercises
Appendix: Mathematics Reference
A.1 Limits
A.2 Series
A.3 Factorials
A.4 Exponentials
A.5 Logarithms
A.6 Differentiation
A.7 Mean Value Theorem
A.8 Integration
A.9 Gaussian Integral
A.10 Gamma Function
A.11 Matrix Algebra
References
Index