Probability: Large Deviations & Concentration Inequalities
Probability: Large Deviations & Concentration Inequalities
💡 Key Takeaways
- Grasp the core idea of large deviations: probabilities of extreme sums decay exponentially with sample size.
- Learn the main concentration inequalities (Hoeffding, Bernstein, Azuma) and how to choose among them.
- Understand the rate function (I) and Cramér's theorem for simple i.i.d. sequences.
- Apply large deviation concepts to bound tail probabilities and assess risk in probabilistic models.
❓ Frequently Asked Questions
What is a concentration inequality?
A bound on the probability that a random variable (or its average) deviates from a typical value like its mean. These bounds often imply exponential decay of tail probabilities.
What is a large deviations principle (LDP)?
An LDP describes the exponential decay rate of unlikely events as the sample size grows: P(X_n in A) ~ exp(-n I(A)). The function I(x) is the rate function that measures the 'cost' of observing x.
What is a rate function and what makes it 'good'?
A rate function I: R -> [0, Infinity) that is lower semicontinuous. It vanishes at the typical value, and a good rate function has compact level sets {x : I(x) <= alpha} for all alpha.
What is Hoeffding's inequality (a common concentration bound)?
If X_i are independent with a_i <= X_i <= b_i and S_n = sum X_i, then P(S_n - E[S_n] >= t) <= exp(-2 t^2 / sum (b_i - a_i)^2). A two-sided bound also holds: P(|S_n - E[S_n]| >= t) <= 2 exp(-2 t^2 / sum (b_i - a_i)^2).
How do large deviations relate to the Central Limit Theorem?
The CLT describes typical fluctuations of order sqrt(n) with a Gaussian limit, while large deviations concern rarer, larger fluctuations and their probabilities decay exponentially with n, captured by the rate function.