Sub-Gaussian distribution

Short description: Continuous probability distribution

In probability theory, a sub-Gaussian distribution, the distribution of a sub-Gaussian random variable, is a probability distribution with strong tail decay. More specifically, the tails of a sub-Gaussian distribution are dominated by (i.e. decay at least as fast as) the tails of a Gaussian. This property gives sub-Gaussian distributions their name.

Formally, the probability distribution of a random variable $X$ is called sub-Gaussian if there is a positive constant C such that for every $t\ge 0$,

$\mathrm{P}(|X|\ge t)\le 2\exp (-t^2/C^2)$.

Alternatively, a random variable is considered sub-Gaussian if its distribution function is upper bounded (up to a constant) by the distribution function of a Gaussian. Specifically, we say that $X$ is sub-Gaussian if for all $s\ge 0$ we have that:

$P(|X|\ge s)\le cP(|Z|\ge s),$

where $c\ge 0$ is constant and $Z$ is a mean zero Gaussian random variable.

Definitions

The sub-Gaussian norm of $X$, denoted as $\Vert X{\Vert }_{gauss}$, is defined by$$\Vert X{\Vert }_{gauss}=\inf \{c>0:\mathrm{E}[\exp \left(\frac{X^2}{c^2}\right)]\le 2\},$$which is the Orlicz norm of $X$ generated by the Orlicz function $\mathrm{\Phi }(u)=e^{u^2}-1.$ By condition $(2)$ below, sub-Gaussian random variables can be characterized as those random variables with finite sub-Gaussian norm.

Sub-Gaussian properties

Let $X$ be a random variable. The following conditions are equivalent:

1. $\mathrm{P}(|X|\ge t)\le 2\exp (-t^2/K_1^2)$ for all $t\ge 0$, where $K_1$ is a positive constant;
2. $\mathrm{E}[\exp (X^2/K_2^2)]\le 2$, where $K_2$ is a positive constant;
3. $\mathrm{E}|X{|}^p\le 2K_3^p\mathrm{\Gamma }(\frac{p}{2}+1)$ for all $p\ge 1$, where $K_3$ is a positive constant.

Proof. $(1)⟹(3)$ By the layer cake representation,$$\begin{array}{rl}\mathrm{E}|X{|}^p& =\underset{0}{\overset{\mathrm{\infty }}{\int }}\mathrm{P}(|X{|}^p\ge t)dt\\ & =\underset{0}{\overset{\mathrm{\infty }}{\int }}p{t}^{p-1}\mathrm{P}(|X|\ge t)dt\\ & \le 2\underset{0}{\overset{\mathrm{\infty }}{\int }}p{t}^{p-1}\exp (-\frac{t^2}{K_1^2})dt\\ \end{array}$$After a change of variables $u=t^2/K_1^2$, we find that$$\begin{array}{rl}\mathrm{E}|X{|}^p& \le 2K_1^p\frac{p}{2}\underset{0}{\overset{\mathrm{\infty }}{\int }}{u}^{\frac{p}{2}-1}{e}^{-u}du\\ & =2K_1^p\frac{p}{2}\mathrm{\Gamma }\left(\frac{p}{2}\right)\\ & =2K_1^p\mathrm{\Gamma }(\frac{p}{2}+1).\end{array}$$

$(3)⟹(2)$ Using the Taylor series for $e^x$:$$e^x=1+\sum _{p=1}^{\mathrm{\infty }}\frac{x^p}{p!},$$ we obtain that$$\begin{array}{rl}\mathrm{E}[\exp (\lambda X^2)]& =1+\sum _{p=1}^{\mathrm{\infty }}\frac{{\lambda }^p\mathrm{E}[X^{2p}]}{p!}\\ & \le 1+\sum _{p=1}^{\mathrm{\infty }}\frac{2{\lambda }^p{K}_3^{2p}\mathrm{\Gamma }(p+1)}{p!}\\ & =1+2\sum _{p=1}^{\mathrm{\infty }}{\lambda }^p{K}_3^{2p}\\ & =2\sum _{p=0}^{\mathrm{\infty }}{\lambda }^p{K}_3^{2p}-1\\ & =\frac{2}{1-\lambda K_3^2}-1\text{for }\lambda K_3^2<1,\end{array}$$which is less than or equal to $2$ for $\lambda \le \frac{1}{3K_3^2}$. Take $K_2\ge 3^{\frac{1}{2}}{K}_3$, then$$\mathrm{E}[\exp (X^2/K_2^2)]\le 2.$$

$(2)⟹(1)$ By Markov's inequality,$$\mathrm{P}(|X|\ge t)=\mathrm{P}(\exp \left(\frac{X^2}{K_2^2}\right)\ge \exp \left(\frac{t^2}{K_2^2}\right))\le \frac{\mathrm{E}[\exp (X^2/K_2^2)]}{\exp \left(\frac{t^2}{K_2^2}\right)}\le 2\exp (-\frac{t^2}{K_2^2}).$$

More equivalent definitions

The following properties are equivalent:

• The distribution of $X$ is sub-Gaussian.
• Laplace transform condition: for some B, b > 0, $\mathrm{E}{e}^{\lambda (X-\mathrm{E}[X])}\le B{e}^{{\lambda }^{2}b}$ holds for all $\lambda$.
• Moment condition: for some K > 0, $\mathrm{E}|X{|}^p\le K^p{p}^{p/2}$ for all $p\ge 1$.
• Moment generating function condition: for some $L>0$, $\mathrm{E}[\exp ({\lambda }^2{X}^2)]\le \exp (L^2{\lambda }^2)$ for all $\lambda$ such that $|\lambda |\le \frac{1}{L}$.
• Union bound condition: for some c > 0, $\mathrm{E}[\max \{|X_1-\mathrm{E}[X]|,\dots ,|X_n-\mathrm{E}[X]|\}]\le c\sqrt{\log n}$ for all n > c, where $X_1,\dots ,X_n$ are i.i.d copies of X.

Examples

A standard normal random variable $X\sim N(0,1)$ is a sub-Gaussian random variable.

Let $X$ be a random variable with symmetric Bernoulli distribution (or Rademacher distribution). That is, $X$ takes values $-1$ and $1$ with probabilities $1/2$ each. Since $X^2=1$, it follows that $$\Vert X{\Vert }_{gauss}=\inf \{c>0:\mathrm{E}[\exp \left(\frac{X^2}{c^2}\right)]\le 2\}=\inf \{c>0:\mathrm{E}[\exp \left(\frac{1}{c^2}\right)]\le 2\}=\frac{1}{\sqrt{\ln 2}},$$and hence $X$ is a sub-Gaussian random variable.

Maximum of Sub-Gaussian Random Variables

Consider a finite collection of subgaussian random variables, X₁, ..., X_n, with corresponding subgaussian parameters $\sigma$. The random variable M_n = max(X₁, ..., X_n) represents the maximum of this collection. The expectation $M_n$ can be bounded above by $\sqrt{2{\sigma }^2\log n}$. Note that no independence assumption is needed to form this bound.

See also

• Platykurtic distribution

References

• Kahane, J.P. (1960). "Propriétés locales des fonctions à séries de Fourier aléatoires". Studia Mathematica 19: 1–25. doi:10.4064/sm-19-1-1-25. 
• Buldygin, V.V.; Kozachenko, Yu.V. (1980). "Sub-Gaussian random variables". Ukrainian Mathematical Journal 32 (6): 483–489. doi:10.1007/BF01087176. 
• Ledoux, Michel; Talagrand, Michel (1991). Probability in Banach Spaces. Springer-Verlag. 
• Stromberg, K.R. (1994). Probability for Analysts. Chapman & Hall/CRC. 
• Litvak, A.E.; Pajor, A.; Rudelson, M.; Tomczak-Jaegermann, N. (2005). "Smallest singular value of random matrices and geometry of random polytopes". Advances in Mathematics 195 (2): 491–523. doi:10.1016/j.aim.2004.08.004. http://www.math.ualberta.ca/~alexandr/OrganizedPapers/lprtlastlast.pdf. 
• Rudelson, Mark; Vershynin, Roman (2010). "Non-asymptotic theory of random matrices: extreme singular values". pp. 1576–1602. doi:10.1142/9789814324359_0111. 
• Rivasplata, O. (2012). "Subgaussian random variables: An expository note". Unpublished. http://www.stat.cmu.edu/~arinaldo/36788/subgaussians.pdf. 
• Vershynin, R. (2018). "High-dimensional probability: An introduction with applications in data science" (PDF). Volume 47 of Cambridge Series in Statistical and Probabilistic Mathematics. Cambridge University Press, Cambridge.
• Zajkowskim, K. (2020). "On norms in some class of exponential type Orlicz spaces of random variables". Positivity. An International Mathematics Journal Devoted to Theory and Applications of Positivity. 24(5): 1231--1240. arXiv:1709.02970. doi.org/10.1007/s11117-019-00729-6.