Quadratic form (statistics)

In multivariate statistics, if $\epsilon$ is a vector of $n$ random variables, and $\mathrm{\Lambda }$ is an $n$-dimensional symmetric matrix, then the scalar quantity ${\epsilon }^T\mathrm{\Lambda }\epsilon$ is known as a quadratic form in $\epsilon$.

Expectation

It can be shown that

$\mathrm{E}\left[{\epsilon }^T\mathrm{\Lambda }\epsilon \right]=\mathrm{tr}\left[\mathrm{\Lambda }\mathrm{\Sigma }\right]+{\mu }^T\mathrm{\Lambda }\mu$

where $\mu$ and $\mathrm{\Sigma }$ are the expected value and variance-covariance matrix of $\epsilon$, respectively, and tr denotes the trace of a matrix. This result only depends on the existence of $\mu$ and $\mathrm{\Sigma }$; in particular, normality of $\epsilon$ is not required.

A book treatment of the topic of quadratic forms in random variables is that of Mathai and Provost.

Proof

Since the quadratic form is a scalar quantity, ${\epsilon }^T\mathrm{\Lambda }\epsilon =\mathrm{tr}({\epsilon }^T\mathrm{\Lambda }\epsilon )$.

Next, by the cyclic property of the trace operator,

$\mathrm{E}[\mathrm{tr}({\epsilon }^T\mathrm{\Lambda }\epsilon )]=\mathrm{E}[\mathrm{tr}(\mathrm{\Lambda }\epsilon {\epsilon }^T)].$

Since the trace operator is a linear combination of the components of the matrix, it therefore follows from the linearity of the expectation operator that

$\mathrm{E}[\mathrm{tr}(\mathrm{\Lambda }\epsilon {\epsilon }^T)]=\mathrm{tr}(\mathrm{\Lambda }\mathrm{E}(\epsilon {\epsilon }^T)).$

A standard property of variances then tells us that this is

$\mathrm{tr}(\mathrm{\Lambda }(\mathrm{\Sigma }+\mu {\mu }^T)).$

Applying the cyclic property of the trace operator again, we get

$\mathrm{tr}(\mathrm{\Lambda }\mathrm{\Sigma })+\mathrm{tr}(\mathrm{\Lambda }\mu {\mu }^T)=\mathrm{tr}(\mathrm{\Lambda }\mathrm{\Sigma })+\mathrm{tr}({\mu }^T\mathrm{\Lambda }\mu )=\mathrm{tr}(\mathrm{\Lambda }\mathrm{\Sigma })+{\mu }^T\mathrm{\Lambda }\mu .$

Variance in the Gaussian case

In general, the variance of a quadratic form depends greatly on the distribution of $\epsilon$. However, if $\epsilon$ does follow a multivariate normal distribution, the variance of the quadratic form becomes particularly tractable. Assume for the moment that $\mathrm{\Lambda }$ is a symmetric matrix. Then,

$\mathrm{var}\left[{\epsilon }^T\mathrm{\Lambda }\epsilon \right]=2\mathrm{tr}\left[\mathrm{\Lambda }\mathrm{\Sigma }\mathrm{\Lambda }\mathrm{\Sigma }\right]+4{\mu }^T\mathrm{\Lambda }\mathrm{\Sigma }\mathrm{\Lambda }\mu$.

In fact, this can be generalized to find the covariance between two quadratic forms on the same $\epsilon$ (once again, ${\mathrm{\Lambda }}_1$ and ${\mathrm{\Lambda }}_2$ must both be symmetric):

$\mathrm{cov}[{\epsilon }^T{\mathrm{\Lambda }}_1\epsilon ,{\epsilon }^T{\mathrm{\Lambda }}_2\epsilon ]=2\mathrm{tr}\left[{\mathrm{\Lambda }}_1\mathrm{\Sigma }{\mathrm{\Lambda }}_2\mathrm{\Sigma }\right]+4{\mu }^T{\mathrm{\Lambda }}_1\mathrm{\Sigma }{\mathrm{\Lambda }}_2\mu$.

In addition, a quadratic form such as this follows a generalized chi-squared distribution.

Computing the variance in the non-symmetric case

The case for general $\mathrm{\Lambda }$ can be derived by noting that

${\epsilon }^T{\mathrm{\Lambda }}^T\epsilon ={\epsilon }^T\mathrm{\Lambda }\epsilon$

so

${\epsilon }^T\tilde{\mathrm{\Lambda }}\epsilon ={\epsilon }^T(\mathrm{\Lambda }+{\mathrm{\Lambda }}^T)\epsilon /2$

is a quadratic form in the symmetric matrix $\tilde{\mathrm{\Lambda }}=(\mathrm{\Lambda }+{\mathrm{\Lambda }}^T)/2$, so the mean and variance expressions are the same, provided $\mathrm{\Lambda }$ is replaced by $\tilde{\mathrm{\Lambda }}$ therein.

Examples of quadratic forms

In the setting where one has a set of observations $y$ and an operator matrix $H$, then the residual sum of squares can be written as a quadratic form in $y$:

$\text{<mrow data-mjx-texclass="ORD">RSS</mrow>}=y^T(I-H{)}^T(I-H)y.$

For procedures where the matrix $H$ is symmetric and idempotent, and the errors are Gaussian with covariance matrix ${\sigma }^{2}I$, $\text{<mrow data-mjx-texclass="ORD">RSS</mrow>}/{\sigma }^2$ has a chi-squared distribution with $k$ degrees of freedom and noncentrality parameter $\lambda$, where

$k=\mathrm{tr}[(I-H{)}^T(I-H)]$

$\lambda ={\mu }^T(I-H{)}^T(I-H)\mu /2$

may be found by matching the first two central moments of a noncentral chi-squared random variable to the expressions given in the first two sections. If $Hy$ estimates $\mu$ with no bias, then the noncentrality $\lambda$ is zero and $\text{<mrow data-mjx-texclass="ORD">RSS</mrow>}/{\sigma }^2$ follows a central chi-squared distribution.

See also

• Quadratic form
• Covariance matrix
• Matrix representation of conic sections