Normal-inverse-Wishart distribution

Short description: Multivariate parameter family of continuous probability distributions

normal-inverse-Wishart

Notation	$(\mathit{\mu },\mathrm{\Sigma })\sim \mathrm{N}\mathrm{I}\mathrm{W}({\mathit{\mu }}_0,\lambda ,\mathrm{\Psi },\nu )$
Parameters	${\mathit{\mu }}_0\in {\mathbb{ℝ}}^D$ location (vector of real) $\lambda >0$ (real) $\mathrm{\Psi }\in {\mathbb{ℝ}}^{D\times D}$ inverse scale matrix (pos. def.) $\nu >D-1$ (real)
Support	$\mathit{\mu }\in {\mathbb{ℝ}}^D;\mathrm{\Sigma }\in {\mathbb{ℝ}}^{D\times D}$ covariance matrix (pos. def.)
PDF	$f(\mathit{\mu },\mathrm{\Sigma }|{\mathit{\mu }}_0,\lambda ,\mathrm{\Psi },\nu )={𝒩}(\mathit{\mu }|{\mathit{\mu }}_0,\frac{1}{\lambda }\mathrm{\Sigma }){{𝒲}}^{-1}(\mathrm{\Sigma }|\mathrm{\Psi },\nu )$

In probability theory and statistics, the normal-inverse-Wishart distribution (or Gaussian-inverse-Wishart distribution) is a multivariate four-parameter family of continuous probability distributions. It is the conjugate prior of a multivariate normal distribution with unknown mean and covariance matrix (the inverse of the precision matrix).

Definition

Suppose

$\mathit{\mu }|{\mathit{\mu }}_0,\lambda ,\mathrm{\Sigma }\sim {𝒩}\left(\mathit{\mu }\right|{\mathit{\mu }}_0,\frac{1}{\lambda }\mathrm{\Sigma })$

has a multivariate normal distribution with mean ${\mathit{\mu }}_0$ and covariance matrix $\frac{1}{\lambda }\mathrm{\Sigma }$, where

$\mathrm{\Sigma }|\mathrm{\Psi },\nu \sim {{𝒲}}^{-1}(\mathrm{\Sigma }|\mathrm{\Psi },\nu )$

has an inverse Wishart distribution. Then $(\mathit{\mu },\mathrm{\Sigma })$ has a normal-inverse-Wishart distribution, denoted as

$(\mathit{\mu },\mathrm{\Sigma })\sim \mathrm{N}\mathrm{I}\mathrm{W}({\mathit{\mu }}_0,\lambda ,\mathrm{\Psi },\nu ).$

Characterization

Probability density function

$f(\mathit{\mu },\mathrm{\Sigma }|{\mathit{\mu }}_0,\lambda ,\mathrm{\Psi },\nu )={𝒩}\left(\mathit{\mu }\right|{\mathit{\mu }}_0,\frac{1}{\lambda }\mathrm{\Sigma }){{𝒲}}^{-1}(\mathrm{\Sigma }|\mathrm{\Psi },\nu )$

The full version of the PDF is as follows:

$f(\mathit{\mu },\mathrm{\Sigma }|{\mathit{\mu }}_0,\lambda ,\mathrm{\Psi },\nu )=\frac{{\lambda }^{D/2}|\mathrm{\Psi }{|}^{\nu /2}|\mathrm{\Sigma }{|}^{-\frac{\nu +D+2}{2}}}{(2\pi {)}^{D/2}{2}^{\frac{\nu D}{2}}{\mathrm{\Gamma }}_D(\frac{\nu }{2})}\text{exp}\{-\frac{1}{2}Tr({\mathrm{\Psi }\mathrm{\Sigma }}^{-1})-\frac{\lambda }{2}(\mathit{\mu }-{\mathit{\mu }}_0{)}^T{\mathrm{\Sigma }}^{-1}(\mathit{\mu }-{\mathit{\mu }}_0)\}$

Here ${\mathrm{\Gamma }}_D[\cdot ]$ is the multivariate gamma function and $Tr(\mathrm{\Psi })$ is the Trace of the given matrix.

Properties

Marginal distributions

By construction, the marginal distribution over $\mathrm{\Sigma }$ is an inverse Wishart distribution, and the conditional distribution over $\mathit{\mu }$ given $\mathrm{\Sigma }$ is a multivariate normal distribution. The marginal distribution over $\mathit{\mu }$ is a multivariate t-distribution.

Posterior distribution of the parameters

Suppose the sampling density is a multivariate normal distribution

${\mathit{y}}_{\mathit{i}}|\mathit{\mu },\mathrm{\Sigma }\sim {{𝒩}}_p(\mathit{\mu },\mathrm{\Sigma })$

where $\mathit{y}$ is an $n\times p$ matrix and ${\mathit{y}}_{\mathit{i}}$ (of length $p$) is row $i$ of the matrix .

With the mean and covariance matrix of the sampling distribution is unknown, we can place a Normal-Inverse-Wishart prior on the mean and covariance parameters jointly

$(\mathit{\mu },\mathrm{\Sigma })\sim \mathrm{N}\mathrm{I}\mathrm{W}({\mathit{\mu }}_0,\lambda ,\mathrm{\Psi },\nu ).$

The resulting posterior distribution for the mean and covariance matrix will also be a Normal-Inverse-Wishart

$(\mathit{\mu },\mathrm{\Sigma }|y)\sim \mathrm{N}\mathrm{I}\mathrm{W}({\mathit{\mu }}_n,{\lambda }_n,{\mathrm{\Psi }}_n,{\nu }_n),$

where

${\mathit{\mu }}_n=\frac{\lambda {\mathit{\mu }}_0+n\overline{\mathit{y}}}{\lambda +n}$

${\lambda }_n=\lambda +n$

${\nu }_n=\nu +n$

${\mathrm{\Psi }}_n=\mathrm{\Psi }\mathit{+}\mathit{S}+\frac{\lambda n}{\lambda +n}(\overline{\mathit{y}}\mathit{-}{\mathit{\mu }}_{\mathit{0}})(\overline{\mathit{y}}\mathit{-}{\mathit{\mu }}_{\mathit{0}}{)}^T\mathrm{w}\mathrm{i}\mathrm{t}\mathrm{h}\mathit{S}=\sum _{i=1}^n({\mathit{y}}_{\mathit{i}}\mathit{-}\overline{\mathit{y}})({\mathit{y}}_{\mathit{i}}\mathit{-}\overline{\mathit{y}}{)}^T$.

To sample from the joint posterior of $(\mathit{\mu },\mathrm{\Sigma })$, one simply draws samples from $\mathrm{\Sigma }|\mathit{y}\sim {{𝒲}}^{-1}({\mathrm{\Psi }}_n,{\nu }_n)$, then draw $\mathit{\mu }|\mathrm{\Sigma },\mathit{y}\sim {{𝒩}}_p({\mathit{\mu }}_n,\mathrm{\Sigma }/{\lambda }_n)$. To draw from the posterior predictive of a new observation, draw $\tilde{\mathit{y}}|\mathit{\mu },\mathrm{\Sigma },\mathit{y}\sim {{𝒩}}_p(\mathit{\mu },\mathrm{\Sigma })$ , given the already drawn values of $\mathit{\mu }$ and $\mathrm{\Sigma }$.

Generating normal-inverse-Wishart random variates

Generation of random variates is straightforward:

1. Sample $\mathrm{\Sigma }$ from an inverse Wishart distribution with parameters $\mathrm{\Psi }$ and $\nu$
2. Sample $\mathit{\mu }$ from a multivariate normal distribution with mean ${\mathit{\mu }}_0$ and variance $\frac{1}{\lambda }\mathrm{\Sigma }$

Related distributions

• The normal-Wishart distribution is essentially the same distribution parameterized by precision rather than variance. If $(\mathit{\mu },\mathrm{\Sigma })\sim \mathrm{N}\mathrm{I}\mathrm{W}({\mathit{\mu }}_0,\lambda ,\mathrm{\Psi },\nu )$ then $(\mathit{\mu },{\mathrm{\Sigma }}^{-1})\sim \mathrm{N}\mathrm{W}({\mathit{\mu }}_0,\lambda ,{\mathrm{\Psi }}^{-1},\nu )$ .
• The normal-inverse-gamma distribution is the one-dimensional equivalent.
• The multivariate normal distribution and inverse Wishart distribution are the component distributions out of which this distribution is made.

References

• Bishop, Christopher M. (2006). Pattern Recognition and Machine Learning. Springer Science+Business Media.
• Murphy, Kevin P. (2007). "Conjugate Bayesian analysis of the Gaussian distribution." [2]