View source for Sample mean and covariance

{{Short description|Statistics computed from a sample of data}}

The '''sample mean''' ('''sample average''') or '''empirical mean''' ('''empirical average'''), and the '''sample covariance''' or '''empirical covariance''' are [[Statistic|statistic]]s computed from a [[Sample (statistics)|sample]] of data on one or more random variables.

The sample mean is the [[Average|average]] value (or [[Mean|mean value]]) of a [[Sample (statistics)|sample]] of numbers taken from a larger [[Statistical population|population]] of numbers, where "population" indicates not number of people but the entirety of relevant data, whether collected or not. A sample of 40 companies' sales from the {{wipe|Fortune 500}} might be used for convenience instead of looking at the population, all 500 companies' sales. The sample mean is used as an [[Estimator|estimator]] for the population mean, the average value in the entire population, where the estimate is more likely to be close to the population mean if the sample is large and representative. The reliability of the sample mean is estimated using the [[Standard error|standard error]], which in turn is calculated using the [[Variance|variance]] of the sample. If the sample is random, the standard error falls with the size of the sample and the sample mean's distribution approaches the normal distribution as the sample size increases.

The term "sample mean" can also be used to refer to a [[Vector (mathematics and physics)|vector]] of average values when the statistician is looking at the values of several variables in the sample, e.g. the sales, profits, and employees of a sample of Fortune 500 companies. In this case, there is not just a sample variance for each variable but a sample [[Covariance matrix|variance-covariance matrix]] (or simply ''covariance matrix'') showing also the relationship between each pair of variables. This would be a 3×3 matrix when 3 variables are being considered. The sample covariance is useful in judging the reliability of the sample means as estimators and is also useful as an estimate of the population covariance matrix.

Due to their ease of calculation and other desirable characteristics, the sample mean and sample covariance are widely used in statistics to  represent the [[Earth:Location|location]] and [[Statistical dispersion|dispersion]] of the [[Probability distribution|distribution]] of values in the sample, and to estimate the values for the population.

==Definition of the sample mean==
{{Further|Arithmetic mean}}

The sample mean is the average of the values of a variable in a sample, which is the sum of those values divided by the number of values. Using mathematical notation, if a sample of ''N'' observations on variable ''X'' is taken from the population, the sample mean is:

:<math> \bar{X}=\frac{1}{N}\sum_{i=1}^{N}X_{i}.</math>

Under this definition, if the sample (1, 4, 1) is taken from the population (1,1,3,4,0,2,1,0), then the sample mean is <math>\bar{x} = (1+4+1)/3 = 2</math>, as compared to the population mean of <math>\mu = (1+1+3+4+0+2+1+0) /8 = 12/8 = 1.5</math>. Even if a sample is random, it is rarely perfectly representative, and other samples would have other sample means even if the samples were all from the same population. The sample (2, 1, 0), for example, would have a sample mean of 1.

If the statistician is interested in  ''K'' variables rather than one, each observation having a value for each of those ''K'' variables, the overall sample mean consists of ''K'' sample means for individual variables. Let <math>x_{ij}</math> be the ''i''<sup>th</sup> independently drawn observation (''i''=1,...,''N'') on the ''j''<sup>th</sup> random variable (''j''=1,...,''K''). These observations can be arranged into ''N''
column vectors, each with ''K'' entries, with the ''K''×1 column vector giving the ''i''-th observations of all variables being denoted <math>\mathbf{x}_i</math> (''i''=1,...,''N'').

The '''sample mean vector''' <math>\mathbf{\bar{x}}</math> is a column vector whose ''j''-th element <math>\bar{x}_{j}</math> is the average value of the ''N'' observations of the ''j''<sup>th</sup> variable:

:<math> \bar{x}_{j}=\frac{1}{N} \sum_{i=1}^{N} x_{ij},\quad j=1,\ldots,K. </math>

Thus, the sample mean vector contains the average of the observations for each variable, and is written

:<math> \mathbf{\bar{x}}=\frac{1}{N}\sum_{i=1}^{N}\mathbf{x}_i = \begin{bmatrix}
\bar{x}_1 \\
\vdots \\
\bar{x}_j \\ 
\vdots \\
\bar{x}_K
\end{bmatrix} </math>

==Definition of sample covariance==

The '''sample covariance matrix''' is a ''K''-by-''K'' [[Matrix (mathematics)|matrix]] <math>\textstyle \mathbf{Q}=\left[  q_{jk}\right]  </math> with entries
:<math> q_{jk}=\frac{1}{N-1}\sum_{i=1}^{N}\left(  x_{ij}-\bar{x}_j \right)  \left( x_{ik}-\bar{x}_k \right), </math>

where <math>q_{jk}</math> is an estimate of the [[Covariance|covariance]] between the {{math|j}}<sup>th</sup>
variable and the {{math|k}}<sup>th</sup> variable of the population underlying the data.
In terms of the observation vectors, the sample covariance is
:<math>\mathbf{Q} = {1 \over {N-1}}\sum_{i=1}^N (\mathbf{x}_i.-\mathbf{\bar{x}}) (\mathbf{x}_i.-\mathbf{\bar{x}})^\mathrm{T},</math>

Alternatively, arranging the observation vectors as the columns of a matrix, so that
:<math>\mathbf{F} = \begin{bmatrix}\mathbf{x}_1 & \mathbf{x}_2 & \dots & \mathbf{x}_N \end{bmatrix}</math>,
which is a matrix of ''K'' rows and ''N'' columns.
Here, the sample covariance matrix can be computed as
:<math>\mathbf{Q} = \frac{1}{N-1}( \mathbf{F} - \mathbf{\bar{x}} \,\mathbf{1}_N^\mathrm{T} ) ( \mathbf{F} - \mathbf{\bar{x}} \,\mathbf{1}_N^\mathrm{T} )^\mathrm{T}</math>,
where <math>\mathbf{1}_N</math> is an ''N'' by {{math|1}}  vector of ones. 
If the observations are arranged as rows instead of columns, so <math>\mathbf{\bar{x}}</math> is now a 1×''K'' row vector and <math>\mathbf{M}=\mathbf{F}^\mathrm{T}</math> is an ''N''×''K'' matrix whose column ''j'' is the vector of ''N'' observations on variable ''j'', then applying transposes 
in the appropriate places yields
:<math>\mathbf{Q} = \frac{1}{N-1}( \mathbf{M} -  \mathbf{1}_N \mathbf{\bar{x}} )^\mathrm{T} ( \mathbf{M} - \mathbf{1}_N \mathbf{\bar{x}} ).</math>
Like covariance matrices for [[Multivariate random variable|random vector]], sample covariance matrices are positive semi-definite. To prove it, note that for any matrix <math>\mathbf{A}</math> the matrix <math>\mathbf{A}^T\mathbf{A}</math> is positive semi-definite. Furthermore, a covariance matrix is positive definite if and only if the rank of the <math>\mathbf{x}_i.-\mathbf{\bar{x}}</math> vectors is K.

==Unbiasedness==
The sample mean and the sample covariance matrix are [[Bias of an estimator|unbiased estimates]] of the [[Mean|mean]] and the [[Covariance matrix|covariance matrix]] of the random vector <math>\textstyle \mathbf{X}</math>, a row vector whose ''j''<sup>th</sup> element (''j = 1, ..., K'') is one of the random variables.<ref name="JohnsonWichern2007">{{cite book|author1=Richard Arnold Johnson|author2=Dean W. Wichern|title=Applied Multivariate Statistical Analysis|url=https://books.google.com/books?id=gFWcQgAACAAJ|access-date=10 August 2012|year=2007|publisher=Pearson Prentice Hall|isbn=978-0-13-187715-3}}</ref> The sample covariance matrix has <math>\textstyle N-1</math> in the denominator rather than <math>\textstyle N</math> due to a variant of [[Bessel's correction]]:  In short, the sample covariance relies on the difference between each observation and the sample mean, but the sample mean is slightly correlated with each observation since it is defined in terms of all observations. If the population mean <math>\operatorname{E}(\mathbf{X})</math> is known, the analogous unbiased estimate

:<math> q_{jk}=\frac{1}{N}\sum_{i=1}^N \left(  x_{ij}-\operatorname{E}(X_j)\right)  \left( x_{ik}-\operatorname{E}(X_k)\right), </math>

using the population mean, has <math>\textstyle N</math> in the denominator. This is an example of why in probability and statistics it is essential to distinguish between [[Random variable|random variable]]s (upper case letters) and [[Realization (probability)|realizations]] of the random variables (lower case letters).

The maximum likelihood  [[Estimation of covariance matrices|estimate of the covariance]]

:<math> q_{jk}=\frac{1}{N}\sum_{i=1}^N \left(  x_{ij}-\bar{x}_j \right)  \left( x_{ik}-\bar{x}_k \right) </math>

for the Gaussian distribution case has ''N'' in the denominator as well. The ratio of 1/''N'' to 1/(''N''&nbsp;&minus;&nbsp;1) approaches 1 for large&nbsp;''N'', so the maximum likelihood estimate approximately equals the unbiased estimate when the sample is large.

==Distribution of the sample mean==

For each random variable, the sample mean is a good [[Estimator|estimator]] of the population mean, where a "good" estimator is defined as being efficient and unbiased. Of course the estimator will likely not be the true value of the [[Statistical population|population]] mean since different samples drawn from the same distribution will give different sample means and hence different estimates of the true mean. Thus the sample mean is a [[Random variable|random variable]], not a constant, and consequently has its own distribution. For a random sample of ''N'' observations on the ''j''<sup>th</sup> random variable, the sample mean's distribution itself has mean equal to the population mean <math>E(X_j)</math> and variance equal to  <math> \sigma^2_j/N</math>, where <math>\sigma^2_j</math> is the population variance.

The arithmetic mean of a [[Statistical population|population]], or population mean, is often denoted ''μ''.<ref>Underhill, L.G.; Bradfield d. (1998) ''Introstat'', Juta and Company Ltd. {{isbn|0-7021-3838-X}} [https://books.google.com/books?id=f6TlVjrSAsgC&pg=PA181 p. 181]</ref> The sample mean <math> \bar{x}</math> (the arithmetic mean of a sample of values drawn from the population) makes a good [[Estimator|estimator]] of the population mean, as its expected value is equal to the population mean (that is, it is an unbiased estimator). The sample mean is a [[Random variable|random variable]], not a constant, since its calculated value will randomly differ depending on which members of the population are sampled, and consequently it will have its own distribution. For a random sample of ''n'' [[Independence (probability theory)|independent]] observations, the expected value of the sample mean is

:<math> \operatorname E (\bar{x}) = \mu </math>

and the [[Variance|variance]] of the sample mean is

: <math> \operatorname{var}(\bar{x}) = \frac{\sigma^2} n. </math>

If the samples are not independent, but [[Correlation|correlated]], then special care has to be taken in order to avoid the problem of [[Pseudoreplication|pseudoreplication]].

If the population is [[Normal distribution|normally distributed]], then the sample mean is normally distributed as follows:

: <math>\bar{x} \thicksim N\left\{\mu, \frac{\sigma^2}{n}\right\}.</math>

If the population is not normally distributed, the sample mean is nonetheless approximately normally distributed if ''n'' is large and&nbsp;''σ''<sup>2</sup>/''n''&nbsp;<&nbsp;+∞. This is a consequence of the [[Central limit theorem|central limit theorem]].

==Weighted samples==

In a weighted sample, each vector <math>\textstyle \textbf{x}_{i}</math> (each set of single observations on each of the ''K'' random variables) is assigned a weight <math>\textstyle w_i \geq0</math>. Without loss of generality, assume that the weights are [[Normalizing constant|normalized]]:

:<math> \sum_{i=1}^{N}w_i = 1. </math>

(If they are not, divide the weights by their sum).
Then the weighted mean vector <math>\textstyle \mathbf{\bar{x}}</math> is given by

:<math> \mathbf{\bar{x}}=\sum_{i=1}^N w_i \mathbf{x}_i.</math>

and the elements <math>q_{jk}</math> of the weighted covariance matrix <math>\textstyle \mathbf{Q}</math> are
<ref name="Galassi-2021-GSL">Mark Galassi, Jim Davies, James Theiler, Brian Gough, Gerard Jungman, Michael Booth, and Fabrice Rossi. [http://www.gnu.org/software/gsl/doc/html/ GNU Scientific Library - Reference manual, Version 2.6], 2021. 
[https://web.archive.org/web/20210418031437/https://www.gnu.org/software/gsl/doc/html/statistics.html#weighted-samples Section Statistics: Weighted Samples]</ref>

:<math> q_{jk}=\frac{1}{1-\sum_{i=1}^{N}w_i^2}
\sum_{i=1}^N w_i \left(  x_{ij}-\bar{x}_j \right)  \left( x_{ik}-\bar{x}_k \right)  . </math>

If all weights are the same, <math>\textstyle w_{i}=1/N</math>, the weighted mean and covariance reduce to the (biased) sample mean and covariance mentioned above.

==Criticism==
The sample mean and sample covariance are not [[Robust statistics|robust statistics]], meaning that they are sensitive to outliers. As robustness is often a desired trait, particularly in real-world applications, robust alternatives may prove desirable, notably [[Quantile|quantile]]-based statistics such as the sample median for location,<ref>[http://www.edge.org/q2008/q08_16.html#kosko The World Question Center 2006: The Sample Mean], [[Biography:Bart Kosko|Bart Kosko]]</ref> and [[Interquartile range|interquartile range]] (IQR) for dispersion. Other alternatives include [[Trimmed estimator|trimming]] and Winsorising, as in the trimmed mean and the [[Winsorized mean]].

==See also==
* [[Estimation of covariance matrices]]
* [[Scatter matrix]]
* [[Unbiased estimation of standard deviation]]

==References==

{{Reflist}}


[[Category:Covariance and correlation]]
[[Category:Estimation methods]]
[[Category:Summary statistics]]
[[Category:Matrices]]
[[Category:U-statistics]]

{{Sourceattribution|Sample mean and covariance}}