Mean and predicted response

In linear regression, mean response and predicted response are values of the dependent variable calculated from the regression parameters and a given value of the independent variable. The values of these two responses are the same, but their calculated variances are different.

Background

In straight line fitting, the model is

$y_i=\alpha +\beta x_i+{\epsilon }_i$

where $y_i$ is the response variable, $x_i$ is the explanatory variable, ε_i is the random error, and $\alpha$ and $\beta$ are parameters. The mean, and predicted, response value for a given explanatory value, x_d, is given by

${\hat{y}}_d=\hat{\alpha }+\hat{\beta }{x}_d,$

while the actual response would be

$y_d=\alpha +\beta x_d+{\epsilon }_d$

Expressions for the values and variances of $\hat{\alpha }$ and $\hat{\beta }$ are given in linear regression.

Mean response

Since the data in this context is defined to be (x, y) pairs for every observation, the mean response at a given value of x, say x_d, is an estimate of the mean of the y values in the population at the x value of x_d, that is $\hat{E}(y\mid x_d)\equiv {\hat{y}}_d$. The variance of the mean response is given by

$\mathrm{Var}(\hat{\alpha }+\hat{\beta }{x}_d)=\mathrm{Var}\left(\hat{\alpha }\right)+(\mathrm{Var}\hat{\beta })x_d^2+2x_d\mathrm{Cov}(\hat{\alpha },\hat{\beta }).$

This expression can be simplified to

$\mathrm{Var}(\hat{\alpha }+\hat{\beta }{x}_d)={\sigma }^2(\frac{1}{m}+\frac{{(x_d-\overline{x})}^2}{\sum (x_i-\overline{x}{)}^2}),$

where m is the number of data points.

To demonstrate this simplification, one can make use of the identity

$\sum (x_i-\overline{x}{)}^2=\sum x_i^2-\frac{1}{m}{(\sum x_i)}^2.$

Predicted response

The predicted response distribution is the predicted distribution of the residuals at the given point x_d. So the variance is given by

$\begin{array}{rl}\mathrm{Var}(y_d-[\hat{\alpha }+\hat{\beta }{x}_d])& =\mathrm{Var}(y_d)+\mathrm{Var}(\hat{\alpha }+\hat{\beta }{x}_d)-2\mathrm{Cov}(y_d,[\hat{\alpha }+\hat{\beta }{x}_d])\\ & =\mathrm{Var}(y_d)+\mathrm{Var}(\hat{\alpha }+\hat{\beta }{x}_d).\end{array}$

The second line follows from the fact that $\mathrm{Cov}(y_d,[\hat{\alpha }+\hat{\beta }{x}_d])$ is zero because the new prediction point is independent of the data used to fit the model. Additionally, the term $\mathrm{Var}(\hat{\alpha }+\hat{\beta }{x}_d)$ was calculated earlier for the mean response.

Since $\mathrm{Var}(y_d)={\sigma }^2$ (a fixed but unknown parameter that can be estimated), the variance of the predicted response is given by

$\begin{array}{rl}\mathrm{Var}(y_d-[\hat{\alpha }+\hat{\beta }{x}_d])& ={\sigma }^2+{\sigma }^2(\frac{1}{m}+\frac{{(x_d-\overline{x})}^2}{\sum (x_i-\overline{x}{)}^2})\\ & ={\sigma }^2(1+\frac{1}{m}+\frac{(x_d-\overline{x}{)}^2}{\sum (x_i-\overline{x}{)}^2}).\end{array}$

Confidence intervals

Further information: Prediction interval

The $100(1-\alpha )\mathrm{\%}$ confidence intervals are computed as $y_d\pm t_{\frac{\alpha }{2},m-n-1}\sqrt{\mathrm{Var}}$. Thus, the confidence interval for predicted response is wider than the interval for mean response. This is expected intuitively – the variance of the population of $y$ values does not shrink when one samples from it, because the random variable ε_i does not decrease, but the variance of the mean of the $y$ does shrink with increased sampling, because the variance in $\hat{\alpha }$ and $\hat{\beta }$ decrease, so the mean response (predicted response value) becomes closer to $\alpha +\beta x_d$.

This is analogous to the difference between the variance of a population and the variance of the sample mean of a population: the variance of a population is a parameter and does not change, but the variance of the sample mean decreases with increased samples.

General linear regression

The general linear model can be written as

$y_i=\sum _{j=1}^n{X}_{ij}{\beta }_j+{\epsilon }_i$

Therefore, since $y_d=\sum _{j=1}^n{X}_{dj}{\hat{\beta }}_j$ the general expression for the variance of the mean response is

$\mathrm{Var}\left(\sum _{j=1}^n{X}_{dj}{\hat{\beta }}_j\right)=\sum _{i=1}^n\sum _{j=1}^n{X}_{di}{S}_{ij}{X}_{dj},$

where S is the covariance matrix of the parameters, given by

$\mathbf{𝐒}={\sigma }^2{\left({\mathbf{𝐗}}^{\mathsf{T}}\mathbf{𝐗}\right)}^{-1}.$

References

• Draper, N. R.; Smith, H. (1998). Applied Regression Analysis (3rd ed.). John Wiley. ISBN 0-471-17082-8.