Covariance & Correlation
SPEA V706 · Math Camp · Day 3
The Plan
Yesterday we summarized one variable. Today we summarize how two variables move together.
Covariance measures linear co-movement, and correlation rescales it into [-1, 1]
The rest of the V-card: bilinearity, the variance of a sum, and the bound |\rho| \le 1
Reading: Hansen, PSE, chapter 2
Covariance and Correlation
Rules to Know: Variance and Covariance
The rest of the V-card. V1 and V2 were yesterday.
\begin{aligned} \textbf{(V1)}\quad & \operatorname{Var}(X) = E[X^2] - (E[X])^2 \\[2pt] \textbf{(V2)}\quad & \operatorname{Var}(aX + b) = a^2 \operatorname{Var}(X) \\ \textbf{(V3)}\quad & \operatorname{Cov}(X, Y) = E[XY] - E[X]\,E[Y] \\[2pt] \textbf{(V4)}\quad & \operatorname{Cov}(aX + bY,\, Z) \\ & \qquad = a\,\operatorname{Cov}(X, Z) + b\,\operatorname{Cov}(Y, Z) \\[2pt] \textbf{(V5)}\quad & \operatorname{Var}(X + Y) \\ & \qquad = \operatorname{Var}(X) + \operatorname{Var}(Y) + 2\,\operatorname{Cov}(X, Y) \\[2pt] \textbf{(V6)}\quad & \operatorname{Corr}(X, Y) = \frac{\operatorname{Cov}(X, Y)}{\sigma_X\,\sigma_Y} \in [-1, 1] \end{aligned}
- (V3) is the definition of covariance, two variable generalization of variance
Covariance (V3)
The covariance asks whether X tends to be above its mean when Y is above its mean:
\operatorname{Cov}(X, Y) = E\big[(X - E[X])(Y - E[Y])\big] = E[XY] - E[X]\,E[Y]
- \operatorname{Var}(X) = \operatorname{Cov}(X, X). Variance is a variable’s covariance with itself.
What does covariance tell us?
Strategy for unpacking:
- Consider discrete random variables with a small number of outcomes.
- Unpack the expected value as a sum over possible outcomes.
- Pay attention to the signs of the deviations from the mean.
- Consider the products of the deviations from the mean.
Alcoholism and Depression
Suppose that we measure the severity of depression and alcohol use disorder using 3-point scales: 0 = Mild/None, 1 = Moderate, 2 = Severe.
Population joint distribution
| A = 0 | A = 1 | A= 2 | |
|---|---|---|---|
| D = 0) | 0.30 | 0.10 | 0.05 |
| D = 1) | 0.10 | 0.15 | 0.05 |
| D = 2) | 0.05 | 0.10 | 0.10 |
First, compute the expected values of A and D: \begin{align*} E[A] &= \sum_{a=0}^{2} a \cdot P(A=a) = 0 \cdot 0.45 + 1 \cdot 0.35 + 2 \cdot 0.20 = 0.75 \\ E[D] &= \sum_{d=0}^{2} d \cdot P(D=d) = 0 \cdot 0.45 + 1 \cdot 0.35 + 2 \cdot 0.20 = 0.75 \end{align*}
Now compute the covariance using the definition: \begin{align*} Cov(A, D) &= E[ (A - E[A])(D-E[D])] \\ &= \sum_{a=0}^{2} \sum_{d=0}^{2} (a - E[A])(d - E[D]) P(A=a, D=d) \\ &= (0-0.75)(0-0.75) \cdot 0.30 + (0-0.75)(1-0.75) \cdot 0.10 + (0-0.75)(2-0.75) \cdot 0.05 \\ &\quad + (1-0.75)(0-0.75) \cdot 0.10 + (1-0.75)(1-0.75) \cdot 0.15 + (1-0.75)(2-0.75) \cdot 0.05 \\ &\quad + (2-0.75)(0-0.75) \cdot 0.05 + (2-0.75)(1-0.75) \cdot 0.10 + (2-0.75)(2-0.75) \cdot 0.10 \\ &= 0.1875 - 0.01875 - 0.046875 - 0.01875 + 0.009375 + 0.0234375 - 0.046875 + 0.0234375 + 0.140625 \\ &= 0.25 \end{align*}
The product of the deviations determines the sign of the entry. - When both terms are positive, the product is positive. - When both terms are negative, the product is also positive. - When one term is positive and the other is negative, the product is negative.
Thus, when covariance is positive the two variables tend to be above or below their means at the same time.
In this (fake) example, alcohol use disorder and depression have positive covariance, which is consistent with the idea that these two disorders are comorbid.
Covariance Is Bilinear (V4)
Covariance is linear in each of its two components
\begin{align*} \operatorname{Cov}(aX + bY,\, Z) &= a\,\operatorname{Cov}(X, Z) \\ &\quad + b\,\operatorname{Cov}(Y, Z) \end{align*}
Constants slide out and sums split apart.
Adding a constant to either argument does nothing: \operatorname{Cov}(X + c, Y) = \operatorname{Cov}(X, Y)
Variance of a Sum (V5)
Because variance is a covariance with itself, the variance of a sum picks up a cross term:
\begin{align*} \operatorname{Var}(X + Y) &= \operatorname{Var}(X) + \operatorname{Var}(Y) \\ &\quad + 2\,\operatorname{Cov}(X, Y) \end{align*}
- If X and Y are uncorrelated, the cross term vanishes and the variances simply add
Correlation (V6)
Covariance is measured in units of X times Y, so its magnitude is hard to intepret.
The correlation is a standardized measure of the linear relationship between two variables.
It is the covariance divided by the product of the two standard deviations:
\rho \equiv \operatorname{Corr}(X, Y) = \frac{\operatorname{Cov}(X, Y)}{\sigma_X \, \sigma_Y}
- It always sits between -1 and 1. Zero means no linear relationship; \pm 1 means a perfect line.
Why |\rho| \le 1: Cauchy–Schwarz
The Cauchy–Schwarz inequality bounds a cross moment by the two second moments (Hansen, Appendix B):
\big|\,E[XY]\,\big| \;\le\; \sqrt{E[X^2]\;E[Y^2]}
Apply it to the centered variables X - E[X] and Y - E[Y]. The left side becomes the covariance and the right side becomes the two standard deviations:
\big|\operatorname{Cov}(X, Y)\big| \;\le\; \sigma_X\,\sigma_Y
Divide both sides by \sigma_X \sigma_Y:
|\operatorname{Corr}(X, Y)| \;\le\; 1
What a Correlation Actually Looks Like
Four clouds of 300 points, labeled with the \rho that generated them.
A correlation of 0.5 is a real relationship and still a very wide cloud.
Independent Implies Uncorrelated
Suppose X and Y are statistically independent. Then the covariance must equal 0:
\begin{align*} \operatorname{Cov}(X, Y) &= E[XY] - E[X]\,E[Y] \\ &= E[X]\,E[Y] - E[X]\,E[Y] \\ &= 0 \end{align*}
Statistical independence implies linear independence (i.e. zero covariance; zero correlation).
But the converse is not true: zero covariance does not imply statistical independence.
Uncorrelated Does Not Imply Independent
Let X take the values -1, 0, 1 with probability 1/3 each, and let Y = X^2. Then Y is a deterministic function of X.
Compute the covariance:
E[X] = \tfrac{1}{3}(-1) + \tfrac{1}{3}(0) + \tfrac{1}{3}(1) = 0 \qquad E[Y] = \tfrac{1}{3}(1) + \tfrac{1}{3}(0) + \tfrac{1}{3}(1) = \tfrac{2}{3}
Now the cross moment. Note XY = X \cdot X^2 = X^3:
E[XY] = E[X^3] = \tfrac{1}{3}(-1) + \tfrac{1}{3}(0) + \tfrac{1}{3}(1) = 0
\operatorname{Cov}(X, Y) = 0 - (0)\big(\tfrac{2}{3}\big) = 0
Even though X and Y are perfectly dependent, they have zero covariance.
The issue is that the dependence is non-linear, and covariance only measures linear dependence.
Simple ideas about OLS regression
Suppose that Y and X are random variables and we want to fit a simple linear regression model that “explains” Y using X.
Y = \alpha + \beta X + e
Where we choose the intercept (\alpha) and slop (\beta) so that the residual has mean zero and Cov(e,X)=0.
\begin{align} Cov(Y, X) &= Cov(\alpha + \beta X + e, X) \\ Cov(Y, X &= Cov(\alpha, X) + \beta Cov(X, X) + Cov(e, X) \\ Cov(Y, X) &= 0 + \beta Var(X) + 0 \\ \frac{Cov(Y,X)}{Var(X)} &= \beta \end{align}
Covariance and Correlation in R
The last two lines print the same slope, which is the point of the previous slide
cov()andcor()are sample analogs, with expectations replaced by averages
Recap
Covariance is E[XY] - E[X]E[Y], and it measures linear co-movement in the units of X times Y
Correlation rescales it by the two standard deviations into [-1, 1], bounded by Cauchy–Schwarz
Covariance is bilinear (V4), and the variance of a sum carries a cross term (V5) unless the two are uncorrelated
Independent implies uncorrelated. Uncorrelated does not imply independent.