SPEA V706 · Math Camp · Day 2
The CDF is a complete description of a random variable.
But CDFs (and PDFs and PMFs) are unwieldy.
The “moments” are functions of the CDF that describe features of a random variable
Reading: Hansen, PSE, chapter 2
Continuous X — weight by the density and integrate:
E[X] = \int_{-\infty}^{\infty} x \, f_X(x)\,dx
Discrete X — weight by the mass and sum:
E[X] = \sum_{x} x \, p_X(x)
For a given random variable (CDF), the expected value is a constant – a fixed real number 1.5 or 93.
But often we use the expected value as “an operator” or action – take the expectation of Z.
\begin{aligned} \textbf{(E1)}\quad & E[a] = a \\[2pt] \textbf{(E2)}\quad & E[aX] = a\,E[X] \\[2pt] \textbf{(E3)}\quad & E[X + Y] = E[X] + E[Y] \\[2pt] \textbf{(E4)}\quad & E[XY] = E[X]\,E[Y] + \operatorname{Cov}(X, Y) \\[2pt] \textbf{(E5)}\quad & E\big[g(X)\big] \ge g\big(E[X]\big) \quad \text{for convex } g \end{aligned}
The expectation operator passes straight through sums and constants:
\begin{align*} E[a + bX + cY] &= a + b\,E[X] + c\,E[Y] \end{align*}
Nothing mysterious is going on. An expectation is a weighted sum, and sums are linear.
Work with a discrete X, where the algebra is easy to see:
E[g(X)] = \sum_{x} g(x)\, p_X(x)
Key things to notice:
Note: The logic is the same for continuous X; simply replace \sum_x \cdots p_X(x) with \int \cdots f_X(x)\,dx .
Let a be a fixed number, so g(x) = a for every x:
\begin{align*} E[a] &= \sum_{x} a \, p_X(x) \\[4pt] &= a \sum_{x} p_X(x) \\[4pt] &= a \cdot 1 \\[4pt] &= a \end{align*}
Line 2 pulls the constant out front. Line 3 uses the fact that the masses sum to one.
Now let g(x) = a x:
\begin{align*} E[aX] &= \sum_{x} a x \, p_X(x) \\[4pt] &= a \sum_{x} x \, p_X(x) \\[4pt] &= a\, E[X] \end{align*}
The only move is factoring a out of the sum. The weights p_X(x) never change, so scaling every possible value of X by a scales the weighted average by a.
With two random variables we have to deal with a joint mass function p_{X,Y}(x,y) = Pr(X = x, Y = y) .
Average g(x,y) = x + y over every pair:
\begin{align*} E[X + Y] &= \sum_{x} \sum_{y} (x + y)\, p_{X,Y}(x,y) \\[4pt] &= \sum_{x} \sum_{y} x\, p_{X,Y}(x,y) + \sum_{y} \sum_{x} y\, p_{X,Y}(x,y) \\[4pt] &= \sum_{x} x \left[ \sum_{y} p_{X,Y}(x,y) \right] + \sum_{y} y \left[ \sum_{x} p_{X,Y}(x,y) \right] \\[4pt] &= \sum_{x} x\, p_X(x) + \sum_{y} y\, p_Y(y) \\[4pt] &= E[X] + E[Y] \end{align*}
Line 2 splits the sum over the addition.
Line 3 pulls x out of the inner sum over y (it is a constant as far as that sum is concerned), and likewise for y.
Line 4 shows that summing the joint masses over one variable leaves the marginal mass function of the other.
Notice what we never used: we never assumed X and Y were independent, and we never assumed p_{X,Y}(x,y) = p_X(x)\, p_Y(y).
Apply the three rules in order to a + bX + cY:
\begin{align*} E[a + bX + cY] &= E[a] + E[bX] + E[cY] && \text{(E3)} \\[4pt] &= a + E[bX] + E[cY] && \text{(E1)} \\[4pt] &= a + b\,E[X] + c\,E[Y] && \text{(E2)} \end{align*}
E[XY] = E[X]\,E[Y] + \operatorname{Cov}(X, Y)
Try to run the (E3) argument on a product.
E[XY] = \sum_{x} \sum_{y} x\,y\; p_{X,Y}(x,y)
In (E3) we pulled x out of the inner sum and what was left summed to a marginal. Here y is still sitting inside, multiplying x, so nothing collapses. The sum is stuck.
It only comes unstuck if the joint mass factors, p_{X,Y}(x,y) = p_X(x)\, p_Y(y):
\sum_{x} \sum_{y} x\,y\; p_X(x)\,p_Y(y) = \Big[\sum_{x} x\, p_X(x)\Big]\Big[\sum_{y} y\, p_Y(y)\Big] = E[X]\,E[Y]
That factoring is only valid if we know that X and Y are statistically independent. (Not generally true.)
Write \mu_X = E[X] and \mu_Y = E[Y]. Define covariance:
\begin{align*} \operatorname{Cov}(X,Y) &= E\big[(X - \mu_X)(Y - \mu_Y)\big] \\[4pt] &= E\big[XY - \mu_X Y - \mu_Y X + \mu_X \mu_Y\big] \\[4pt] &= E[XY] - \mu_X E[Y] - \mu_Y E[X] + \mu_X \mu_Y \\[4pt] &= E[XY] - \mu_X \mu_Y - \mu_Y \mu_X + \mu_X \mu_Y \\[4pt] &= E[XY] - \mu_X \mu_Y \end{align*}
Line 2 multiplies out the product inside the brackets — ordinary algebra.
Line 3 uses linearity to break up the terms: (E3) splits it, and (E1) and (E2) handle the constants \mu_X and \mu_Y.
Line 4 substitutes E[X] = \mu_X and E[Y] = \mu_Y, and the last two terms cancel.
Move the product of means to the other side:
\begin{align*} \operatorname{Cov}(X,Y) &= E[XY] - E[X]\,E[Y] \\[6pt] \Longrightarrow\quad E[XY] &= E[X]\,E[Y] + \operatorname{Cov}(X,Y) \end{align*}
Zero covariance kills the correction term and E[XY] = E[X]\,E[Y]. But zero covariance is weaker than statistical independence.
Expectations do not pass through nonlinear functions.
For a convex function g (curves upward, like x^2 or e^x):
E\big[g(X)\big] \;\ge\; g\big(E[X]\big)
For a concave g (curves downward, like \log x) the inequality flips.
Let X equal 1 or 3, each with probability \tfrac{1}{2}, and take g(x) = x^2. Then E[X] = 2.
Square first, then average:
E[X^2] = \tfrac{1}{2}(1)^2 + \tfrac{1}{2}(3)^2 = \tfrac{1 + 9}{2} = 5
Average first, then square:
\big(E[X]\big)^2 = 2^2 = 4
Squaring punishes the low draw less than it rewards the high draw, so the average of the squares sits above the square of the average.
In our example, the gap is E\[X^2] - E[X]^2 = 5 - 4 = 1.
The spread of X around its mean is
E\big[(X - 2)^2\big] = \tfrac{1}{2}(1 - 2)^2 + \tfrac{1}{2}(3 - 2)^2 = \tfrac{1 + 1}{2} = 1
Variance is 1, just like the gap between “square then average” and “average then square”.
One small fact before the proof. If h(x) \ge 0 for every x, then
E\big[h(X)\big] = \sum_{x} h(x)\, p_X(x) \;\ge\; 0
Every term in that sum is a nonnegative number times a nonnegative probability, so the total cannot be negative.
Apply this to h = u - v and use (E3):
u(x) \ge v(x) \;\text{ for all } x \quad \Longrightarrow \quad E\big[u(X)\big] \ge E\big[v(X)\big]
An inequality that holds draw by draw survives averaging.
A convex g curves upward. The useful way to say that: at any point m you can draw a straight line through \big(m, g(m)\big) that the function never dips below.
g(x) \;\ge\; g(m) + \lambda\,(x - m) \qquad \text{for all } x
The slope is \lambda = g'(m) when g is differentiable. This is the supporting line at m.
Check it on g(x) = x^2 at m = 2, where the tangent is 4x - 4:
Put the supporting line at the mean. Set m = E[X], so for every x
g(x) \;\ge\; g\big(E[X]\big) + \lambda\,\big(x - E[X]\big)
That holds draw by draw, so we can take expectations of both sides:
\begin{align*} E\big[g(X)\big] &\ge E\Big[\, g\big(E[X]\big) + \lambda\big(X - E[X]\big) \Big] \\[4pt] &= g\big(E[X]\big) + \lambda\,E\big[X - E[X]\big] && \text{(E1), (E2), (E3)} \\[4pt] &= g\big(E[X]\big) + \lambda\,\big(E[X] - E[X]\big) \\[4pt] &= g\big(E[X]\big) \end{align*}
If g is concave then -g is convex, so Jensen applies to -g:
E\big[-g(X)\big] \ge -g\big(E[X]\big)
Multiply through by -1 and the inequality turns around:
E\big[g(X)\big] \;\le\; g\big(E[X]\big)
Wages are the standard case. \log is concave, so
E[\log W] \;\le\; \log E[W]
Average log wage is below log average wage.
\begin{align} \operatorname{Var}(X) &= E\big[(X - E[X])^2\big] \\ &= E[X^2] - (E[X])^2 \end{align}
We start with the definition of variance:
\operatorname{Var}(X) = E\big[(X - E[X])^2\big]
Expanding the square inside the expectation:
\operatorname{Var}(X) = E\big[X^2 - 2X E[X] + (E[X])^2\big]
Using linearity of expectation:
\operatorname{Var}(X) = E[X^2] - 2E[X]E[X] + (E[X])^2
Simplifying:
\operatorname{Var}(X) = E[X^2] - 2(E[X])^2 + (E[X])^2
\operatorname{Var}(X) = E[X^2] - (E[X])^2
Take g(x) = x^2, which is convex, and read the identity we just proved from right to left:
\underbrace{E[X^2]}_{E[g(X)]} - \underbrace{(E[X])^2}_{g(E[X])} = \operatorname{Var}(X)
The distance between “square then average” and “average then square” is exactly the variance.
For our two-point X the gap was 5 - 4 = 1, and the variance was 1.
Variance measures the “spread” of X with units of the X-squared, which is not intuitive.
The standard deviation is the square root of the variance:
\sigma_X = \sqrt{\operatorname{Var}(X)}
Today, we will focus on the first two rules. Tomorrow we will discuss covariance.
\begin{aligned} \textbf{(V1)}\quad & \operatorname{Var}(X) = E[X^2] - (E[X])^2 \\[2pt] \textbf{(V2)}\quad & \operatorname{Var}(aX + b) = a^2 \operatorname{Var}(X) \\ \textbf{(V3)}\quad & \operatorname{Cov}(X, Y) = E[XY] - E[X]\,E[Y] \\[2pt] \textbf{(V4)}\quad & \operatorname{Cov}(aX + bY,\, Z) \\ & \qquad = a\,\operatorname{Cov}(X, Z) + b\,\operatorname{Cov}(Y, Z) \\[2pt] \textbf{(V5)}\quad & \operatorname{Var}(X + Y) \\ & \qquad = \operatorname{Var}(X) + \operatorname{Var}(Y) + 2\,\operatorname{Cov}(X, Y) \\[2pt] \textbf{(V6)}\quad & \operatorname{Corr}(X, Y) = \frac{\operatorname{Cov}(X, Y)}{\sigma_X\,\sigma_Y} \in [-1, 1] \end{aligned}
Shift X by a constant and scale it by a. Push it through the definition:
\begin{align*} \operatorname{Var}(aX + b) &= E\big[(aX + b - E[aX + b])^2\big] \\ &= E\big[(aX - a\,E[X])^2\big] \\ &= E\big[a^2 (X - E[X])^2\big] \\ &= a^2 \operatorname{Var}(X) \end{align*}
The additive constant b cancels in the second line. Sliding a distribution left or right does not change its spread.
The multiplicative constant a comes out squared. Doubling X quadruples its variance.
Let X be the number of children in a household. The last column accumulates the mass into the CDF:
| x | \Pr(X = x) | F(x) |
|---|---|---|
| 0 | 0.15 | 0.15 |
| 1 | 0.25 | 0.40 |
| 2 | 0.30 | 0.70 |
| 3 | 0.20 | 0.90 |
| 4 | 0.10 | 1.00 |
Start with the mean. Weight each value by its probability:
\begin{align*} E[X] &= 0(.15) + 1(.25) + 2(.30) \\ &\quad + 3(.20) + 4(.10) \\ &= 1.85 \end{align*}
Now the raw second moment, weighting the squares the same way:
\begin{align*} E[X^2] &= 0(.15) + 1(.25) + 4(.30) \\ &\quad + 9(.20) + 16(.10) \\ &= 4.85 \end{align*}
Feed both into V1, then take the root:
\begin{align*} \operatorname{Var}(X) &= E[X^2] - (E[X])^2 \\ &= 4.85 - (1.85)^2 \\ &= 4.85 - 3.4225 = 1.4275 \end{align*}
\sigma_X = \sqrt{1.4275} \approx 1.19
Real data instead of a hand table. Compute the mean, variance, and standard deviation of one variable, then several at once.
var() and sd() are the sample analogs of the operators on the slides: the population definition with expectations replaced by averagesRaw moments are defined by taking the expectation of powers of the variable.
Centered moments are defined by taking the expectation of powers of the deviation from the expected value.
We summarize a distribution by its moments. Raw moments locate and size it, centered moments describe its shape.
The expectation E[X] is the probability-weighted average
Expectation is linear (E1–E3): constants slide out and sums split apart
Products do not split (E4), and nonlinear functions do not pass through (E5, Jensen)
Variance is E[X^2] - (E[X])^2, and rescaling squares the constant (V1–V2). Its root \sigma_X reports spread in X’s own units.
Tomorrow: covariance and correlation, the same operators applied to two variables at once