Sampling Distributions

SPEA V706 · Math Camp · Day 4

Coady Wing

The Plan

  • populations vs samples

  • estimands, estimators, parameters

  • Sampling distributions

  • Reading: Hansen, PSE, chapters 6–7

The Population

  • Suppose the we are interested in a random variable X
    • Examples:
      • X = value of a dice role, F_X = uniform on \{1,2,3,4,5,6\}
      • X = height of a randomly chosen student in a class, F_X = empirical CDF of the 50 students
      • X = income of a randomly chosen household in the U.S., F_X = CDF of the U.S. income distribution
  • The CDF of X is “the population”

Analogies that people like to use

  1. A literal population: the set of all 50 students in the class, or all households in the U.S.
  • Be careful: for statistics and probability theory, it is the distribution of the random variable that matters, not the set of units.
  • The population is the distribution of heights among the 50 students, or the distribution of incomes among the households in the U.S.
  1. A generative process:
    • bingo ball cage containing N balls with numbers 1 to 10.
    • turn the crank 5 times (jumbling the balls around) and a ball pops out and you record the number.
    • replace the ball
    • repeat this process as many times as you want: the number of draws you can take is infinite;
    • the bingo ball cage is a machine for generating draws from a distribution.

These two ideas are equivalent. But the generative process analogy is often less confusing.

If you are puzzling over what the population is in a given problem, ask yourself how to see it like a bingo ball machine.

The Sample

If X is a random variable with CDF F_X, we can think data collection as a method of generating realizations from F_X. - Turn the crank on the bingo ball to generate 1 draw from F_X. - Repeat this process to generate multiple draws.

A sample is a set of observations drawn from some population CDF.

Use i = 1...n to index individual observations on draws from F_X.

\{X_1, \dots, X_n\} are the individual draws.

A random sample is a set n “independent and identically distributed (iid)” observations.

  • Independent: the value of observation i is statistically independent of the value of observation j for all i \neq j.

  • Identically distributed: every draw comes from the same F_X

Estimand vs. Estimator

An estimand is the population quantity we want to learn, say E[Y]. It is a fixed number, a feature of F.

An estimator is a recipe that turns a sample into a guess at the estimand

The estimate is the value of the estimator for a particular sample.

Thus the estimate is a random variable: if you draw a new sample, you get a different number.

The Sample Mean as an Estimator

Our estimand is the expected value of Y, denoted E[Y].

Suppose we have a random sample of size n from F_Y() with observations \{Y_1, \dots, Y_n\}.

The sample mean is an intuitive estimator of E[Y]:

\bar Y_n = \frac{1}{n} \sum_{i=1}^{n} Y_i

The Sampling Distribution

\bar Y_n is a random variable, so it has a distribution.

That distribution is the sampling distribution of the estimator.

  • Repeated Random Sampling: Imagine drawing a fresh sample of size n, computing \bar Y_n, and writing the number down. Then repeat this process many times.

  • The histogram of those sample avarages is the sampling distribution of the the mean

  • It is not the distribution of Y. It is the distribution of a summary of a sample of Y’s.

Distribution of the sample mean with different sample sizes

  • The same population, sampled at n = 5, n = 30, and n = 200, with 5000 samples at each size

  • All three pile up on the same true mean.

  • Nothing about the population changed here.

Bias: Is the Arrow Aimed Right?

The bias of an estimator is how far off it is on average across repeated samples:

\operatorname{Bias}(\hat\theta) = E[\hat\theta] - \theta

The sample mean is unbiased for E[Y]. E[\bar Y_n] = \frac{1}{n} \sum_{i=1}^{n} E[Y_i] = \frac{1}{n}\,n\,\mu = \mu

  • On average, over many samples, the sample mean lands right on the population expected value.

  • Unbiasedness says nothing about any specific sample. Your particular estimate can still be badly wrong.

Sampling Variance: How Wide Is the Spread?

Start with V2 to pull the 1/n out, then V5 to split the variance of the sum. Independence kills every cross term:

\begin{align*} \operatorname{Var}(\bar Y_n) &= \operatorname{Var}\left(\frac{1}{n}\sum_{i=1}^n Y_i\right) \\ &= \frac{1}{n^2}\operatorname{Var}\left(\sum_{i=1}^n Y_i\right) \\ &= \frac{1}{n^2}\sum_{i=1}^n \operatorname{Var}(Y_i) \\ &= \frac{1}{n^2}\,n\,\sigma^2 \;=\; \frac{\sigma^2}{n} \end{align*}

  • The 1/n came out squared by V2
  • The covariance terms in V5 vanished only because the draws are independent.
  • More data shrinks the sampling variance toward zero.

The Standard Error

The standard error is the standard deviation of the sampling distribution:

\operatorname{se}(\bar Y_n) = \sqrt{\operatorname{Var}(\bar Y_n)} = \frac{\sigma}{\sqrt{n}}

  • It reports the typical size of estimation error, in the units of Y

Large Sample Theory

Large Sample Theory

So far, we have been thinking about the sampling distribution of \bar Y_n for a fixed n across repeated samples.

A different thought experiment is to imagine drawing a single sample, but letting n grow.

The Law of Large Numbers

The sample mean converges to the population mean as the sample grows:

\bar Y_n \;\xrightarrow{\;p\;}\; E[Y] \qquad \text{as } n \to \infty

The sampling variance \sigma^2/n collapses to zero as n grows

Watching the LLN Work

  • Three running averages, each recomputed as we add draws, with the horizontal axis on a log scale
  • Early on they wander. As n grows every path is squeezed onto the true mean of 0.

The Central Limit Theorem

The LLN says \bar Y_n lands on \mu. The CLT tells us the shape of the distribution around it:

\sqrt{n}\,\frac{\bar Y_n - \mu}{\sigma} \;\xrightarrow{\;d\;}\; N(0, 1)

Under weak conditions on the underlying random variable, standardized sample means are distributed approximately Normal as n grows

  • For large n, \;\bar Y_n \approx N(\mu, \sigma^2 / n)

The CLT

  • The population here is right-skewed, an Exponential
  • At n = 1 it inherits the skew. By n = 30 it is tight, symmetric, and bell-shaped.

Watching Both Theorems in R

pacman::p_load(data.table, ggplot2)
set.seed(20260814)

# 5000 sample means of size n = 30, from a
# right-skewed Exponential(1) population
sims <- data.table(
  xbar = replicate(
    5000, mean(rexp(30, rate = 1))))

# the LLN in one number: close to the
# population mean of 1
sims[, mean(xbar)]

# the CLT in one picture: bell-shaped
p <- ggplot(sims, aes(xbar)) +
  geom_histogram(bins = 30,
    fill = "#000080")
p   # draw the histogram

Recap

  • An estimand is a feature of the population or DGP.
  • An estimator is a strategy for estimating the value of the estimand from a sample of observations.
  • The value of the estimator (the estimate) is a random variable, and its distribution across repeated samples is the sampling distribution
  • The sample mean is unbiased, with sampling variance \sigma^2/n and standard error \sigma/\sqrt{n}
  • The LLN says sample averages converge to population means as n increases.
  • The CLT says standardized averages are distributed approximately Normal as n grows, even if the population is not Normal.