Parametric Distributions

The Plan

• A selection of named distributions that are often useful.
• Three Discrete families: Bernoulli, Binomial, Poisson
• Three Continuous families: Uniform, Exponential, Normal
• Reading: Hansen, PSE, chapter 3

A Whole Distribution in One or Two Numbers

• A generic CDF is abstract – it refers to a generic random variable.

• A parametric distribution is a specific functional form for the CDF.

• A parametric family is a set of distributions, where each member of the family is distinghuished by one or two parameters.

Bernoulli: One Coin Flip

X \in \{0, 1\} with \Pr(X = 1) = p.

CDF of Bernoulli: F_X(x) = \begin{cases} 0 & x < 0 \\ 1-p & 0 \le x < 1 \\ 1 & x \ge 1 \end{cases} PMF of Bernoulli: f_X(x) = \begin{cases} 1-p & x = 0 \\ p & x = 1 \\ 0 & \text{otherwise} \end{cases}

• People say X is distributed as a Bernoulli with parameter p, written as X \sim \text{Bernoulli}(p) .

• Bernoulli is the “family”. A specific Bernoulli random variable is determined by the value of the parameter p which is the probability of success.

  • Suppose that X is the value of a fair coin flip with heads = 1 and tails = 0. Then X \sim \text{Bernoulli}(0.5).

  • Draw a card from a shuffled deck and let X = 1 if it is a heart and X = 0 otherwise. Then X \sim \text{Bernoulli}(0.25).

• Moments of X ~ \text{Bernoulli}(p)

  • E[X] = 0 \cdot (1-p) + 1 \cdot p = p
  • \operatorname{Var}(X) = E[X^2] - (E[X])^2 = p - p^2 = p(1-p).

• The variance is largest at p = 0.5 and vanishes at p = 0 or p = 1.

Binomial: Counting Successes

Let X be the count of successes after n independent Bernoulli trials, each with probability p.

Then X has a Binomial distribution, written as X \sim \text{Binomial}(n, p).

CDF of Binomial: F_X(x) = \sum_{k=0}^{\lfloor x \rfloor} \binom{n}{k} p^k (1-p)^{n-k} PMF of Binomial: f_X(x) = \binom{n}{x} p^x (1-p)^{n-x} for x = 0, 1, \dots, n.

• The binomial coefficient counts the orderings, and the rest is the probability of any one of them

• Moments of X ~ \text{Binomial}(n, p):

  • E[X] = \sum_{x=0}^{n} x \binom{n}{x} p^x (1-p)^{n-x} = np
    
  • \operatorname{Var}(X) = \sum_{x=0}^{n} (x - np)^2 \binom{n}{x} p^x (1-p)^{n-x} = np(1-p)

Poisson: Events in a Fixed Window

Suppose that X measure the number of job applications received in a day and the average number of applications per day is \lambda.

Then X has a Poisson distribution, written as X \sim \text{Poisson}(\lambda).

CDF of Poisson: F_X(x) = \sum_{k=0}^{\lfloor x \rfloor} \frac{e^{-\lambda}\lambda^k}{k!} PMF of Poisson: f_X(x) = \frac{e^{-\lambda}\lambda^x}{x!}, \qquad x = 0, 1, 2, \dots

Moments of X \sim \text{Poisson}(\lambda):

• E[X] = \sum_{x=0}^{\infty} x \frac{e^{-\lambda}\lambda^x}{x!} = \lambda

• \operatorname{Var}(X) = \sum_{x=0}^{\infty} (x - \lambda)^2 \frac{e^{-\lambda}\lambda^x}{x!} = \lambda

• Weird: Expected value and variance are the same number!

Job Search

You are on the job market. Employers reach out at random times, and you want to know how long the search is going to take.

• Callbacks arrive at an average rate of \lambda = 3 per month.

• Let X_t be the number of callbacks in a window t months long, and model it as X_t \sim \text{Poisson}(\lambda t).

Interviews in one month

In one month, the number of interviews is X_1 \sim \text{Poisson}(3):

• \Pr(X_1 = 0) = e^{-3} \approx 0.05 – a month with nothing at all
• \Pr(X_1 = 1) = \dfrac{e^{-3} 3^1}{1!} \approx 0.15
• \Pr(X_1 = 2) = \dfrac{e^{-3} 3^2}{2!} \approx 0.22

Over three months, X_3 \sim \text{Poisson}(9)

• \lambda is a rate per unit of time, so the count in a window of length t has mean \lambda t.

Waiting for an interview

Let T be the waiting time until the first interview. You are still waiting at time t if you have received no callbacks in the first t months.

The probability that you have to wait more than time t is: \Pr(T > t) = \Pr(X_t = 0) = e^{-\lambda t} = e^{-3 t}, \qquad t \ge 0

The CDF of the waiting time is

F_T(t) = \Pr(T \le t) = 1 - e^{-\lambda t} = 1 - e^{-3 t}, \qquad t \ge 0

This is the CDF of an Exponential distribution with rate \lambda p = 0.6.

Exponential: Waiting Times

X \sim \text{Exp}(\lambda) is the time until the next event, when events arrive at rate \lambda:

CDF of Exponential: F_X(x) = 1 - e^{-\lambda x}, \qquad x \ge 0 PDF of Exponential: f_X(x) = \lambda e^{-\lambda x}, \qquad x \ge 0

Expected value and variance: E[X] = \frac{1}{\lambda} \qquad \operatorname{Var}(X) = \frac{1}{\lambda^2}

Duration data: unemployment spells, time to failure, time between hospital admissions

Right-skewed and non-negative

Job Search Example

With \lambda = 3 interviews per month, expected waiting time is

E[T] = \frac{1}{\lambda} = \frac{1}{3} \text{ months} \approx 10 \text{ days}

Uniform Random Variabl

X \sim U(a, b) has constant density on [a, b]:

f_X(x) = \frac{1}{b - a}, \qquad a \le x \le b

E[X] = \frac{a + b}{2} \qquad \operatorname{Var}(X) = \frac{(b-a)^2}{12}

• Flat density: every value in the range is as likely as any other
• CDF is a straight line climbing from 0 to 1

The Normal Distribution: The Bell Curve

X \sim N(\mu, \sigma^2) has the bell-shaped density

f_X(x) = \frac{1}{\sigma\sqrt{2\pi}}\exp\left(-\frac{(x-\mu)^2}{2\sigma^2}\right)

E[X] = \mu \qquad \operatorname{Var}(X) = \sigma^2

• Symmetric around \mu, and completely described by its first two moments
• A linear function of a Normal is Normal, and a sum of independent Normals is Normal

Standardizing

Any Normal can be shifted and scaled into the standard Normal N(0,1):

Z = \frac{X - \mu}{\sigma} \sim N(0, 1)

• Subtracting \mu centers it and dividing by \sigma sets the spread to one

A Real Variable, Roughly Bell-Shaped

• Log hourly wages for 526 workers (1976 CPS), with a Normal density laid on top at the same mean and standard deviation

• The fit is not perfect, and the shape is close. Many real variables are roughly bell-shaped.

Six Families

• Three discrete on the top row, three continuous on the bottom

More Random Variables

Three more families often come up in hypothesis testing.

• Chi-square \chi^2_k: the sum of k squared independent standard Normals. Mean k, variance 2k.

• Student’s t: a standard Normal divided by the root of an independent scaled \chi^2.

• F: a ratio of two independent scaled \chi^2 variables.

Named Distribution Functions in R

Every family has the same four functions, with a prefix that tells you which question you are asking.

# d = density, p = CDF, q = quantile, r = random draws
dnorm(0)              # height of the N(0,1) density at 0
pnorm(1.96)           # Pr(Z <= 1.96), about 0.975
qnorm(0.975)          # the value with 97.5% below it: 1.96
rnorm(5, mean = 0, sd = 1)   # five draws

# the same four letters work for every family
dbinom(3, size = 10, prob = 0.2)
rpois(5, lambda = 2)
runif(5, min = 0, max = 1)
rexp(5, rate = 1)

• p and q are inverses of each other, which is the CDF-and-quantile relationship from Day 2
• set.seed() before any r function, or your results change every time you run the script

Recap

• A parametric family compresses a whole distribution into one or two parameters
• Discrete: Bernoulli (p), Binomial (np), Poisson (\lambda, with variance also \lambda)
• Continuous: Uniform (flat), Exponential (skewed waiting times), Normal (symmetric, two moments)
• Standardizing turns any Normal into N(0,1), which is why one table covers them all