Probability & Random Variables

The Plan

• Vocabulary of chance: sample space, outcomes, events, probability function
• Joint and conditional probability, independence, and Bayes’ rule
• Random variables and their distributions
• Reading: Hansen, Probability and Statistics for Economists, chapters 1–2

A Chance Experiment

A chance experiment is a situation whose outcome is not settled in advance.

• a coin flip
• a roll of the dice
• the age of a randomly chosen worker
• which of 500 applicants gets audited

The Sample Space

An outcome is one possible result of the experiment.

The sample space is the set of all possible outcomes.

• Use \Omega to represent the sample space

• \omega \in \Omega is one particular outcome

For a 6-sided die, the sample space is \Omega = \{1, 2, 3, 4, 5, 6\}.

Suppose we roll the die 10 times. Use i=1...10 to index each roll. \omega_i is whatever came up on the $i^{it} roll.

Events

An event is a subset of the sample space.

Roll a six-sided die:

• A = \{2, 4, 6\} is the event “roll an even number”
• B = \{2\} is the event “roll a two”

Unions, Intersections, and Complements

• Union: A \cup B is the set of outcomes that belongs to event A, or event B, or both A and B.

• Intersection: A \cap B is the set of outcomes that belong to both event A and event B

• Complement: A^c is the set of outcomes that do not belong to event A”

Examples: Unions, Intersections, and Complements

Suppose the sample space is \{1, 2, 3, 4, 5, 6\}.

Event A = \{1, 2, 3\}

Event B = \{2, 3, 4\}.

• Union: A \cup B = \{1, 2, 3, 4\}
• Intersection: A \cap B = \{2, 3\}
• Complement: A^c = \{4, 5, 6\}

The Probability Function

\Pr(A) is a function that maps the sample space to the real number line, with the following properties:

• 0 \le \Pr(A) \le 1

• \Pr(\Omega) = 1 and \Pr(\varnothing) = 0

• \Pr(A \cup B) = \Pr(A) + \Pr(B) when A and B are disjoint

• If A and B are not disjoint, then: \Pr(A \cup B) = \Pr(A) + \Pr(B) - \Pr(A \cap B).

What does a probability mean?

Frequentist: If you repeat the chance experiment (roll the die) many times, then \Pr(A) is the fraction of times that event A occurs after many repetitions.

Bayesian: \Pr(A) represents your degree of belief that event A will occur, based on the available evidence.

Some examples of probabilities – games of chance

• The probability of rolling a 6 on a fair 6-sided die is \frac{1}{6}

• The probability of a drawing a red card from a standard deck of 52 cards is \frac{26}{52} = \frac{1}{2}

• The probability of rolling a sum of 7 on two fair 6-sided dice is \frac{6}{36} = \frac{1}{6}

Some examples of probabilities – real world

• Use j=1...N to index the set of newly enrolled undergraduate students at Indiana University in 2026.

• Suppose that GPA_j is student j’s high school GPA.

• Suppose we randomly select 1 student from the N newly enrolled students.

  • What is the probability that the selected student has a high school GPA above 3.5?

  • What is the probability that the selected student will graduate in 4 years?

Joint and Conditional Probability

\Pr(A \cap B) is the joint probability that both A and B occur.

\Pr(A \mid B) is the conditional probability of A, given that B has occurred.

Conditional probability is defined as:

\Pr(A \mid B) = \frac{\Pr(A \cap B)}{\Pr(B)} \qquad \text{for } \Pr(B) > 0

• The denominator renormalizes: B is the new sample space and we ask what fraction of the B outcomes are also in A?

Joint and Conditional Probability – Example

What is the probability that a student will graduate in 4 years, given that they have a high school GPA above 3.5?

The Multiplication Rule and Independence

Rearrange the conditional probability definition to get the multiplication rule:

\Pr(A \cap B) = \Pr(A \mid B)\,\Pr(B)

A and B are independent when learning B tells you nothing about A:

\Pr(A \mid B) = \Pr(A) \qquad \Longleftrightarrow \qquad \Pr(A \cap B) = \Pr(A)\,\Pr(B)

Bayes’ Rule

Suppose you know \Pr(B \mid A) but you want to know \Pr(A \mid B)?

Write the conditional probability, then substitute the multiplication rule in the numerator:

\begin{align*} \Pr(A \mid B) &= \frac{\Pr(A \cap B)}{\Pr(B)} \\[4pt] &= \frac{\Pr(B \mid A)\,\Pr(A)}{\Pr(B)} \end{align*}

Now apply Bayes’ rule:

\begin{align} \Pr(Cancer \mid Positive) &= \frac{\Pr(Positive \mid Cancer)\Pr(Cancer)}{\Pr(Positive)} \\ &= \frac{(0.828)(0.0064)}{0.0985} \\ &\approx 0.054. \end{align}

Random Variables

Talking about probability in terms of chance experiments can be cumbersome and hard to relate to the real world.

It is very common to model chance experiments using the idea of a random variable.

What is a random variable?

A random variable is a function that assigns a real number to each outcome in a sample space.

X : \Omega \to \mathbb{R} is a random variable thattakes outcomes from the sample space as input and returns a real number as output.

Suppose we roll two 6-sided dice. We can define many different random variables.

• X^{sum} is the sum of the two dice.
• S^{product} is the product of the two dice.
• D^{difference} is the difference between the two dice.
• X1 is the value of the first die, X2 is the value of the second die.

Working with random variables usually easier than working with the sample space and events directly.

Probability Distributions

We can describe a random variable by its probability distribution, which is a function that assigns probabilities to the values of the random variable.

Two main types of probability functions: cumulative distribution function (CDF) and probability density function (PDF) or probability mass function (PMF).

The Cumulative Distribution Function

A random variable is described by its CDF:

F_X(x) = \Pr(X \le x)

• As x runs from -\infty to +\infty, F_X climbs weakly from 0 to 1

• F_X is a complete summary of X: if you know F, you can use it to find the probability of any event.

When we say “the population” or “the data generating process,” this is the object we mean.

Continuous and Discrete

X is continuous if F_X is smooth and differentiable.

X is discrete if F_X is a staircase, jumping at the values X can take.

• Wages, test scores, and BMI we treat as continuous
• Number of children, and any 0/1 indicator, are discrete
• Mixed variables exist too. Hours worked piles up mass at zero and is continuous above it.

Probability Density Functions

For continuous X — the density is the derivative of the CDF:

f_X(x) = F_X'(x)

Integrate it to get a probability:

\Pr(a \le X \le b) = \int_a^b f_X(x)\,dx

Integrating the density over the whole real line gives 1:

Probability Mass Functions

For discrete X — the PMF is the probability that the random variable will take on each value from its support.

Suppose that X can take on values x_1, x_2, \ldots, x_k. Then the PMF is p_X(x_j) = \Pr(X = x_j)

The PMF is the size of the steps in the CDF – it’s the discrete analogue of the density.

p_X(x_j) = F_X(x_j) - F_X(x_{j-1})

Sum the PMF over all possible values to get 1: \sum_{j=1}^k p_X(x_j) = 1

The Same Variable, Two Views

• Log hourly wages for 526 workers, drawn as an empirical CDF and as a density

• The pink guide on the left reads the median off the CDF backwards, from probability 0.5 to the value

Joint Distributions

A joint distribution describes two or more random variables at once.

The joint CDF of two random variables X and Y is

F_{X,Y}(x, y) = \Pr(X \le x,\ Y \le y)

If X and Y are continuous, the joint density is:

f_{X,Y}(x,y) = \frac{\partial^2}{\partial x \partial y} F_{X,Y}(x, y)

Marginal and Conditional CDFs

The marginal CDF of X is:

F_X(x) = \Pr(X \le x) = \int_{-\infty}^x f_X(t)\,dt

The conditional CDF of Y given X = x is:

F_{Y \mid X}(y \mid x) = \Pr(Y \le y \mid X = x) = \int_{-\infty}^y f_{Y \mid X}(t \mid x)\,dt

Marginal and Conditional Densities

Marginal — integrate the other variable out and recover one variable on its own:

f_X(x) = \int f_{X,Y}(x, y)\,dy

Conditional — fix one variable and ask about the other:

f_{Y \mid X}(y \mid x) = \frac{f_{X,Y}(x, y)}{f_X(x)}

That is the same renormalizing move as \Pr(A \mid B), now with densities.

Independence

X and Y are statistically independent when the joint distribution or density factors into the marginals:

f_{X,Y}(x,y) = f_X(x)\,f_Y(y) \qquad \text{equivalently} \qquad f_{Y \mid X}(y \mid x) = f_Y(y)

Probability and Statistics

Probability starts from a known F and reasons forward to what the data might look like.

Statistics starts from realized data and reasons backward to what F might be.

• We never see F. We see a sample of draws from it.

• The job is to use those draws to learn features of F, and to be honest about how uncertain the answer is

• “Population,” “DGP,” and “the truth” are all names for F

Recap

• A random variable is a function on the sample space, and it is what we work with instead of \Omega

• The CDF is a complete description of a random variable; the density or mass function is the same information in slope form

• Conditioning renormalizes onto the event you were told about, and Bayes’ rule flips the direction of conditioning

• A joint distribution carries the relationship between variables

• F is the population.