Probability#
In this chapter we will go over how to perform probability computations in Python.
Basic probability calculations#
Let’s create a vector of outcomes from one to 6, using the np.arange() function to create such a sequence. This function takes the minimum and maximum values as its inputs, but note that the maximum is not included in the sequence; that is, the sequence goes up to but not including the maximum. Thus, we would have to give 1 and 7 as the minimum and maximum in order to get a sequence of numbers from 1 to 6:
import numpy as np
outcomes = np.arange(1, 7)
print(outcomes)
[1 2 3 4 5 6]
Now let’s create a vector of logical values based on whether the outcome in each position is equal to 1. Remember that == tests for equality of each element in a vector:
outcome1isTrue = outcomes == 1
print(outcome1isTrue)
[ True False False False False False]
Remember that the simple probability of an outcome is number of occurrences of the outcome divided by the total number of events. To compute a probability, we can take advantage of the fact that TRUE/FALSE are equivalent to 1/0 in Python. The formula for the mean (sum of values divided by the number of values) is thus exactly the same as the formula for the simple probability! So, we can compute the probability of the event by simply taking the mean of the logical vector.
p1isTrue = np.mean(outcome1isTrue)
print(p1isTrue)
0.16666666666666666
Empirical frequency#
Let’s walk through how we computed empirical frequency of rain in San Francisco.
First we load the data:
#+
import pandas as pd
SFrain = pd.read_csv('https://raw.githubusercontent.com/statsthinking21/statsthinking21-python/master/notebooks/data/SanFranciscoRain.csv')
# we will remove the STATION and NAME variables
# since they are identical for all rows
SFrain = SFrain.drop(columns=['STATION', 'NAME'])
print(SFrain)
#-
DATE PRCP
0 2017-01-01 0.05
1 2017-01-02 0.10
2 2017-01-03 0.40
3 2017-01-04 0.89
4 2017-01-05 0.01
.. ... ...
360 2017-12-27 0.00
361 2017-12-28 0.00
362 2017-12-29 0.00
363 2017-12-30 0.00
364 2017-12-31 0.00
[365 rows x 2 columns]
We see that the data frame contains a variable called PRCP which denotes the amount of rain each day. Let’s create a new variable called rainToday that denotes whether the amount of precipitation was above zero:
SFrain['rainToday'] = SFrain['PRCP'] > 0
print(SFrain)
DATE PRCP rainToday
0 2017-01-01 0.05 True
1 2017-01-02 0.10 True
2 2017-01-03 0.40 True
3 2017-01-04 0.89 True
4 2017-01-05 0.01 True
.. ... ... ...
360 2017-12-27 0.00 False
361 2017-12-28 0.00 False
362 2017-12-29 0.00 False
363 2017-12-30 0.00 False
364 2017-12-31 0.00 False
[365 rows x 3 columns]
Now we will summarize the data to compute the probability of rain:
pRainInSF = SFrain['rainToday'].mean()
print(pRainInSF)
0.2
Conditional probability#
Let’s determine the conditional probability of someone having hearing problems, given that they are over 70 years of age, using the NHANES dataset. First, let’s create a new variable called Over70 that denotes whether each individual is over 70 or not.
! pip install nhanes
from nhanes.load import load_NHANES_data
nhanes_data = load_NHANES_data()
nhanes_data['Over70'] = nhanes_data['AgeInYearsAtScreening'] > 70
Requirement already satisfied: nhanes in /miniconda/lib/python3.12/site-packages (0.5.1)
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
Now let’s create a cleaned-up dataset that only includes the over70 variable along with the variable called HaveSeriousDifficultyHearing that denotes whether a person reports having serious hearing difficulty (coded as 1 for “yes” and 0 for “no”).
hearing_data = nhanes_data[['Over70', 'HaveSeriousDifficultyHearing']].dropna()
print(hearing_data)
Over70 HaveSeriousDifficultyHearing
SEQN
93703.0 False 0.0
93704.0 False 0.0
93705.0 False 0.0
93706.0 False 0.0
93707.0 False 0.0
... ... ...
102952.0 False 0.0
102953.0 False 0.0
102954.0 False 0.0
102955.0 False 0.0
102956.0 False 0.0
[8365 rows x 2 columns]
First, what’s the probability of being over 70?
p_over_70 = hearing_data['Over70'].mean()
print(p_over_70)
0.10531978481769277
Second, what’s the probability of having hearing problems?
p_hearing_problem = hearing_data['HaveSeriousDifficultyHearing'].mean()
print(p_hearing_problem)
0.06276150627615062
What’s the probability for each combination of hearing problems/no problems and over 70/ not? We can create a table that finds the joint probability for each combination, using the pd.crosstab() function:
joint_table = pd.crosstab(hearing_data['Over70'], hearing_data['HaveSeriousDifficultyHearing'], normalize=True)
print(joint_table)
HaveSeriousDifficultyHearing 0.0 1.0
Over70
False 0.859653 0.035027
True 0.077585 0.027735
Finally, what’s the probability of someone having hearing problems, given that they are over 70 years of age? To do this, we limit the computation of the probability of having hearing problems to only include those people who are over 70:
p_hearingproblem_given_over_70 = hearing_data.query('Over70 == True')['HaveSeriousDifficultyHearing'].mean()
print(p_hearingproblem_given_over_70)
0.2633371169125993
Now compute the opposite: What is the probability of being over 70 given that one has a hearing problem?
p_over_70_given_hearingproblem = hearing_data.query('HaveSeriousDifficultyHearing == True')['Over70'].mean()
print(p_over_70_given_hearingproblem)
0.4419047619047619