Probability

In this chapter we will go over how to perform probability computations in Python.

Basic probability calculations

Let’s create a vector of outcomes from one to 6, using the np.arange() function to create such a sequence. This function takes the minimum and maximum values as its inputs, but note that the maximum is not included in the sequence; that is, the sequence goes up to but not including the maximum. Thus, we would have to give 1 and 7 as the minimum and maximum in order to get a sequence of numbers from 1 to 6:

import numpy as np
outcomes = np.arange(1, 7)
outcomes

array([1, 2, 3, 4, 5, 6])

Now let’s create a vector of logical values based on whether the outcome in each position is equal to 1. Remember that == tests for equality of each element in a vector:

outcome1isTrue = outcomes == 1 
outcome1isTrue

array([ True, False, False, False, False, False])

Remember that the simple probability of an outcome is number of occurrences of the outcome divided by the total number of events. To compute a probability, we can take advantage of the fact that TRUE/FALSE are equivalent to 1/0 in Python. The formula for the mean (sum of values divided by the number of values) is thus exactly the same as the formula for the simple probability! So, we can compute the probability of the event by simply taking the mean of the logical vector.

p1isTrue = np.mean(outcome1isTrue)
p1isTrue

0.16666666666666666

Empirical frequency

Let’s walk through how we computed empirical frequency of rain in San Francisco.

First we load the data:

#+
import pandas as pd
SFrain = pd.read_csv('https://raw.githubusercontent.com/statsthinking21/statsthinking21-python/master/notebooks/data/SanFranciscoRain.csv')

# we will remove the STATION and NAME variables 
# since they are identical for all rows

SFrain = SFrain.drop(columns=['STATION', 'NAME'])
SFrain
#-

	DATE	PRCP
0	2017-01-01	0.05
1	2017-01-02	0.10
2	2017-01-03	0.40
3	2017-01-04	0.89
4	2017-01-05	0.01
...	...	...
360	2017-12-27	0.00
361	2017-12-28	0.00
362	2017-12-29	0.00
363	2017-12-30	0.00
364	2017-12-31	0.00

365 rows × 2 columns

We see that the data frame contains a variable called PRCP which denotes the amount of rain each day. Let’s create a new variable called rainToday that denotes whether the amount of precipitation was above zero:

SFrain['rainToday'] = SFrain['PRCP'] > 0
SFrain

	DATE	PRCP	rainToday
0	2017-01-01	0.05	True
1	2017-01-02	0.10	True
2	2017-01-03	0.40	True
3	2017-01-04	0.89	True
4	2017-01-05	0.01	True
...	...	...	...
360	2017-12-27	0.00	False
361	2017-12-28	0.00	False
362	2017-12-29	0.00	False
363	2017-12-30	0.00	False
364	2017-12-31	0.00	False

365 rows × 3 columns

Now we will summarize the data to compute the probability of rain:

pRainInSF = SFrain['rainToday'].mean()
pRainInSF

0.2

Conditional probability

Let’s determine the conditional probability of someone having hearing problems, given that they are over 70 years of age, using the NHANES dataset. First, let’s create a new variable called Over70 that denotes whether each individual is over 70 or not.

from nhanes.load import load_NHANES_data
nhanes_data = load_NHANES_data()

nhanes_data['Over70'] = nhanes_data['AgeInYearsAtScreening'] > 70

Now let’s create a cleaned-up dataset that only includes the over70 variable along with the variable called HaveSeriousDifficultyHearing that denotes whether a person reports having serious hearing difficulty (coded as 1 for “yes” and 0 for “no”).

hearing_data = nhanes_data[['Over70', 'HaveSeriousDifficultyHearing']].dropna()
hearing_data

	Over70	HaveSeriousDifficultyHearing
SEQN
93703.0	False	0.0
93704.0	False	0.0
93705.0	False	0.0
93706.0	False	0.0
93707.0	False	0.0
...	...	...
102952.0	False	0.0
102953.0	False	0.0
102954.0	False	0.0
102955.0	False	0.0
102956.0	False	0.0

8365 rows × 2 columns

First, what’s the probability of being over 70?

p_over_70 = hearing_data['Over70'].mean()
p_over_70

0.10531978481769277

Second, what’s the probability of having hearing problems?

p_hearing_problem = hearing_data['HaveSeriousDifficultyHearing'].mean()
p_hearing_problem

0.06276150627615062

What’s the probability for each combination of hearing problems/no problems and over 70/ not? We can create a table that finds the joint probability for each combination, using the pd.crosstab() function:

joint_table = pd.crosstab(hearing_data.Over70, hearing_data['HaveSeriousDifficultyHearing'], normalize=True)
joint_table

HaveSeriousDifficultyHearing	0.0	1.0
Over70
False	0.859653	0.035027
True	0.077585	0.027735

Finally, what’s the probability of someone having hearing problems, given that they are over 70 years of age? To do this, we limit the computation of the probability of having hearing problems to only include those people who are over 70:

p_hearingproblem_given_over_70 = hearing_data.query('Over70 == True')['HaveSeriousDifficultyHearing'].mean()
p_hearingproblem_given_over_70

0.2633371169125993

Now compute the opposite: What is the probability of being over 70 given that one has a hearing problem?

p_over_70_given_hearingproblem = hearing_data.query('HaveSeriousDifficultyHearing == True')['Over70'].mean()
p_over_70_given_hearingproblem

0.4419047619047619