What Is Bootstrapping?

You might have heard the word bootstrapping used in business or finance parlance to describe the way that a startup was self-funded and built from the ground up by its founders. That meaning of bootstrapping stems from the phrase “pull yourself up by your bootstraps,” meaning to succeed on your own, without help from anyone else. It harkens back to frontier concepts of self-reliance.

However, according to the linguist Benjamin Zimmer, the true origin of the term bootstrapping was as an ironic statement used to imply that someone had claimed to do something that was actually impossible. The claim made was that a man pulled himself up and over a fence by the force of pulling upward on his own bootstraps. From a scientific perspective, it is pretty clear that you can’t actually lift yourself up in the air and over a tall object by simply yanking on your laces.

Luckily, in the context of statistics and data science, bootstrapping means something more specific and possible. Bootstrapping is a method of inferring results for a population from results found on a collection of smaller random samples of that population, using replacement during the sampling process. This relates back to the original phrase because it belies the notion that the sample is only relying on smaller samples of itself to make calculations on, in order to draw conclusions for the larger population.

This method of bootstrapping does require a considerable amount of computational power, but most computers can easily handle that as long as sample sizes and iterations are kept to reasonable proportions. The main benefit is that bootstrapping saves a lot of time during the phase of conducting research when it is too difficult, time-consuming or costly to survey the entire population being looked at.

Replacement and Sampling

In order to better understand bootstrapping, it is helpful to understand what’s meant by replacement and the impact that replacement has on probability. Replacement means that every time an item is drawn from the pool, that same item remains a part of the sample pool that will be drawn from in the next instance. This rule continues to apply for all subsequent samples. If you were to have completely removed the first sample from the sampling pool without placing it back in and then drew the second sample, the items drawn in that sample would not be as likely to occur as the items in the first sample because the overall population would now be smaller. By removing data with each random sampling, the population measured in subsequent samples would continue to shrink.

A relatable example of how sampling with and without replacement works is the game of drawing names out of a hat for raffle prizes. If the person pulling out the names uses replacement by putting a winner’s name back in the hat before drawing names for the next prize, although it is highly unlikely, replacement can allow that same person to win all of the prizes.

In data science, the process is slightly more complex than just drawing names out of a hat. As you draw samples, you will make statistical calculations based on each one and then find the mean of that statistic across all samples. Once you have all of the statistics for each bootstrapped sample, you can plot them to understand the shape of your data and calculate bias, variance, hypothesis testing and confidence intervals. Because each bootstrapped sample represents a randomly chosen subset of the population, we can make inferences about the entire population.

In a similar way to how bootstrapping is used to infer population results from averaged statistical measures calculated on multiple bags of random samples with replacement, it can be used to infer population results of machine learning models trained on random samples with replacement.

When a machine learning model is built based on bootstrapped data, the model is trained on the bootstrapped data and then tested on the out of bag (OOB) data. The OOB is the portion of the original population that has never been selected in any of the random samples. Because the model has not seen this data before, the model’s quality can be accurately assessed by testing it. If the model performs well on this OOB test data, that indicates that it should also perform similarly well on new data that it’s later exposed to.

How often are your samples never chosen? The size of your train set and the size of your test set can be better understood by knowing that it is calculated using the probability of items being chosen in your random samples. When all of your samples have been bagged, that is as many total samples as the size of your dataset, approximately ⅔ or .632 will have been randomly drawn. Therefore, what remains unchosen or OOB is the total, or 1 – 0.632 = 0.368. What remains out of bag, unsampled, is about one-third of your dataset. You might also see this written as 1/e.

Each bootstrapped sample has an equal number of data points as the size of your dataset. For the question of how many times to bootstrap, 1,000 times is often appropriate, and in some cases more can help to find a high level of certainty about the reliability of your statistics. Bootstrapping can work reasonably well with smaller sample counts too, though results become less reliable as the sample size shrinks further.

Bootstrap Sampling in Machine Learning

Building and improving machine learning models make up a good portion of the work that data scientists do, and bootstrapping shows up in two main ways in that work.

The first is bagging, short for bootstrap aggregating: training multiple versions of a model, each on its own bootstrapped sample of the training data, then combining their predictions. Random Forest is the most common example of this in practice — each tree is trained on a different bootstrapped sample, and the OOB data described above gives you a built-in estimate of how well the model generalizes, without setting aside a separate validation set.

The second is model evaluation. Rather than reporting a single accuracy or F1 score for a model, you can bootstrap the test set to build a confidence interval around that metric. This becomes useful when comparing two models and trying to determine whether one is meaningfully better, or just performed better on this particular test set by chance.

Implementing Bootstrapping in Python

There are a few ways to implement bootstrapping in Python, depending on how much control you want over the process:

• scipy.stats.bootstrap is the standard library-level implementation. It handles percentile, BCa, and basic bootstrap confidence intervals without requiring you to write the resampling loop yourself.
• sklearn.utils.resample from Scikit-learn is useful when you want to generate a single bootstrapped sample as part of a larger pipeline, such as building a bagged ensemble.
• Writing the resampling loop manually with NumPy is worth understanding even if you'd use one of the above in practice, since it makes clear exactly what's happening at each step.

Here's a bootstrapped confidence interval for a mean, using scipy.stats.bootstrap:

import numpy as np
from scipy.stats import bootstrap

data = (np.array([4, 6, 3, 5, 7, 2, 8, 6, 5, 4]),) # data as a one-element tuple

result = bootstrap(data, np.mean, n_resamples=1000, confidence_level=0.95, method="percentile")
print(result.confidence_interval)

And here's the same idea built manually, which is useful for seeing exactly how the percentile method works under the hood:

import numpy as np

def bootstrapped_conf_interval(data, metric, num_runs=1000, conf=0.95):
"""
Calculate a confidence interval for a model performance metric.

Parameters
----------
data : list
Data points in a format acceptable to get_metric
metric : str
One of 'accuracy', 'precision', 'recall', or 'f1-score'
conf : float
Desired confidence level, e.g. 0.95
num_runs : int
Number of bootstrapped samples to draw

Returns
-------
tuple
Confidence interval as (lower, upper), e.g. (.2, .3)
"""
results = []
for i in range(num_runs):
bootstrapped_data = np.random.choice(data, len(data))
results.append(get_metric(bootstrapped_data, metric))
results.sort()

bootstrapped_mean = sum(results) / float(len(results))
x_bar = get_metric(data, metric)

left_index = int(num_runs * (1 - conf) / 2)
delta_interval = [
results[left_index] - bootstrapped_mean,
results[-left_index] - bootstrapped_mean,
]

interval = [delta_interval[0] + x_bar, delta_interval[1] + x_bar]
return interval

(get_metric() here is a stand-in for whichever function you use to compute your chosen metric on a given sample of data.)

Bootstrapping Example

Let’s say you wanted to know how many shoes you have. How would you find out? You would probably just count them—finding out the answer to this question does not require bootstrapping. But what if you wanted to know the average number of shoes that everyone in your office owns?

You work for a big technology company, so your office is large. After speaking to HR, you’re able to conclude that there are 1,000 people total in the building and 200 people on your floor. This is starting to feel a little more feasible to you, and you’re thinking that you could theoretically ask each person how many shoes they have. This may still be impractical and time-consuming, though. Instead, you decide to bootstrap it.

On the first day you survey 50 of the 200 people on your floor (without replacement) and record how many shoes each person has. This is your dataset. Instead of repeating this procedure every day, you take those 50 data points and create a whole lot of bootstrapped samples from them. Each bag (one bootstrapped sample set) has 50 randomly chosen samples, with replacement. Using replacement means that any of them are counted more than once and some are never counted at all. For each of these bags, you measure the mean number of shoes owned.

After you have done this 100 times, you have 100 estimates of the average number of shoes your co-workers own. You can use them to calculate confidence intervals—and be able to conclude something like, “It is 95% likely that the average number of shoes owned by people in this company is between 6 and 10.”

Advantages and Disadvantages of Bootstrapping

Michael R. Chernick, author of the book Bootstrap Methods: A Practitioner’s Guide (second edition, published by Wiley in 2007) said this about bootstrapping:

“A reason for using it is that sometimes you can’t rely on parametric assumptions, and in some situations the bootstrap works better than other non-parametric methods. It can be applied to a wide variety of problems including nonlinear regression, classification, confidence interval estimation, bias estimation, adjustment of p-values and time series analysis to name a few.”

Here are some things to consider when deciding whether to use bootstrapping in machine learning:

Advantages

• Does not require a large sample size. It can be used on small datasets.
• Handles outliers well, without needing to set arbitrary cutoffs to exclude them. Because resampling is random, an outlier is simply less likely to appear in many of the bootstrapped samples. Bayesian bootstrapping takes this further by weighting outliers down rather than excluding them outright.

Disadvantages

• Can require a long amount of computation time.
• Results it provides cannot be understood to be correct with 100% certainty. There will be a margin of error.
• Can fail when distributions are not finite.

Conclusion

Bootstrapping is one of the many methods and techniques that data scientists use. Particularly useful for assessing the quality of a machine learning model, bootstrapping is a method of inferring results for a population from results found on a collection of smaller random samples of the population, using replacement during the sampling process.

The study of data science will help you acquire a rich understanding of statistical and machine learning concepts. Launch your career in data science by visiting Master’s in Data Science to find out more about related degree programs.