When working with large datasets in pandas, it’s often necessary to split the data into training and test sets for machine learning model development. In this tutorial, we’ll explore how to create random samples from a pandas DataFrame for training and testing.
Introduction to Train-Test Split
The train-test split is a fundamental concept in machine learning that involves dividing a dataset into two parts: a training set and a test set. The training set is used to train the model, while the test set is used to evaluate its performance. By splitting the data in this way, we can prevent overfitting and get a more accurate estimate of the model’s performance on unseen data.
Using Scikit-Learn’s train_test_split Function
One of the most convenient ways to split a pandas DataFrame into training and test sets is by using scikit-learn’s train_test_split function. This function takes in the DataFrame, as well as the proportion of data to be used for testing (default is 0.25), and returns two DataFrames: one for training and one for testing.
from sklearn.model_selection import train_test_split
# Create a sample DataFrame
import pandas as pd
import numpy as np
df = pd.DataFrame(np.random.randn(100, 2))
# Split the data into training and test sets (80% for training)
train_df, test_df = train_test_split(df, test_size=0.2)
print("Training set shape:", train_df.shape)
print("Test set shape:", test_df.shape)
Using Pandas’ sample Method
Another way to split a DataFrame into training and test sets is by using pandas’ sample method. This method allows you to specify the fraction of data to be used for sampling, as well as a random seed for reproducibility.
# Split the data into training and test sets (80% for training)
train_df = df.sample(frac=0.8, random_state=42)
# Create the test set by dropping the indices of the training set
test_df = df.drop(train_df.index)
print("Training set shape:", train_df.shape)
print("Test set shape:", test_df.shape)
Using NumPy’s random Function
You can also use NumPy’s random function to split a DataFrame into training and test sets. This method involves creating a random mask for the data, where each row is assigned a random value between 0 and 1.
# Create a random mask for the data (80% of rows will be True)
mask = np.random.rand(len(df)) < 0.8
# Split the data into training and test sets using the mask
train_df = df[mask]
test_df = df[~mask]
print("Training set shape:", train_df.shape)
print("Test set shape:", test_df.shape)
Tips and Best Practices
When splitting your data into training and test sets, keep the following tips in mind:
- Use a suitable proportion of data for testing (e.g., 20%).
- Set a random seed for reproducibility.
- Ensure that your test set is representative of the overall population.
- Avoid using too small or too large of a test set.
By following these guidelines and using one of the methods described above, you can easily split your pandas DataFrame into training and test sets for machine learning model development.