Adding Rows to Pandas DataFrames

Pandas DataFrames are powerful data structures for tabular data. A common task is adding new rows to an existing DataFrame. This tutorial will cover several ways to accomplish this, from simple appends to more efficient methods for larger datasets.

Understanding the Basics

Before diving into the code, it’s important to understand that Pandas DataFrames are designed for column-oriented operations. While row-wise appending is possible, it’s often less efficient than building the DataFrame directly. However, for many common scenarios, the convenience of row-wise additions outweighs the performance considerations.

Method 1: `pd.concat()`

The pd.concat() function is a versatile tool for joining Pandas objects, including DataFrames. To add a row, you first need to create a new DataFrame containing that row. Then, concatenate it with the original DataFrame.

import pandas as pd

# Example DataFrame
s1 = pd.Series([5, 6, 7])
s2 = pd.Series([7, 8, 9])

df = pd.DataFrame([list(s1), list(s2)], columns=["A", "B", "C"])
print("Original DataFrame:\n", df)

# New row as a list
new_row = [2, 3, 4]

# Create a DataFrame from the new row
df_new = pd.DataFrame([new_row], columns=["A", "B", "C"])

# Concatenate the new DataFrame with the original
df_combined = pd.concat([df_new, df], ignore_index=True)

print("\nDataFrame with added row:\n", df_combined)

The ignore_index=True argument is crucial. It resets the index of the resulting DataFrame, ensuring a contiguous index from 0 to the total number of rows. Without it, the new row would retain its original index (likely 0), leading to duplicate index values.

Method 2: `DataFrame.append()` (Deprecated in newer versions)

Prior to Pandas version 2.0, the append() method was a convenient way to add rows. However, it is now deprecated in favor of pd.concat() due to performance and maintainability concerns.

While it may still work in older code, it’s best to migrate to pd.concat() for future compatibility. The syntax was similar to the pd.concat() example, though slightly more concise.

Method 3: Using `loc` for Direct Assignment

Another approach involves using the loc indexer to directly assign a new row at a specific index. This requires manipulating the index to create space for the new row.

import pandas as pd

# Example DataFrame
s1 = pd.Series([5, 6, 7])
s2 = pd.Series([7, 8, 9])

df = pd.DataFrame([list(s1), list(s2)], columns=["A", "B", "C"])
print("Original DataFrame:\n", df)

# New row as a list
new_row = [2, 3, 4]

# Add a new row at index -1 (which will shift existing rows)
df.loc[-1] = new_row

# Shift the index to account for the new row
df.index = df.index + 1

# Sort the index to put the new row at the top
df = df.sort_index()

print("\nDataFrame with added row:\n", df)

This method is less intuitive than using pd.concat() and can be error-prone if the index manipulation is not done correctly.

Method 4: Pre-allocation and Direct Assignment (For Efficiency)

For situations where you’re adding a large number of rows, or performance is critical, pre-allocating space in the DataFrame and then assigning the rows is the most efficient approach.

import pandas as pd
import numpy as np

# Example DataFrame
s1 = pd.Series([5, 6, 7])
s2 = pd.Series([7, 8, 9])

df = pd.DataFrame([list(s1), list(s2)], columns=["A", "B", "C"])
print("Original DataFrame:\n", df)

# New row as a list
new_row = [2, 3, 4]

# Create an empty DataFrame with pre-allocated space
index = np.array([0, 1, 2])
df_preallocated = pd.DataFrame(columns=["A", "B", "C"], index=index)

# Assign existing data to the pre-allocated DataFrame
df_preallocated.loc[1:] = [list(s1), list(s2)]

# Assign the new row
df_preallocated.loc[0] = new_row

print("\nDataFrame with added row:\n", df_preallocated)

This method avoids the overhead of creating and concatenating DataFrames repeatedly. It’s particularly useful when you know the final size of the DataFrame in advance.

Choosing the Right Method

For adding a single row or a small number of rows, pd.concat() is usually the most straightforward and readable option.
Avoid using DataFrame.append() as it is deprecated.
For performance-critical applications with a large number of rows, pre-allocating space and assigning values directly is the most efficient approach.
Using loc for direct assignment is generally less preferred due to its complexity and potential for errors.