Removing Duplicate Rows in R Data Frames

Identifying and Removing Duplicate Data in R

Data cleaning is a crucial step in any data analysis workflow. A common task is identifying and removing duplicate rows from a data frame. This tutorial will guide you through various methods for achieving this in R, from basic approaches to leveraging powerful packages like dplyr and data.table.

What are Duplicate Rows?

Duplicate rows are those that have identical values across all (or a specified subset of) columns. Identifying and removing these duplicates is essential to avoid biased results and ensure data accuracy.

Base R Approach: duplicated()

The base R duplicated() function is a straightforward way to identify duplicate rows. It returns a logical vector indicating which rows are duplicates of previous rows.

# Create a sample data frame
df <- data.frame(
  col1 = c(1, 2, 2, 3, 3, 3),
  col2 = c("A", "B", "B", "C", "C", "C")
)

# Identify duplicate rows
duplicates <- duplicated(df)

# Print the duplicate rows
print(df[duplicates, ])

# Remove duplicate rows, keeping the first occurrence
unique_df <- df[!duplicates, ]

# Print the data frame with duplicates removed
print(unique_df)

In this example, duplicated(df) returns FALSE for the first row, TRUE for the second (as it’s a duplicate of the first), and so on. By negating this logical vector (!duplicates), we select only the unique rows.

Keeping the Last Occurrence:

By default, duplicated() keeps the first occurrence of a duplicate row. If you want to keep the last occurrence, you can sort the data frame before applying duplicated():

# Sort the data frame
sorted_df <- df[order(df$col1), ] #Sort by col1 for demonstration

# Identify duplicates after sorting
duplicates <- duplicated(sorted_df)

# Remove duplicates, keeping the last occurrence (now the first)
unique_df <- sorted_df[!duplicates, ]

print(unique_df)

Using dplyr for Flexible Duplicate Removal

The dplyr package provides a more elegant and flexible approach to duplicate removal, especially when you want to remove duplicates based on specific columns.

library(dplyr)

# Create a sample data frame
df <- data.frame(
  col1 = c(1, 2, 2, 3, 3, 3),
  col2 = c("A", "B", "B", "C", "C", "C"),
  col3 = c(10, 20, 20, 30, 30, 30)
)

# Remove duplicates based on col1 and col2, keeping all columns
unique_df <- df %>% distinct(col1, col2, .keep_all = TRUE)

# Print the result
print(unique_df)

#Remove complete duplicates (across all columns)
unique_df_all <- df %>% distinct()

print(unique_df_all)

The distinct() function, combined with the .keep_all = TRUE argument, allows you to specify which columns to consider when identifying duplicates while preserving all columns in the resulting data frame. Without .keep_all = TRUE, only the columns specified in distinct() will be retained.

Leveraging data.table for Performance

For very large datasets, the data.table package offers significant performance advantages.

library(data.table)

# Create a sample data frame
df <- data.frame(
  col1 = c(1, 2, 2, 3, 3, 3),
  col2 = c("A", "B", "B", "C", "C", "C"),
  col3 = c(10, 20, 20, 30, 30, 30)
)

# Convert to data.table
dt <- as.data.table(df)

# Remove duplicates based on col1 and col2
unique_dt <- unique(dt, by = c("col1", "col2"))

# Print the result
print(unique_dt)

The unique() function in data.table, with the by argument, efficiently identifies and removes duplicates based on the specified columns. data.table is especially advantageous when dealing with datasets that exceed the memory capacity of your machine, as it operates more efficiently with large data.

Choosing the Right Method

• For small to medium-sized datasets and straightforward duplicate removal (based on all columns), base R’s duplicated() function is a simple and effective solution.
• For more complex scenarios where you need to remove duplicates based on specific columns while keeping other columns, dplyr‘s distinct() function offers greater flexibility and readability.
• For very large datasets or when performance is critical, data.table provides the best performance and scalability.