Efficiently Merging Lists in C#: Techniques for Combining, Ordering, and Eliminating Duplicates

Merging lists is a common task in programming that often arises when handling collections of data. In C#, there are several methods to join two lists while maintaining order and removing duplicates. This tutorial explores different techniques using the .NET framework, focusing on both performance and readability.

Introduction

Suppose you have two lists containing elements such as strings or integers. The goal is to combine these lists into one, ensuring that the resulting list maintains the original order from both input lists and contains no duplicate entries. C# provides several methods in its standard libraries to achieve this efficiently without writing complex algorithms from scratch.

Methods for Combining Lists

Using Concat Method

The Concat method is a LINQ extension provided by System.Linq. It combines two sequences, maintaining their order but does not remove duplicates:

List<string> list1 = new List<string> { "apple", "banana" };
List<string> list2 = new List<string> { "banana", "cherry" };

IEnumerable<string> concatenated = list1.Concat(list2);

The Concat method returns an IEnumerable<T>, which is a read-only sequence. To convert this to a modifiable list, you can use the .ToList() extension:

List<string> combinedList = concatenated.ToList();

Removing Duplicates

To eliminate duplicates while maintaining order, combine Concat with Distinct from LINQ:

Using Union Method

The Union method is another powerful LINQ operation that automatically removes duplicates while preserving the first occurrence of each item based on their natural ordering or a custom comparer.

List<string> list1 = new List<string> { "apple", "banana" };
List<string> list2 = new List<string> { "banana", "cherry" };

IEnumerable<string> unionedList = list1.Union(list2);

The result of Union is an IEnumerable<T> with unique elements. Convert it to a list if necessary:

List<string> distinctCombinedList = unionedList.ToList();

Custom Comparers

For more control over how duplicates are identified, you can implement IEqualityComparer<T> and provide it to the Union method. This approach is useful when dealing with complex types or custom equality logic.

Example for integers using a custom comparer:

List<int> list1 = new List<int> { 1, 2, 3 };
List<int> list2 = new List<int> { 2, 3, 4 };

var result = list1.Union(list2, new CustomComparer());

foreach (int x in result)
{
    Console.WriteLine(x);
}

public class CustomComparer : IEqualityComparer<int>
{
    public bool Equals(int x, int y) => x == y;
    
    public int GetHashCode(int obj) => obj.GetHashCode();
}

Performance Considerations

Concat vs. AddRange: While AddRange modifies the original list directly and may lead to implicit conversions issues when combining different types, Concat returns a new sequence, which is often preferred for functional programming styles.
Memory Usage: Both Concat and Union create a new sequence; however, using .ToList() will allocate additional memory. Consider this trade-off based on your application’s performance requirements.

Conclusion

This tutorial covered methods to combine lists in C#, focusing on maintaining order and removing duplicates efficiently. Using LINQ extensions like Concat, Union, and custom comparers can significantly simplify the task while providing flexibility for various scenarios. By understanding these techniques, developers can write clean, efficient code that handles collections effectively.