Backend interview prep

Start Learning

Remove Duplicates Using Streams

Java coding interview problem for Java 8 Streams: Remove Duplicates Using Streams.

Removing duplicate elements is one of the most common Java Collections interview problems.

This problem helps understand:

  • Stream API
  • Set behavior
  • distinct() operation
  • Object equality
  • equals() and hashCode()
  • Data cleaning patterns

What are Duplicate Elements?

A duplicate element is a value that appears more than once in a collection.

Example:

Input:

[10,20,30,20,40,10]

Frequency:

10 → 2 times

20 → 2 times

30 → 1 time

40 → 1 time

After removing duplicates:

[10,20,30,40]

Difference Between Removing Duplicates and Finding Duplicates

These two problems look similar but have different goals.

Finding Duplicates

Question:

Which elements appear more than once?

Example:

Input:

[1,2,3,2,4,1]

Output:

[1,2]

Removing Duplicates

Question:

Keep only unique elements.

Example:

Input:

[1,2,3,2,4,1]

Output:

[1,2,3,4]

Understanding Unique Elements

A unique collection means:

Every value appears only once

Example:

Before:

Java

Spring

Java

AWS

After:

Java

Spring

AWS

Why Duplicate Removal is Important?

Duplicate removal is a common operation in enterprise applications.

Data Processing

Clean:

  • Customer records
  • Transaction data
  • Logs

Database Migration

Remove:

  • Duplicate rows
  • Duplicate identifiers

Search Systems

Avoid:

  • Duplicate results
  • Repeated recommendations

Analytics

Generate accurate:

  • Reports
  • Metrics
  • Statistics

Real-World Applications

Customer Management

Input:

Email list

Remove:

Duplicate emails

E-Commerce

Remove duplicate:

  • Product IDs
  • Orders
  • Recommendations

Security Systems

Remove duplicate:

  • User sessions
  • Events

Problem Statement

Given a list of elements, remove duplicate values using Java Streams.

Example 1

Input:

[1,2,3,2,4,1,5]

Output:

[1,2,3,4,5]

Example 2

Input:

["Java","Spring","Java","AWS"]

Output:

["Java","Spring","AWS"]

Java Stream API Overview

Java Stream API provides a functional way to process collections.

Stream flow:

Collection

    ↓

Stream

    ↓

Operations

    ↓

Result

Example:

list.stream()
    .distinct()
    .toList();

Stream Pipeline Concept

A Stream pipeline contains:

Source

 ↓

Intermediate Operations

 ↓

Terminal Operation

Example:

numbers.stream()

.distinct()

.collect()

Flow:

List

 ↓

stream()

 ↓

distinct()

 ↓

collect()

 ↓

Unique List

Approach 1 — Traditional HashSet Approach

Before Streams, developers commonly used Set.

Algorithm

  1. Create an empty Set.
  2. Traverse the list.
  3. Add elements to Set.
  4. Set automatically removes duplicates.

Java Program

import java.util.*;

public class RemoveDuplicatesUsingSet {


    public static List<Integer> removeDuplicates(
            List<Integer> numbers) {


        Set<Integer> unique =
                new LinkedHashSet<>();


        unique.addAll(numbers);


        return new ArrayList<>(
                unique
        );

    }


    public static void main(String[] args) {


        List<Integer> numbers =
                Arrays.asList(
                    1,
                    2,
                    3,
                    2,
                    4,
                    1
                );


        System.out.println(
            removeDuplicates(numbers)
        );

    }

}

Output

[1,2,3,4]

Step-by-Step Explanation

Input:

[1,2,3,2,4,1]

Initial Set:

[]

Add:

1

Set:

[1]

Add:

2

Set:

[1,2]

Add:

3

Set:

[1,2,3]

Add duplicate:

2

Already exists.

Ignored.

Add:

4

Set:

[1,2,3,4]

Add duplicate:

1

Ignored.

Final:

[1,2,3,4]

Why LinkedHashSet?

There are different Set implementations.

HashSet

Provides:

No ordering guarantee

LinkedHashSet

Maintains:

Insertion order

Example:

Input:

Java

Spring

Java

AWS

LinkedHashSet output:

Java

Spring

AWS

Approach 2 — Using Stream distinct()

Java Stream provides:

distinct()

for removing duplicates.

Syntax

stream.distinct()

Example:

numbers.stream()

.distinct()

.toList();

Java Program — Using distinct()

import java.util.*;

public class RemoveDuplicatesUsingStreams {


    public static List<Integer> removeDuplicates(
            List<Integer> numbers) {


        return numbers.stream()

                .distinct()

                .toList();

    }


    public static void main(String[] args) {


        List<Integer> numbers =
                Arrays.asList(
                    10,
                    20,
                    30,
                    20,
                    40,
                    10
                );


        System.out.println(
            removeDuplicates(numbers)
        );

    }

}

Output

[10,20,30,40]

Dry Run — distinct()

Input:

[10,20,30,20,40,10]

First element:

10

Store:

[10]

Second:

20

Store:

[10,20]

Third:

30

Store:

[10,20,30]

Fourth:

20

Already seen.

Skip.

Fifth:

40

Add:

[10,20,30,40]

Sixth:

10

Skip.

Result:

[10,20,30,40]

Removing Duplicate Strings

Example:

Input:

[
"Java",
"Spring",
"Java",
"AWS"
]

Code:

List<String> unique =

languages.stream()

.distinct()

.toList();

Output:

[
Java,
Spring,
AWS
]

Removing Duplicate Objects

For primitive values:

distinct()

works directly.

For custom objects:

Java needs:

equals()

and

hashCode()

Example:

Employee:

id

name

department

Duplicate rule:

Same employee id

Employee Class

class Employee {


    private int id;

    private String name;


    public Employee(
            int id,
            String name) {

        this.id = id;

        this.name = name;

    }


    public int getId(){

        return id;

    }


    @Override
    public boolean equals(
            Object obj) {


        Employee employee =
                (Employee)obj;


        return this.id ==
               employee.id;

    }


    @Override
    public int hashCode(){

        return id;

    }

}

Remove Duplicate Employees

List<Employee> uniqueEmployees =

employees.stream()

.distinct()

.toList();

Why equals() and hashCode()?

Stream distinct() internally uses Set behavior.

Set determines duplicates using:

hashCode()

+

equals()

Without overriding:

Two objects with same data

may be considered different

Preserving Order While Removing Duplicates

Example:

Input:

[5,3,5,2,3]

Expected:

[5,3,2]

Stream:

numbers.stream()

.distinct()

.toList();

Output preserves:

Encounter order

Complexity Analysis

Let:

n = number of elements

Traversal:

O(n)

Hash lookup:

Average:

O(1)

Total Time:

O(n)

Space:

O(n)

because unique values are stored internally.

Advantages

  • Simple syntax.
  • Preserves encounter order.
  • Works with Streams.
  • Good readability.
  • Handles objects with proper equality.

Drawbacks

  • Requires memory for tracking values.
  • Object deduplication requires equals/hashCode.
  • Not suitable when custom duplicate rules are needed.

Deep Dive Into Stream distinct()

The Stream API method:

distinct()

removes duplicate elements from a stream.

How distinct() Works Internally

distinct() internally maintains a Set of already seen elements.

Conceptually:

Stream Elements

        ↓

Check HashSet

        ↓

Already Exists?

        ↓

Yes → Ignore

No  → Keep

Example:

Input:

[10,20,10,30,20]

Internal tracking:

Seen Set:

{}

Process:

10 → Add

20 → Add

10 → Already exists

30 → Add

20 → Already exists

Output:

[10,20,30]

Internal Data Structure

For sequential streams:

HashSet

is used internally.

The duplicate check uses:

hashCode()

+

equals()

Remove Duplicate Objects Using Streams

Consider:

Employee objects:

Employee(101,"John")

Employee(102,"Alice")

Employee(101,"John")

Requirement:

Remove duplicate employees based on:

Employee ID

Employee Class

class Employee {


    private int id;

    private String name;


    public Employee(
            int id,
            String name) {

        this.id = id;

        this.name = name;

    }


    public int getId() {

        return id;

    }


    public String getName() {

        return name;

    }


    @Override
    public String toString() {

        return id + " " + name;

    }

}

Problem

If we directly use:

employees.stream()
.distinct()

it compares complete objects.

We need custom duplicate logic:

Same ID = Duplicate

Approach 1 — Using Set With filter()

Create a Set to track IDs.

Set<Integer> employeeIds =
        new HashSet<>();


List<Employee> uniqueEmployees =

employees.stream()

.filter(
    employee ->
    employeeIds.add(
        employee.getId()
    )
)

.toList();

Dry Run

Input:

101 John

102 Alice

101 Bob

Set:

{}

First:

101

Add:

{101}

Keep.

Second:

102

Add:

{101,102}

Keep.

Third:

101

Already exists.

Remove.

Output:

101 John

102 Alice

Approach 2 — Using Collectors.toMap()

Another powerful approach:

Use a Map where:

Key = Unique Identifier

Example:

Employee ID:

101

Code:

Map<Integer,Employee> employeeMap =

employees.stream()

.collect(

Collectors.toMap(

Employee::getId,

Function.identity(),

(existing,replacement)
        -> existing

)

);


List<Employee> uniqueEmployees =

new ArrayList<>(
    employeeMap.values()
);

Understanding Merge Function

When duplicate key appears:

(existing,replacement)
        -> existing

means:

Keep the first object.

Example:

101 John

101 Bob

Result:

101 John

To keep latest object:

(existing,replacement)
        -> replacement

Result:

101 Bob

Remove Duplicate Employees by Name

Requirement:

Same name = duplicate

Use:

Set<String> names =
        new HashSet<>();


List<Employee> result =

employees.stream()

.filter(
    employee ->
    names.add(
        employee.getName()
    )
)

.toList();

Remove Duplicate Employees by Department

Example:

Employees:

John IT

Alice HR

Bob IT

Requirement:

One employee per department.

Code:

Set<String> departments =
        new HashSet<>();


List<Employee> result =

employees.stream()

.filter(

employee ->
departments.add(
    employee.getDepartment()
)

)

.toList();

Output:

John IT

Alice HR

TreeSet Approach for Sorted Unique Values

Sometimes requirement:

Remove duplicates

+

Sort values

Example:

Input:

[5,2,8,2,1,5]

Expected:

[1,2,5,8]

Use:

List<Integer> result =

numbers.stream()

.collect(

Collectors.toCollection(
    TreeSet::new
)

)

.stream()

.toList();

TreeSet Internals

TreeSet uses:

Red-Black Tree

Provides:

  • Unique elements
  • Sorted order

Complexity:

Insertion:

O(log n)

Custom Key Extraction Approach

A reusable pattern:

static <T> Predicate<T> distinctByKey(
        Function<T,?> keyExtractor) {


    Set<Object> seen =
            ConcurrentHashMap
            .newKeySet();


    return item ->
            seen.add(
                keyExtractor.apply(item)
            );

}

Usage:

employees.stream()

.filter(
    distinctByKey(
        Employee::getId
    )
)

.toList();

This removes duplicates using:

Any custom field

Stream vs Loop Comparison

Feature Loop + Set Stream distinct()
Code More Less
Readability Good Excellent
Custom rules Easy Needs customization
Performance Very similar Very similar
Functional style No Yes

HashSet Internal Working

Duplicate removal depends on HashSet behavior.

When adding:

set.add(value)

Java performs:

Value

 ↓

hashCode()

 ↓

Bucket Location

 ↓

equals()

 ↓

Duplicate Check

Example:

Two employees:

Employee(101)

Employee(101)

If:

hashCode same

+

equals true

then:

Duplicate

Handling Null Values

Input:

[10,20,null,10]

Stream:

numbers.stream()

.filter(
    Objects::nonNull
)

.distinct()

.toList();

Output:

[10,20]

Keeping Null Value

distinct() supports null.

Example:

[10,20,null,10,null]

Output:

[10,20,null]

Parallel Stream Considerations

Example:

parallelStream()
.distinct()

Java maintains correctness.

However:

Parallel processing requires:

  • Additional coordination
  • More memory
  • Thread management

For small collections:

Prefer:

stream()

For very large data:

Evaluate:

parallelStream()

Common Interview Mistakes

Mistake 1

Using:

distinct()

for custom object fields.

Problem:

Two objects may look same but have different references.

Solution:

Use:

equals/hashCode

or

Set with key extractor

Mistake 2

Using HashSet when order matters.

Example:

Need:

Original order

Use:

LinkedHashSet

Mistake 3

Ignoring duplicate business rules.

Ask:

Duplicate based on what field?

Examples:

  • ID
  • Name
  • Email
  • Department

Mistake 4

Using TreeSet without Comparator.

Objects need:

Comparable

or

Comparator

Edge Cases

Case Handling
Empty list Return empty list
Single element Already unique
All duplicates Return one value
Null values Filter or handle
Objects Define equality

Interview Follow-up Questions

Q1. How does Stream distinct() work internally?

Q2. Difference between HashSet and LinkedHashSet?

Q3. Remove duplicate objects by field.

Q4. Remove duplicates and maintain order.

Q5. Remove duplicates and sort values.

Q6. Difference between distinct() and groupingBy()?

Q7. How does HashSet identify duplicates?

Related Java Collection Problems

  • Find Duplicate Elements Using Streams
  • Character Frequency Using Streams
  • Count Word Frequency Using HashMap
  • Convert List to Map
  • Group Employees by Department
  • Find Intersection of Two Lists
  • Custom Comparator Examples

Key Takeaways

Duplicate removal patterns:

Primitive Values

        ↓

stream()

        ↓

distinct()

Custom Objects:

Object List

        ↓

Unique Key

        ↓

Set / Map

        ↓

Deduplicated Result

Recommended approaches:

Simple values

Use:

distinct()

Custom object fields

Use:

filter()

+

Set

or:

Collectors.toMap()

Sorted unique values

Use:

TreeSet

Complexity:

HashSet approach:

Time: O(n)

Space: O(n)

TreeSet approach:

Time: O(n log n)

Space: O(n)

Frequently Asked Interview Questions

Q1. Does distinct() preserve order?

Yes, for ordered streams.

Q2. What does distinct() use internally?

A Set-based mechanism using hashCode and equals.

Q3. How to remove duplicates based on a field?

Use a Set containing the extracted field value.

Q4. How to keep the latest duplicate object?

Use:

Collectors.toMap()

with replacement merge function.

Interview Tip

When asked:

"Remove duplicates using Java Streams."

Explain:

  1. For primitive values use distinct().
  2. For objects define equality or use key-based filtering.
  3. Use LinkedHashSet when order matters.
  4. Use TreeSet when sorted unique output is required.
  5. Discuss HashSet internals and complexity.

For senior Java interviews, discuss:

  • Stream internals.
  • HashSet behavior.
  • Object equality contracts.
  • Collector strategies.
  • Performance trade-offs.

This demonstrates strong understanding of Java Streams, Collections, and real-world data deduplication techniques.