Advanced Deduplication Strategies in Polars: Beyond Sort and Drop
Deduplication is one of the most common yet performance-critical operations in data engineering. Whether you are building Change Data Capture (CDC) pipelines, cleaning telemetry clickstreams, or merging customer profiles across data silos, removing duplicate records correctly and efficiently is vital.
In Pandas, deduplication almost always revolves around two familiar patterns: df.drop_duplicates() or sorting first and then dropping (df.sort_values().drop_duplicates()). While simple, these operations in Pandas can quickly become memory and runtime bottlenecks on large datasets because they rely on eager single-threaded execution and global sorting ($O(N \log N)$ complexity).
Polars reimagines deduplication using parallelized hash-based algorithms ($O(N)$ complexity), expressive window functions, and lazy evaluation. In this post, we’ll explore deduplication strategies ranging from basic multi-threaded unique extraction to advanced window-based filtering, conflict resolution, near-duplicate time debouncing, and out-of-core streaming.
1. The Architectural Shift: Pandas vs. Polars Engine
Before diving into syntax, let’s understand why traditional Pandas deduplication techniques struggle at scale and how Polars resolves those bottlenecks.
The Pandas Approach: Global Sorting & Eager Copies
In Pandas, if you want to keep the latest event per user_id, the standard idiom is:
# Pandas: Eager global sort + drop duplicates
df_pandas = df_pandas.sort_values(
by=["user_id", "timestamp"],
ascending=[True, False]
).drop_duplicates(subset=["user_id"], keep="first")
Under the hood:
- Global Sorting ($O(N \log N)$): Pandas reorders every single row across all columns in memory.
- Single-Threaded Scan: Pandas scans through the sorted index sequentially to track seen keys.
- Memory Overhead: The sort step creates full intermediate copies of the DataFrame in memory, often leading to
OutOfMemory(OOM) errors on large datasets.
The Polars Paradigm: Multithreaded Hashing ($O(N)$) & Expressions
Polars decouples sorting from deduplication:
- Parallel Hash Tables ($O(N)$): Polars partitions the input data across available CPU cores, computes hash keys for the target subset columns in parallel, and extracts unique entries without sorting the entire dataset.
- Expression Engine: Rather than creating full intermediate DataFrames, Polars allows you to express deduplication logic inside expressions like
.filter(),.over(), or.group_by(). - Zero-Copy Arrow Memory: Memory allocations are minimized by leveraging Apache Arrow arrays.
2. Core Deduplication API: unique() Demystified
The primary entry point for basic deduplication in Polars is the .unique() method (available on both DataFrames and Expressions).
Syntax & Parameter Comparison
| Feature / Parameter | Pandas drop_duplicates() | Polars unique() |
|---|---|---|
| Primary Method | df.drop_duplicates() | df.unique() |
| Target Columns | subset=['col1', 'col2'] | subset=['col1', 'col2'] |
| Keep Strategy | 'first', 'last', False | 'first', 'last', 'any', 'none' |
| Ordering Control | Preserves index order | maintain_order=True or False |
| Lazy Engine Support | ❌ (Eager only) | ✅ (LazyFrame.unique()) |
Let’s inspect how parameters impact execution using a sample dataset:
import polars as pl
# Sample event dataset with duplicate users and timestamps
data = {
"user_id": [101, 102, 101, 103, 102, 101, 104],
"timestamp": [100, 105, 102, 101, 108, 99, 103],
"status": ["click", "view", "purchase", "view", "click", "view", "purchase"],
"amount": [10.0, None, 50.0, 5.0, 15.0, 10.0, 100.0]
}
df = pl.DataFrame(data)
print(df)
Output:
shape: (7, 4)
┌─────────┬───────────┬──────────┬────────┐
│ user_id ┆ timestamp ┆ status ┆ amount │
│ --- ┆ --- ┆ --- ┆ --- │
│ i64 ┆ i64 ┆ str ┆ f64 │
╞═════════╪═══════════╪══════════╪════════╡
│ 101 ┆ 100 ┆ click ┆ 10.0 │
│ 102 ┆ 105 ┆ view ┆ null │
│ 101 ┆ 102 ┆ purchase ┆ 50.0 │
│ 103 ┆ 101 ┆ view ┆ 5.0 │
│ 102 ┆ 108 ┆ click ┆ 15.0 │
│ 101 ┆ 99 ┆ view ┆ 10.0 │
│ 104 ┆ 103 ┆ purchase ┆ 100.0 │
└─────────┴───────────┴──────────┴────────┘
Strategy A: Keeping the First / Last Row (maintain_order=True)
When order matters (e.g. retaining the exact first or last physical occurrence in the file):
# Keep the first occurrence of each user_id while preserving physical row order
df_first = df.unique(subset=["user_id"], keep="first", maintain_order=True)
print(df_first)
Output:
shape: (4, 4)
┌─────────┬───────────┬──────────┬────────┐
│ user_id ┆ timestamp ┆ status ┆ amount │
│ --- ┆ --- ┆ --- ┆ --- │
│ i64 ┆ i64 ┆ str ┆ f64 │
╞═════════╪═══════════╪══════════╪════════╡
│ 101 ┆ 100 ┆ click ┆ 10.0 │
│ 102 ┆ 105 ┆ view ┆ null │
│ 103 ┆ 101 ┆ view ┆ 5.0 │
│ 104 ┆ 103 ┆ purchase ┆ 100.0 │
└─────────┴───────────┴──────────┴────────┘
Strategy B: High-Throughput Non-Deterministic (keep='any', maintain_order=False)
In many analytical workloads, you simply need one representative record per partition and do not care about insertion order.
Polars introduces keep="any" combined with maintain_order=False. This allows Polars to process chunks in parallel across threads without holding synchronization locks to preserve row order.
# Maximum parallel performance: thread-safe parallel chunk extraction
df_fast = df.unique(subset=["user_id"], keep="any", maintain_order=False)
print(df_fast)
Performance Tip: Setting
maintain_order=Falsecan result in a 2x to 5x speedup on multi-core systems for massive datasets because threads do not need to coordinate row indices.
3. Duplicate Inspection & Masking Expressions
In Pandas, identifying duplicate rows is done via df.duplicated(), which returns a boolean Series.
Polars offers specialized, highly efficient expression-level context methods for duplicate checking:
pl.col("col").is_duplicated(): ReturnsTruefor all rows whose key appears more than once.pl.col("col").is_unique(): ReturnsTrueonly for rows whose key appears exactly once.pl.col("col").is_first_distinct(): ReturnsTruefor the first occurrence of each distinct value.pl.col("col").is_last_distinct(): ReturnsTruefor the last occurrence of each distinct value.
Inspecting Duplicates in Selection Contexts
# Inspect duplicate status without modifying or copying the DataFrame
duplicate_analysis = df.select([
pl.col("user_id"),
pl.col("user_id").is_duplicated().alias("is_dup"),
pl.col("user_id").is_unique().alias("is_uniq"),
pl.col("user_id").is_first_distinct().alias("is_first"),
pl.col("user_id").is_last_distinct().alias("is_last")
])
print(duplicate_analysis)
Output:
shape: (7, 5)
┌─────────┬────────┬─────────┬──────────┬─────────┐
│ user_id ┆ is_dup ┆ is_uniq ┆ is_first ┆ is_last │
│ --- ┆ --- ┆ --- ┆ --- ┆ --- │
│ i64 ┆ bool ┆ bool ┆ bool ┆ bool │
╞═════════╪════════╪═════════╪══════════╪═════════╡
│ 101 ┆ true ┆ false ┆ true ┆ false │
│ 102 ┆ true ┆ false ┆ true ┆ false │
│ 101 ┆ true ┆ false ┆ false ┆ false │
│ 103 ┆ false ┆ true ┆ true ┆ true │
│ 102 ┆ true ┆ false ┆ false ┆ true │
│ 101 ┆ true ┆ false ┆ false ┆ true │
│ 104 ┆ false ┆ true ┆ true ┆ true │
└─────────┴────────┴─────────┴──────────┴─────────┘
Filtering Out All Duplicated Entries (keep='none')
If you want to discard any record that has a duplicate (keeping only pristine, single-occurrence entries):
# Equivalent to Pandas df.drop_duplicates(subset=["user_id"], keep=False)
df_pristine = df.filter(pl.col("user_id").is_unique())
# Or using unique syntax:
# df_pristine = df.unique(subset=["user_id"], keep="none")
print(df_pristine)
4. Advanced Strategy 1: Sorting + Deduplication (“Latest Record” Pattern)
A classic requirement in data engineering is getting the latest state for each entity (e.g. the row with the maximum timestamp per user_id).
Let’s contrast the Pandas approach with four idiomatic Polars alternatives, analyzing the memory and complexity tradeoffs of each.
# Goal: Get the record with the latest timestamp for each user_id
Method 1: Eager/Lazy sort() + unique()
This is the closest analog to Pandas, but significantly faster in Polars.
# Method 1: Sort by timestamp descending, then keep the first unique user_id
res1 = df.sort("timestamp", descending=True).unique(
subset=["user_id"],
keep="first",
maintain_order=True
)
print(res1)
Method 2: Window Expression Filtering with max().over()
Instead of sorting the whole DataFrame, filter rows where timestamp equals the group-level maximum:
# Method 2: Expressive window filtering
res2 = df.filter(
pl.col("timestamp") == pl.col("timestamp").max().over("user_id")
)
print(res2)
Note on Ties: If two rows for the same
user_idshare the exact same maximum timestamp,max().over()will return both rows. If you need strictly one row per group, use Method 3 or Method 4.
Method 3: Window Expression with sort_by() + first().over()
Combine is_first_distinct() or sort_by() inside an .over() window to handle ties deterministically:
# Method 3: Deterministic tie-breaking window filter
res3 = df.filter(
pl.col("timestamp") == pl.col("timestamp").sort_by("timestamp", descending=True).first().over("user_id")
)
print(res3)
Method 4: Aggregation with group_by() + sort_by() (Zero Global Sort)
If you are performing a downstream aggregation step, you can extract the latest values directly within group_by() using sort_by() inside expression aggregations:
# Method 4: GroupBy aggregation with in-group expression sorting
res4 = df.group_by("user_id").agg([
pl.col("timestamp").sort_by("timestamp").last().alias("latest_timestamp"),
pl.col("status").sort_by("timestamp").last().alias("latest_status"),
pl.col("amount").sort_by("timestamp").last().alias("latest_amount")
])
print(res4)
Output:
shape: (4, 4)
┌─────────┬──────────────────┬───────────────┬───────────────┐
│ user_id ┆ latest_timestamp ┆ latest_status ┆ latest_amount │
│ --- ┆ --- ┆ --- ┆ --- │
│ i64 ┆ i64 ┆ str ┆ f64 │
╞═════════╪══════════════════╪═══════════════╪═══════════════╡
│ 101 ┆ 102 ┆ purchase ┆ 50.0 │
│ 102 ┆ 108 ┆ click ┆ 15.0 │
│ 103 ┆ 101 ┆ view ┆ 5.0 │
│ 104 ┆ 103 ┆ purchase ┆ 100.0 │
└─────────┴──────────────────┴───────────────┴───────────────┘
Strategy Comparison Matrix
| Method | Syntax Complexity | Global Sort Required? | Memory Overhead | Handles Timestamp Ties? | Best Use Case |
|---|---|---|---|---|---|
1. sort().unique() | Low | Yes | Medium | Retains first physical row | Quick replacement for Pandas code |
2. max().over() | Low | No | Low | Keeps all tied max rows | Simple window filtering when timestamps are unique |
3. sort_by().first().over() | Medium | No | Low | Retains strictly 1 row | Strict window filtering with custom tie-breakers |
4. group_by().agg() | High | No | Lowest | Customizable per column | Complex CDC/ETL transformations with non-key aggregations |
5. Advanced Strategy 2: Conflict Resolution & Custom Aggregations
In real-world data pipelines, duplicate rows often contain contradictory data across non-key columns (e.g., one row has a null email, while another has an updated phone number).
Standard drop_duplicates() arbitrarily discards non-key values. With Polars, you can resolve conflicts explicitly using expression aggregations.
Real-World Example: Merging User Profile Updates
Suppose we have duplicate user entries coming from multiple syncs:
user_updates = pl.DataFrame({
"user_id": [1, 1, 1, 2, 2],
"email": ["[email protected]", None, "[email protected]", "[email protected]", None],
"phone": [None, "+1-555-0199", None, None, "+1-555-0200"],
"tags": [["vip"], ["buyer"], ["vip", "active"], ["lead"], ["buyer"]]
})
print(user_updates)
shape: (5, 4)
┌─────────┬───────────────┬──────────────┬──────────────────┐
│ user_id ┆ email ┆ phone ┆ tags │
│ --- ┆ --- ┆ --- ┆ --- │
│ i64 ┆ str ┆ str ┆ list[str] │
╞═════════╪═══════════════╪══════════════╪══════════════════╡
│ 1 ┆ [email protected] ┆ null ┆ ["vip"] │
│ 1 ┆ null ┆ +1-555-0199 ┆ ["buyer"] │
│ 1 ┆ [email protected] ┆ null ┆ ["vip", "active"]│
│ 2 ┆ [email protected] ┆ null ┆ ["lead"] │
│ 2 ┆ null ┆ +1-555-0200 ┆ ["buyer"] │
└─────────┴───────────────┴──────────────┴──────────────────┘
Instead of dropping rows, let’s coalesce non-null emails/phones and union all distinct tags:
resolved_users = user_updates.group_by("user_id").agg([
# Take the latest non-null email
pl.col("email").drop_nulls().last().alias("email"),
# Take the first non-null phone number
pl.col("phone").drop_nulls().first().alias("phone"),
# Combine list tags, flatten, and extract unique tags across duplicates
pl.col("tags").explode().unique().alias("all_tags")
])
print(resolved_users)
Output:
shape: (2, 4)
┌─────────┬───────────────┬──────────────┬──────────────────────────┐
│ user_id ┆ email ┆ phone ┆ all_tags │
│ --- ┆ --- ┆ --- ┆ --- │
│ i64 ┆ str ┆ str ┆ list[str] │
╞═════════╪═══════════════╪══════════════╪══════════════════════════╡
│ 1 ┆ [email protected] ┆ +1-555-0199 ┆ ["vip", "buyer", "active"]│
│ 2 ┆ [email protected] ┆ +1-555-0200 ┆ ["lead", "buyer"] │
└─────────┴───────────────┴──────────────┴──────────────────────────┘
This approach guarantees zero data loss during deduplication.
6. Advanced Strategy 3: Near-Duplicate & Time-Window Event Debouncing
In clickstream processing or IoT sensor data, duplicate events are often caused by client-side retries, network glitches, or rapid double-clicking. These duplicates don’t share exact timestamps—they occur within a sliding time window $\Delta t$.
Debouncing Rapid Duplicate Clicks
Suppose we want to filter out click events from the same user that occur within 5 seconds of the previous click:
click_stream = pl.DataFrame({
"user_id": [101, 101, 101, 102, 102],
"timestamp": [100, 102, 120, 200, 204], # 100 & 102 are within 2s of each other!
"action": ["button_click", "button_click", "button_click", "checkout", "checkout"]
})
# Calculate time difference relative to the previous event per user
debounced = click_stream.with_columns(
time_delta = pl.col("timestamp") - pl.col("timestamp").shift(1).over("user_id")
).filter(
# Keep row if it's the first event (null delta) OR if delta > 5 seconds
pl.col("time_delta").is_null() | (pl.col("time_delta") > 5)
).drop("time_delta")
print(debounced)
Output:
shape: (3, 3)
┌─────────┬───────────┬──────────────┐
│ user_id ┆ timestamp ┆ action │
│ --- ┆ --- ┆ --- │
│ i64 ┆ i64 ┆ str │
╞═════════╪═══════════╪══════════════╡
│ 101 ┆ 100 ┆ button_click │
│ 101 ┆ 120 ┆ button_click │
│ 102 ┆ 200 ┆ checkout │
└─────────┴───────────┴──────────────┘
The duplicate click at $t=102$ (delta = 2s) and checkout at $t=204$ (delta = 4s) were successfully debounced!
7. Advanced Strategy 4: Lazy Evaluation & Streaming Out-of-Core Deduplication
When working with datasets larger than RAM (e.g., 50GB Parquet files on a 16GB laptop), loading data eagerly with pl.read_parquet() will cause memory crashes.
Polars handles larger-than-memory deduplication seamlessly via LazyFrame and .sink_parquet() with streaming enabled.
import polars as pl
# 1. Scan dataset lazily (no data is read into memory yet)
lazy_df = pl.scan_parquet("large_clickstream_*.parquet")
# 2. Build the query plan: deduplicate lazily
processed_plan = (
lazy_df
.filter(pl.col("status") == "COMPLETED")
.unique(subset=["user_id", "session_id"], keep="any", maintain_order=False)
)
# 3. Stream query execution directly to output file in batches
processed_plan.sink_parquet(
"deduplicated_output.parquet",
compression="snappy"
)
What Happens Under the Hood?
- Filter Pushdown: Polars filters
status == "COMPLETED"before performing unique hash generation, drastically reducing data size. - Streaming Engine: Data is processed in batches (chunks), building streaming hash tables without loading the entire 50GB dataset into memory at once.
8. Deduplication Cheat Sheet: Pandas vs. Polars
Here is a handy reference guide for mapping Pandas deduplication code to modern Polars expressions:
| Task | Pandas Code | Polars Equivalent |
|---|---|---|
| Drop exact duplicate rows | df.drop_duplicates() | df.unique() |
| Deduplicate on subset | df.drop_duplicates(subset=['a', 'b']) | df.unique(subset=['a', 'b']) |
| Fastest multi-thread deduplication | N/A | df.unique(subset=['a'], keep='any', maintain_order=False) |
| Keep latest record by date | df.sort_values('date').drop_duplicates('id', keep='last') | df.sort('date').unique('id', keep='last', maintain_order=True) |
| Filter latest record without full sort | N/A | df.filter(pl.col('date') == pl.col('date').max().over('id')) |
| Boolean duplicate mask | df['id'].duplicated() | pl.col('id').is_duplicated() |
| Keep only unique records | df.drop_duplicates(subset=['id'], keep=False) | df.filter(pl.col('id').is_unique()) |
| Resolve missing non-key data | N/A | df.group_by('id').agg(pl.col('email').drop_nulls().first()) |
| Debounce time-window duplicates | Custom loop / complex groupby | df.with_columns(dt=pl.col('t').diff().over('id')).filter(pl.col('dt') > 5) |
| Out-of-Core Deduplication | N/A | pl.scan_parquet(...).unique(...).sink_parquet(...) |
Summary
Deduplication in Polars is far more than a simple syntax rewrite of df.drop_duplicates(). By understanding Polars’ expression engine and execution optimizations:
- Use
keep="any"andmaintain_order=Falsefor maximum multi-core CPU throughput when insertion order doesn’t matter. - Replace costly global sorting (
sort_values().drop_duplicates()) with window expressions (max().over()) orgroup_by().agg()expressions. - Handle messy duplicates with conflict resolution aggregations (
drop_nulls(),flatten().unique()). - Leverage lazy evaluation and streaming for out-of-core deduplication on datasets larger than RAM.
Happy Data Wrangling! Explore more in our Polars Track Series.