huniq – fast duplicate filtering without sorting

huniq is a command-line utility for removing duplicate records from text streams without first sorting the input.

Conventional uniq normally requires duplicate lines to be adjacent. When duplicates appear throughout a file, users often combine sort and uniq to remove them. huniq avoids the sorting step by using hash-based data structures to identify records that have already been encountered.

This approach is particularly useful when the order of input records matters, or when sorting a large dataset would introduce unnecessary processing overhead.

huniq also supports counting repeated records. Unlike its standard filtering mode, the counting mode does not guarantee that records will appear in their original order.

This is free and open source software.

Key Features

• Removes repeated records from unsorted input using hash-based duplicate detection.
• Preserves the order of first occurrences in normal filtering mode.
• Count mode (-c or –count) reports the number of occurrences of each distinct record.
• Supports custom record delimiters with -d or –delim.
• Accepts NUL-delimited records through the -0 or –null option.
• Processes standard input, making it suitable for shell pipelines.
• Avoids the separate sorting stage normally required by sort | uniq for non-adjacent duplicates.

Website: github.com/koraa/huniq
Support:
Developer: Karolin Varner
License: BSD 3-Clause ‘New’ or ‘Revised’ License

huniq is written in Rust. Learn Rust with our recommended free books and free tutorials.

Alternatives to uniq

Explore Linux
	Best Free and Open Source Software - A huge collection of recommended free and open source software covering every major category.
	In-depth Linux Software Reviews - Detailed assessments of Linux software covering features, usability and overall quality.
	The Big List of Active Linux Distros - An extensive collection of actively developed Linux distributions.
	Open Source Alternatives - Alternatives to software and services from Google, Microsoft, Apple, Adobe, IBM, Autodesk, Oracle and others.
	Essential Linux System Tools - Indispensable utilities for monitoring, maintaining and managing Linux systems.
	Linux Productivity Tools - Useful utilities for improving productivity and streamlining everyday tasks.
	Alternatives to Popular CLI Tools - Modern alternatives to familiar command-line utilities supplied with Linux systems.
	Awesome Free Linux Game Tools - Tools, utilities and companion software that improve gaming on Linux.
	Machine Learning on Linux - Machine learning and deep learning software, self-hosted applications and practical tools.
	Saving Money with Linux - Practical ways to reduce electricity consumption and lower computing costs with Linux.
	Linux Candy - Fun, entertaining and sometimes delightfully quirky software from the lighter side of Linux.