semiuniq – remove nearby repeated lines

semiuniq is a command-line utility designed to remove repeated lines that occur close to one another in a text stream.

Unlike traditional uniq, which only removes consecutive duplicates, semiuniq remembers a configurable number of recently encountered lines. If the same line appears again while it is still remembered, that occurrence is discarded.

This approach is useful for processing shell histories, logs and other text where repeated entries tend to appear relatively close together. It avoids sorting the input and preserves the original ordering of records that pass through the filter.

semiuniq does not guarantee that every duplicate will be removed. A line can appear again after its earlier occurrence has been discarded from the recent-history cache.

The cache size allows users to balance duplicate detection against memory consumption.

This is free and open source software.

Key Features

• Removes duplicate lines found within a configurable recent-history window.
• Uses a least recently used cache to remember previously encountered records.
• Accepts a numeric argument specifying how many previous lines are remembered.
• A window size of zero passes the input through without removing duplicates.
• A window size of one behaves similarly to traditional uniq for consecutive duplicates.
• Larger window sizes detect repeated lines separated by other records.
• Preserves input ordering without requiring a sorting operation.
• Stores line hashes rather than complete lines in its cache.

Website: github.com/kljensen/semiuniq
Support:
Developer: Kyle L. Jensen
License: The Unlicense

semiuniq is written in Rust. Learn Rust with our recommended free books and free tutorials.

Alternatives to uniq

Explore Linux
	Best Free and Open Source Software - A huge collection of recommended free and open source software covering every major category.
	In-depth Linux Software Reviews - Detailed assessments of Linux software covering features, usability and overall quality.
	The Big List of Active Linux Distros - An extensive collection of actively developed Linux distributions.
	Open Source Alternatives - Alternatives to software and services from Google, Microsoft, Apple, Adobe, IBM, Autodesk, Oracle and others.
	Essential Linux System Tools - Indispensable utilities for monitoring, maintaining and managing Linux systems.
	Linux Productivity Tools - Useful utilities for improving productivity and streamlining everyday tasks.
	Alternatives to Popular CLI Tools - Modern alternatives to familiar command-line utilities supplied with Linux systems.
	Awesome Free Linux Game Tools - Tools, utilities and companion software that improve gaming on Linux.
	Machine Learning on Linux - Machine learning and deep learning software, self-hosted applications and practical tools.
	Saving Money with Linux - Practical ways to reduce electricity consumption and lower computing costs with Linux.
	Linux Candy - Fun, entertaining and sometimes delightfully quirky software from the lighter side of Linux.