smash is a command-line interface tool to smash through to find duplicate files efficiently by slicing a file (or blob) into multiple segments and computing a hash using a fast non-cryptographic algorithm such as xxhash or murmur3.
smash does not support pruning of duplicates or empty files natively and it’s encouraged you vet the output report before pruning via automated tools.
This is free and open source software.
Key Features
- Super fast analysis of large files thanks to slicing.
- Suited for finding duplicates on bandwidth constrained networks, devices or very large files but plenty capable on smaller ones!
- Supports a variety of non-cryptographic algorithms.
- xxhash.
- murmur3.
- SHA-256.
- SHA-512.
- MD5.
- FNV128.
- FNV128a
- Read-only view of the underlying filesystem when analyzing
- Reports on duplicate files & empty (0 byte) files
- Outputs a report in JSON. You can use tools like jq to operate on.
- Used to dedupe multi-TB of astrophysics datasets, images and video content and run regularly to report duplicates.
- Cross-platform support – runs under Linux, FreeBSD, macOS, and Windows.
Website: github.com/thushan/smash
Support:
Developer: Thushan Fernando
License: Apache License 2.0
smash is written in Go. Learn Go with our recommended free books and free tutorials.
Related Software
| Find and Delete Duplicate Files with these CLI Tools | |
|---|---|
| Czkawka | Find duplicate files, big files, empty files, similar images, and much more |
| fdupes | Great CLI tool that's written in C |
| fclones | Efficient duplicate file finder and remover |
| rmlint | Fast tool to remove duplicates and other lint |
| jdupes | Powerful CLI duplicate file finder and 'enhanced' fork of fdupes |
| smash | Find duplicate files super fast |
| rdfind | CLI redundant data find tool written in C++ |
| duff | Command-line utility for finding duplicate files |
| rmdupes | Option to use a reference directory |
| Periscope | Organize storage and safely remove redundant files |
| Go Find Duplicates | Scans directories for duplicate files and directories |
| samanlainen | Delete duplicate files with SHA512 hashing |
| FSlint | Python based CLI and GUI tool |
| sdupes | Fast duplicate file detection utility. |
| dupefi | Duplicate file finder designed with Linux philosophy |
| Dupster | Duplicate file finder |
| duple | Find and remove duplicate files |
| ddh | Directory Differential hTool |
| backdown | Safely and ergonomically remove duplicate files |
Read our verdict in the software roundup.
 Explore our comprehensive directory of recommended free and open source software. Our carefully curated collection spans every major software category.This directory is part of our ongoing series of informative articles for Linux enthusiasts. It features hundreds of detailed reviews, along with open source alternatives to proprietary solutions from major corporations such as Google, Microsoft, Apple, Adobe, IBM, Cisco, Oracle, and Autodesk. You’ll also find interesting projects to try, hardware coverage, free programming books and tutorials, and much more. Discovered a useful open source Linux program that we haven’t covered yet? Let us know by completing this form. |