smash is a command-line interface tool to smash through to find duplicate files efficiently by slicing a file (or blob) into multiple segments and computing a hash using a fast non-cryptographic algorithm such as xxhash or murmur3.
smash does not support pruning of duplicates or empty files natively and it’s encouraged you vet the output report before pruning via automated tools.
This is free and open source software.
Key Features
- Super fast analysis of large files thanks to slicing.
- Suited for finding duplicates on bandwidth constrained networks, devices or very large files but plenty capable on smaller ones!
- Supports a variety of non-cryptographic algorithms.
- xxhash.
- murmur3.
- SHA-256.
- SHA-512.
- MD5.
- FNV128.
- FNV128a
- Read-only view of the underlying filesystem when analyzing
- Reports on duplicate files & empty (0 byte) files
- Outputs a report in JSON. You can use tools like jq to operate on.
- Used to dedupe multi-TB of astrophysics datasets, images and video content and run regularly to report duplicates.
- Cross-platform support – runs under Linux, FreeBSD, macOS, and Windows.
Website: github.com/thushan/smash
Support:
Developer: Thushan Fernando
License: Apache License 2.0
smash is written in Go. Learn Go with our recommended free books and free tutorials.
Related Software
| Find and Delete Duplicate Files with these CLI Tools | |
|---|---|
| Czkawka | Find duplicate files, big files, empty files, similar images, and much more |
| fdupes | Great CLI tool that's written in C |
| fclones | Efficient duplicate file finder and remover |
| rmlint | Fast tool to remove duplicates and other lint |
| jdupes | Powerful CLI duplicate file finder and 'enhanced' fork of fdupes |
| smash | Find duplicate files super fast |
| rdfind | CLI redundant data find tool written in C++ |
| duff | Command-line utility for finding duplicate files |
| rmdupes | Option to use a reference directory |
| Periscope | Organize storage and safely remove redundant files |
| Go Find Duplicates | Scans directories for duplicate files and directories |
| dskDitto | Parallel duplicate-file detector with interactive menus |
| samanlainen | Delete duplicate files with SHA512 hashing |
| FSlint | Python based CLI and GUI tool |
| sdupes | Fast duplicate file detection utility. |
| dupefi | Duplicate file finder designed with Linux philosophy |
| Dupster | Duplicate file finder |
| duple | Find and remove duplicate files |
| ddh | Directory Differential hTool |
| backdown | Safely and ergonomically remove duplicate files |
Read our verdict in the software roundup.
 LinuxLinks helps you discover the best free and open source software for Linux. Explore our extensive collection of recommended applications, including alternatives to popular proprietary software.We also publish software reviews, Linux tutorials, hardware reviews, free programming books and articles covering the wider open source community. Browse our Linux resources to discover more. Know a useful free and open source Linux application we haven't covered? Suggest a program for inclusion. Has your software been featured on LinuxLinks? Display one of our Featured on LinuxLinks badges on your website or GitHub README. |
Please read our Comment Policy before commenting.