S3

s5cmd – parallel S3 and filesystem tool

s5cmd is a command-line utility for working with Amazon S3, S3-compatible object storage and local filesystems. It is designed for workloads involving large numbers of objects, where the overhead of executing individual storage commands can make conventional file operations unnecessarily slow.

The program uses concurrent workers to perform transfers and object-management operations in parallel. Its command-file facility allows many operations to be submitted together, avoiding the overhead of repeatedly launching separate processes. This makes s5cmd particularly useful for data migrations, backups, processing large collections of files and automating cloud storage workflows.

s5cmd also supports synchronisation between local directories and object storage, wildcard-based selection and filtering. Administrators can preview proposed operations before executing them, obtain structured output for further processing and control the number of concurrent workers.

This is free and open source software.

Key Features

  • List buckets and objects in Amazon S3 and compatible storage services.
  • Upload, download, copy, move, rename and delete objects.
  • Create and remove storage buckets.
  • Synchronise local directories and S3 object prefixes.
  • Parallel execution of storage operations using configurable worker pools.
  • Control concurrency for multipart uploads and downloads.
  • Execute batches of commands from files or standard input.
  • Wildcard matching across multiple levels of object prefixes.
  • Include and exclude filters for selecting files and objects.
  • Dry-run mode for previewing operations without modifying storage.
  • Structured JSON logging for processing command results.
  • Query JSON object contents using S3 Select SQL expressions.
  • Summarise object counts and storage usage.
  • Server-side encryption using AWS Key Management Service.
  • Manage access control lists during object transfers.
  • Support for AWS profiles, IAM roles and anonymous requests.
  • Configurable retry handling for transient storage errors.
  • Support for Google Cloud Storage and other S3-compatible services.
  • Cross-platform support – runs on Linux, macOS, and Windows.

Website: github.com/peak/s5cmd
Support:
Developer: Peak
License: MIT License

s5cmd help

s5cmd is written in Go. Learn Go with our recommended free books and free tutorials.


Related Software

S3 Terminal Tools
s5cmdS3 and local filesystem execution tool
clawsTerminal user interface for managing AWS resources
s3cmdManage objects in Amazon S3 storage
RcloneCommand line program to sync files and directories
STUBrowse buckets and objects directly from your terminal, preview files, and more
streeVisualize the directory tree structure of an S3 bucket
MinIO ClientWork with object storage from a Unix-style shell
s3tuiSimple S3 CLI client for file transfers
S3ryAWS S3 interactive terminal client

Read our verdict in the software roundup.


Best Free and Open Source Software LinuxLinks helps you discover the best free and open source software for Linux. Explore our extensive collection of recommended applications, including alternatives to popular proprietary software.

We also publish software reviews, Linux tutorials, hardware reviews, free programming books and articles covering the wider open source community. Browse our Linux resources to discover more.

Know a useful free and open source Linux application we haven't covered? Suggest a program for inclusion.

Has your software been featured on LinuxLinks? Display one of our Featured on LinuxLinks badges on your website or GitHub README.
Subscribe

Please read our Comment Policy before commenting.

Notify of
guest
0 Comments
Oldest
Newest Most Voted