Compression

Best Linux Multi-Core Compression Tools

Last Updated on May 23, 2022

With Best Compression

The time taken to complete this test using the best compression option varies significantly within our group of tools. The fastest software is lbzip2 and pigz, both completing the task in under 9 seconds.

plzip is the slowest of the group taking nearly 100 seconds.

Multi-core compression

Multi-core compression

If space is paramount, pxz, plzip, pixz and lrzip offer impressive compression ratios. But pxz and pixz are quicker to complete.

Recent versions of pigz include the Zopfli engine which achieves higher compression than other zlib implementations but takes much longer to complete the compression. pigz uses Zopfli with the -11 flag.

Using Zopfli, pigz takes a whopping 14 minutes 25 seconds to complete the test. And while the compression ratio is better, the compressed file weights in at 104MB (as opposed to 109MB with the -9 flag). That’s still larger than the output from most of the multi-core tools with their fastest compression option.

pxz also has an extreme option which is triggered with the -e flag. Using the extreme option is designed to improve the compression ratio by using more CPU time. But compression ratio wasn’t improved in our tests. With the -9 flag, the tarball is 62MB. Yet, using the -e flag, the tarball was 65MB. We’ll need to run more tests to determine if this is just an anomaly.

Next page: Page 5 – lrzip with different compression methods

Pages in this article:
Page 1 – Introduction
Page 2 – Charts with Default Compression
Page 3 – Charts with Fastest Compression
Page 4 – Charts with Best Compression
Page 5 – lrzip with Different Compression Methods

Methodology used for the tests

We took a 537MB tarball of a popular source package. The tarball was copied to RAM (/dev/shm), and the tests ran in RAM on a quad-core CPU without hyper-threading (Core i5-2500K), with no X server running, and under negligible load.

Each test was run three times with the latest version (at the time of writing) of each multi-core compression tool. The average results are recorded in the charts above. The tests show the relative difference between the multi-core compression tools. They are for indicative purposes only.


Learn more about the features offered by the multi-core compression tool. We’ve compiled a dedicated page for each tool explaining, in detail, the features they offer.

Multi-Core Compression Tools
pigzParallel implementation of gzip. It's a fully functional replacement for gzip
PBZIP2Parallel implementation of the bzip2 block-sorting file compressor
PXZRuns LZMA compression on multiple cores and processors
lbzip2Parallel bzip2 compression utility, suited for serial and parallel processing
plzipMassively parallel (multi-threaded) lossless data compressor based on lzlib
lrzipCompression utility that excels at compressing large files
pixzParallel indexing XZ compression, fully compatible with XZ. LZMA and LZMA2
Best Free and Open Source SoftwareRead our complete collection of recommended free and open source software. Our curated compilation covers all categories of software.

The software collection forms part of our series of informative articles for Linux enthusiasts. There are hundreds of in-depth reviews, open source alternatives to proprietary software from large corporations like Google, Microsoft, Apple, Adobe, IBM, Cisco, Oracle, and Autodesk.

There are also fun things to try, hardware, free programming books and tutorials, and much more.
Subscribe
Notify of
guest

This site uses Akismet to reduce spam. Learn how your comment data is processed.

5 Comments
Oldest
Newest Most Voted
Inline Feedbacks
View all comments
Andy Turfer
Andy Turfer
4 years ago

Thank you so much! I’m going to try out some of these.

Michael D
Michael D
2 years ago

Some comparison between them would be very usefull

Mark
Mark
2 years ago

I did a similar study a few years ago and ended up using pbzip2 as my go-to compression utility.

The main reason is that it can do multi-core de-compression as well, unlike pigz.

The compression algorithm is fairly slow, so it works best when you have 30+ cores to throw at it.

Keep in mind that to use pbzip2 to de-compress with multiple cores, you need to compress with pbzip2 first. It adds some hints to the file to let the decompression know how to split up the work properly.

coucou
coucou
2 months ago
Reply to  Mark

Seems you brought me a solution, thanks !
Do you know if initramfs is using a single threaded kernel bzip2 routine or my /bin/pbzip2 for decompression at runtime ?

Cordialement.

coucou
coucou
2 months ago

Multicore compression : great ! But … all decompressions are done on a SINGLE core ! Why ?
I tried option to select the number of threads but in vain.

Any idea to do multicore decompression ?

Cordialement.

One sided Multicore