Linux: Parallel file compression
Compressing files on Linux has gone through many iterations over the years. The three main methods to compress a file on Linux are: gzip, bzip2, and xz. All of these compression methods are single threaded, and can be "slow" on todays multi-core CPUs because they do not fully utilize all the cores. Fortunately the community has addressed this and created parallel implementations of each:
Single Threaded | Parallel Version |
---|---|
gzip | pigz |
bzip2 | pbzip2 |
xz | pxz |
All of these are available via Yum in Fedora and CentOS.