Zstd Compression For EROFS Published: Better Than LZ4 But Higher CPU Costs

[phoronix]

Phoronix: Zstd Compression For EROFS Published: Better Than LZ4 But Higher CPU Costs

As noted recently, EROFS has been exploring Zstd compression support for this open-source read-only Linux file-system. Today the patch was posted for enabling Zstandard use...

https://www.phoronix.com/news/Zstd-EROFS-Compression

[curfew]

"Better than LZ4 but has its problems."

Then it is not better, it's an alternative. Let's stop lying.

[pkese]

I'm kind of sad that btrfs decided not to support LZ4 (they only do zlib, zstd and lzo).

LZ4 is currently the only compression algorithm that can decompress at approximately the speed of modern SSDs (~ 4 GB / s).
Everything else is way below that.

[uid313]

Imagine if instead of all file systems independently implementing Zstd and LZ4 support that all file systems could just use whatever compression algorithm was present in the kernel.

[ddriver]

LZ4 is perfect for me. High speed storage isn't expensive, wasting cpu time to penny pinch storage doesn't seem to be worth it across my use case spectrum. Its merits in network applications is of diminishing returns as well. Best for "compress once and distribute" is optimal, but it is a rather lousy delegating the cost to the end user, which in many cases will be a power constrained device.

Also, this way I don't have to live in disgust with myself for having integrated something that bears Zuck symbology.

[hsiangkao]

Imagine if instead of all file systems independently implementing Zstd and LZ4 support that all file systems could just use whatever compression algorithm was present in the kernel.

Each algorithm has their own internal detail/characteristics, such as sliding window sizes, etc. Some algorithm are suitable for streaming decompression but others not. Also partial/overlapped decompression needs some advanced usages too.

[edxposed]

Are there any CPUs that support ISA level zstd hardware acceleration?

[skeevy420]

Y'all know I'm usually the first person to go LZ4 is better than Zstd, but, in this case, Zstd really is better than LZ4. Seriously, read the name of the file system. Enhanced Read Only File System. Read Only. Because of that you only get hit with the Zstd's slower writes once and then you get the benefit of both higher compression and industry leading decompression speeds. It's about using the right tool for the job at hand and Zstd is a great tool for this use-case.

Cranking up Zstd is how I store mostly read-only data like Steam games.
LZ4 is what I use with Steam game prefixes that read-write data.
Use the right tool for the right job.

[skeevy420]

Are there any CPUs that support ISA level zstd hardware acceleration?

Intel had a hardware encoder in their QATs. I'm not sure about any CPUs. Personally, I'd rather have that over AVX-1024.

[Anux]

I just made some quick test and am not sure if LZ4 is still faster in decompression, I even used --favor-decSpeed and different --fast levels but any zstd --fast level had less decompression time. Need to investigate more.