优化的压缩包:compress

jopen 9年前

这个包基于一个优化过的紧缩(Deflate)功能,是gzip/zip/zlib使用的包。

It offers slightly better compression at lower compression settings, and up to 3x faster encoding at highest compression level.

usage

The packages are drop-in replacements for standard libraries. Simply replace the import path to use them:

old import new import
compress/gzip github.com/klauspost/compress/gzip
compress/zlib github.com/klauspost/compress/zlib
archive/zip github.com/klauspost/compress/zip
compress/deflate github.com/klauspost/compress/deflate

You may also be interested in pgzip, which is a drop in replacement for gzip, which support multithreaded compression on big files and the optimized crc32 package used by these packages.

The packages contains the same as the standard library, so you can use the godoc for that: gzip, zip, zlib, flate.

Currently there is only minor speedup on decompression (primarily CRC32 calculation).

deflate optimizations

  • Minimum matches are 4 bytes, this leads to fewer searches and better compression.
  • Stronger hash (iSCSI CRC32) for matches on x64 with SSE 4.2 support. This leads to fewer hash collisions.
  • Literal byte matching using SSE 4.2 for faster string comparisons.
  • Bulk hashing on matches.
  • Much faster dictionary indexing withNewWriterDict()/Reset().
  • Make Bit Coder faster by assuming we are on a 64 bit CPU.
BenchmarkEncodeDigitsSpeed1e4        571065        571799        +0.13%  BenchmarkEncodeDigitsSpeed1e5        3680010       4645932       +26.25%  BenchmarkEncodeDigitsSpeed1e6        34667982      45532604      +31.34%  BenchmarkEncodeDigitsDefault1e4      770694        619535        -19.61%  BenchmarkEncodeDigitsDefault1e5      13682782      6032845       -55.91%  BenchmarkEncodeDigitsDefault1e6      152778738     61443514      -59.78%  BenchmarkEncodeDigitsCompress1e4     771094        620635        -19.51%  BenchmarkEncodeDigitsCompress1e5     13683782      5999343       -56.16%  BenchmarkEncodeDigitsCompress1e6     152648731     61228502      -59.89%  BenchmarkEncodeTwainSpeed1e4         595100        570165        -4.19%  BenchmarkEncodeTwainSpeed1e5         3432796       3376593       -1.64%  BenchmarkEncodeTwainSpeed1e6         31573806      30687755      -2.81%  BenchmarkEncodeTwainDefault1e4       828697        674388        -18.62%  BenchmarkEncodeTwainDefault1e5       11572161      6733885       -41.81%  BenchmarkEncodeTwainDefault1e6       122607013     68998946      -43.72%  BenchmarkEncodeTwainCompress1e4      833297        679738        -18.43%  BenchmarkEncodeTwainCompress1e5      14539831      7372921       -49.29%  BenchmarkEncodeTwainCompress1e6      160019152     77099410      -51.82%    benchmark                            old MB/s     new MB/s     speedup  BenchmarkEncodeDigitsSpeed1e4        17.51        17.49        1.00x  BenchmarkEncodeDigitsSpeed1e5        27.17        21.52        0.79x  BenchmarkEncodeDigitsSpeed1e6        28.85        21.96        0.76x  BenchmarkEncodeDigitsDefault1e4      12.98        16.14        1.24x  BenchmarkEncodeDigitsDefault1e5      7.31         16.58        2.27x  BenchmarkEncodeDigitsDefault1e6      6.55         16.28        2.49x  BenchmarkEncodeDigitsCompress1e4     12.97        16.11        1.24x  BenchmarkEncodeDigitsCompress1e5     7.31         16.67        2.28x  BenchmarkEncodeDigitsCompress1e6     6.55         16.33        2.49x  BenchmarkEncodeTwainSpeed1e4         16.80        17.54        1.04x  BenchmarkEncodeTwainSpeed1e5         29.13        29.62        1.02x  BenchmarkEncodeTwainSpeed1e6         31.67        32.59        1.03x  BenchmarkEncodeTwainDefault1e4       12.07        14.83        1.23x  BenchmarkEncodeTwainDefault1e5       8.64         14.85        1.72x  BenchmarkEncodeTwainDefault1e6       8.16         14.49        1.78x  BenchmarkEncodeTwainCompress1e4      12.00        14.71        1.23x  BenchmarkEncodeTwainCompress1e5      6.88         13.56        1.97x  BenchmarkEncodeTwainCompress1e6      6.25         12.97        2.08x
  • "Speed" is compression level 1
  • "Default" is compression level 6
  • "Compress" is compression level 9
  • Test files are Digits (no matches) and Twain (plain text) .

As can be seen speed on low-matching soucesDigitsare a tiny bit slower at compression level 1, but for default compression it shows a very good speedup.


项目主页:http://www.open-open.com/lib/view/home/1438160460894