Go并行gzip压缩:pgzip
jopen
10年前
Go并行gzip压缩。它完全兼容gzip可直接替换 "compress/gzip"。它将待压缩内容拆分成多块然后并行压缩,对于大数据量压缩非常好用。
var b bytes.Buffer w := gzip.NewWriter(&b) w.SetConcurrency(100000, 10) w.Write([]byte("hello, world\n")) w.Close()
性能比较
Compression cost is usually about 0.2% with default settings with a block size of 250k.
Example with GOMAXPROC set to 4 (dual core with 2 hyperthreads)
Compressor | MB/sec | speedup | size | size overhead |
---|---|---|---|---|
gzip (golang) | 15.082MB/s | 1.0x | 6.405.193 | 0% |
pgzip (golang) | 26.736MB/s | 1.8x | 6.421.585 | 0.2% |
bgzf (golang) | 29.525MB/s | 1.9x | 6.875.913 | 7.3% |