zstd压缩与lz4压缩比较

原创 2024-01-09 11:19:43 1.3w 阅读 AI写同款

收录于

开源库

码农诗人

码龄11年墨衍会员 · AI 创作全网分发

987粉丝84篇原创

关注

Zstd 压缩性能分析 About Zstd Zstd，全称 Zstandard，是 Facebook 于 2016 年开源的新无损压缩算法。与 zlib、lz4、xz 等当前流行的压缩算法不同，Zstd 寻求一种压缩性能与压缩率通吃的方案，而实际上它也确实做到了。在由官方所列出的表格中，可以看到，Zstd 不仅具备优秀的压缩性能，在压缩率上也有非常亮眼的表现。 [外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-RXOO1Z8l-1596090419061)(https://engineering.fb 阅读详情

AI摘要

本文详细介绍了如何使用C语言调用Zstd和LZ4压缩算法，并通过实际业务数据展示了两者在压缩率和速度上的差异，Zstd压缩率更高但解压速度较慢，而LZ4则以速度见长。

最近由于业务需求需要对数据进行压缩，相关压缩算法介绍的都比较多，比较详细，笔者在这里就不赘述。本篇主要针对C语言调用zstd、lz4压缩算法的接口调用,并对其做简单的比较。下述案例中的源码也可通过该链接下载：压缩算法包

ZSTD源码编译及测试

源码编译

1、获取zstd源码，网址：https://github.com/facebook/zstd/，打开网址如下操作：

点击红色获取Releases版本源码，拉到页面最下边如下图：

2、将源码移到自己的环境解压（tar -zxcf zstd-1.5.2.tar.gz）、编译(make)，生成zstd静态库、动态库及程序调用所需头文件。如下图所示：

案例调用

1、创建一个与刚解压的zstd-1.5.2目录同级的目录zstd_demo；将案例代码与Makefile文件放入目录中，如下图所示：

2、zstd_demo.c与Makefile文件如下：

/********************************************
Date	:	2022-05-17 
Author	:	lijd
Func	:	用生产业务数据测试zstd压缩算法
********************************************/

#include <stdio.h>
#include <sys/time.h>
#include "zstd.h"

#include <string.h>

char g_str[] ="2323024C5A474E41374535334C583636363032330101006016050D121015020013FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF0006D1F4A901D0C74FFFFFFFFF880002001CFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF88F001000C0100000006D1F4A901D0C74FA12323034C5A474E41374535334C583636363032330101006016050C0B012E020006FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF0006D1F4B001D0C8EFFFFFFFFF880002001CFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF88F001000C0100000006D1F4B001D0C8EF872323034C5A474E41374535334C583636363032330101006016050C0B012F020007FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF0006D1F4B701D0C8E7FFFFFFFF880002001CFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF88F001000C0100000006D1F4B701D0C8E7872323034C5A474E41374535334C583636363032330101006016050C0B0130020008FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF0006D1F4B701D0C8E0FFFFFFFF880002001CFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF88F001000C0100000006D1F4B701D0C8E0972323024C5A474E41374535334C583636363032330101006016050D121017020014FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF0006D1F4A901D0C74FFFFFFFFF880002001CFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF88F001000C0100000006D1F4A901D0C74FA42323034C5A474E41374535334C583636363032330101006016050C0B0131020009FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF0006D1F4B701D0C8DDFFFFFFFF880002001CFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF88F001000C0100000006D1F4B701D0C8DD972323024C5A474E41374535334C583636363032330101006016050D121018020015FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF0006D1F4A901D0C74FFFFFFFFF880002001CFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF88F001000C0100000006D1F4A901D0C74FAA2323034C5A474E41374535334C583636363032330101006016050C0B013202000AFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF0006D1F4B501D0C8D6FFFFFFFF880002001CFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF88F001000C0100000006D1F4B501D0C8D6972323034C5A474E41374535334C583636363032330101006016050C0B013302000BFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF0006D1F4B001D0C8CEFFFFFFFF880002001CFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF88F001000C0100000006D1F4B001D0C8CE97";

// 十六进制转字符串
int HexToStr(char *p_hexstr, int iHexLen, char *pdststr)
{
	int iret = 0;
	while(p_hexstr != NULL && pdststr != NULL && iHexLen > 1)
	{
		char cTemp = '0';
		
		// printf("1:%c, 2:%c\n", p_hexstr[0], p_hexstr[1]);
		
		// 小写统一转大写
		if(p_hexstr[0] >= '0' && p_hexstr[0] <= '9')
		{
			cTemp = p_hexstr[0] - '0';
		}
		else if(p_hexstr[0] >= 'A' && p_hexstr[0] <= 'F')
		{
			cTemp = p_hexstr[0] - 'A' + 10;
		}
		else if(p_hexstr[0] >= 'a' && p_hexstr[0] <= 'f')
		{
			cTemp = p_hexstr[0] - 'a' + 10;
		}
		else
		{
			printf("the hex str is error!\n");
			break;
		}
		
		*pdststr = cTemp * 16;
		
		if(p_hexstr[1] >= '0' && p_hexstr[1] <= '9')
		{
			cTemp = p_hexstr[1] - '0';
		}
		else if(p_hexstr[1] >= 'A' && p_hexstr[1] <= 'F')
		{
			cTemp = p_hexstr[1] - 'A' + 10;
		}
		else if(p_hexstr[1] >= 'a' && p_hexstr[1] <= 'f')
		{
			cTemp = p_hexstr[1] - 'a' + 10;
		}
		else
		{
			printf("the hex str is error!\n");
			break;
		}
		*pdststr += cTemp;
		
		// printf("---iHexLen:%d, pdststr:%c\n", iHexLen, *pdststr);
		
		iHexLen -= 2;
		p_hexstr += 2;
		pdststr++;
		iret++;
	}
	
	// printf("iret : %d\n", iret);
	
	return iret;
}

// 字符串转十六进制
int StrToHex(const char *p_strstr, int iStrLen, char *pdststr)
{
	int index_str = 0, index_hex = 0;

	const char cHex[] = "0123456789ABCDEF";
	while(index_str < iStrLen && p_strstr != NULL && pdststr != NULL)
	{
		pdststr[index_hex++] = cHex[((UCHAR)p_strstr[index_str])/16];
		pdststr[index_hex++] = cHex[((UCHAR)p_strstr[index_str++])%16];
	}

	return index_hex;
}

int main()
{
	struct timeval curTime;
	char src[2048] = {0};
	char dst[2048] = {0}, src2[2048] = {0};
	
	int iStrlen = HexToStr(g_str, strlen(g_str), src);
	
	printf("业务数据原始长度 : %d 字节\n", iStrlen);
	
	gettimeofday(&curTime, NULL);
    int time_usec = curTime.tv_usec;
	
	int cSize = ZSTD_compress(dst, 2048, src, iStrlen, 3);
	
	gettimeofday(&curTime, NULL);
    time_usec = curTime.tv_usec - time_usec;
	
	printf("ZSTD 压缩所用时间 : %u 微秒, 压缩后长度 : %d 字节\n", time_usec, cSize);
	
	gettimeofday(&curTime, NULL);
	time_usec = curTime.tv_usec;
	
	int cSize2 = ZSTD_decompress(src2, 2048, dst, cSize);
	
	gettimeofday(&curTime, NULL);
    time_usec = curTime.tv_usec - time_usec;
	printf("压缩后数据解压所需时间 : %u 微秒, 解压后数据长度 : %d 字节\n", time_usec, cSize2);
	
	return 0;
}

# Func: zstd_demo
# Auth: lijd
# Date: 2022/05/17

CC       := gcc
TARGET   := zstd_demo
INCLUDES := ../zstd-1.5.2/lib/
LIBDIRS  := ../zstd-1.5.2/lib/
LIBS     := zstd
FLAGES   := $(addprefix -I, $(INCLUDES)) $(addprefix -L, $(LIBDIRS)) $(addprefix -l, $(LIBS))
SRCS  	 := $(wildcard *.c)

$(TARGET): $(SRCS)
	$(CC) -g -Wall -o $@ $^ $(FLAGES)
	@echo "==================================== Build OK!!! ===================================="

.PHONY: clean
clean:
	@rm -rf $(TARGET)

3、编译成功后，运行可能找不到依赖的动态库，需要将环境变量设置一下，注意笔者用的这种设置方式只是暂时的。如下图：

lz4源码编译及测试

源码编译

         1、获取zstd源码，网址：Index of /ubuntu/pool/main/l/lz4，打开网址如下操作：

        2、将源码移到自己的环境解压（tar -zxcf lz4_1.9.2.orig.tar.gz）、编译(make)，生成lz4静态库、动态库及程序调用所需头文件。如下图所示：

案例调用

        1、创建一个与刚解压的lz4_1.9.2目录同级的目录lz4_demo；将test.c与Makefile文件放入目录中，代码只有调用压缩解压的函数接口有变动，其余都一样，Makefile只有小的改动。这里只给出变动的main函数如下：
int main()
{
	struct timeval curTime;
	char src[2048] = {0};
	char dst[2048] = {0}, src2[2048] = {0};
	
	int iStrlen = HexToStr(g_str, strlen(g_str), src);
	
	printf("业务数据原始长度 : %d 字节\n", iStrlen);
	
	gettimeofday(&curTime, NULL);
    int time_usec = curTime.tv_usec;
	
	int cSize = LZ4_compress_default(src, dst, iStrlen, 2048);
	
	gettimeofday(&curTime, NULL);
    time_usec = curTime.tv_usec - time_usec;
	
	printf("LZ4 压缩所用时间 : %u 微秒, 压缩后长度 : %d 字节\n", time_usec, cSize);
	
	gettimeofday(&curTime, NULL);
	time_usec = curTime.tv_usec;
	
	int cSize2 = LZ4_decompress_safe(dst, src2, cSize, 2048);
	
	gettimeofday(&curTime, NULL);
    time_usec = curTime.tv_usec - time_usec;
	printf("压缩后数据解压所需时间 : %u 微秒, 解压后数据长度 : %d 字节\n", time_usec, cSize2);
	
	return 0;
}
2、编译成功后运行如下图：