Hive 文件压缩格式对比

最新推荐文章于 2025-08-05 14:59:22 发布
最新推荐文章于 2025-08-05 14:59:22 发布
阅读量1.4k 收藏 1
点赞数
CC 4.0 BY-SA版权
分类专栏： HIve
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
本文链接：https://blog.youkuaiyun.com/xiaoxiongaa0/article/details/89400121
本文详细介绍了在Hadoop环境下配置不同压缩格式的方法，包括Gzip、BZip2、Snappy，并通过WordCount示例展示了压缩效果。同时，通过Hive SQL实现了对小文件的数据加载和压缩输出，对比了Snappy、BZip2和Gzip在数据大小、压缩效率上的差异。 
[hadoop@hadoop004 hadoop]$ vim core-site.xml

<configuration>
<property>
  <name>fs.defaultFS</name>
  <value>hdfs://hadoop004:9000</value>
</property>
<property>
                  <name>io.compression.codecs</name>
                  <value>org.apache.hadoop.io.compress.GzipCodec,
                        org.apache.hadoop.io.compress.DefaultCodec,
                        org.apache.hadoop.io.compress.BZip2Codec,
                        org.apache.hadoop.io.compress.SnappyCodec
                  </value>
      </property>
</configuration>
[hadoop@hadoop004 hadoop]$ vim mapred-site.xml

<configuration>
<property>
    <name>mapreduce.framework.name</name>
    <value>yarn</value>
</property>
<!-- 配置 Map段输出的压缩,snappy-->
  <property>
      <name>mapreduce.map.output.compress</name>
      <value>true</value>
  </property>
  <property>
      <name>mapreduce.map.output.compress.codec</name>
      <value>org.apache.hadoop.io.compress.SnappyCodec</value>
   </property>

<!--开启MapReduce输出文件压缩-->
<property>
<name>mapreduce.output.fileoutputformat.compress</name>
<value>true</value>
</property>
<property>
<name>mapreduce.output.fileoutputformat.compress.codec</name>
<value>org.apache.hadoop.io.compress.BZip2Codec</value>
</property>

</configuration>
[hadoop@hadoop004 sbin]$ hadoop jar /home/hadoop/app/hadoop-2.6.0-cdh5.7.0/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.6.0-cdh5.7.0.jar wordcount /data/wc/  /data/wc/output/
19/04/18 20:13:53 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8032
19/04/18 20:13:55 INFO input.FileInputFormat: Total input paths to process : 1
19/04/18 20:13:55 INFO mapreduce.JobSubmitter: number of splits:1
19/04/18 20:13:55 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1555589556022_0001
19/04/18 20:13:55 INFO impl.YarnClientImpl: Submitted application application_1555589556022_0001
19/04/18 20:13:55 INFO mapreduce.Job: The url to track the job: http://hadoop004:8088/proxy/application_1555589556022_0001/
19/04/18 20:13:55 INFO mapreduce.Job: Running job: job_1555589556022_0001
19/04/18 20:14:03 INFO mapreduce.Job: Job job_1555589556022_0001 running in uber mode : false
19/04/18 20:14:03 INFO mapreduce.Job:  map 0% reduce 0%
19/04/18 20:14:08 INFO mapreduce.Job:  map 100% reduce 0%
19/04/18 20:14:15 INFO mapreduce.Job:  map 100% reduce 100%
19/04/18 20:14:15 INFO mapreduce.Job: Job job_1555589556022_0001 completed successfully
19/04/18 20:14:15 INFO mapreduce.Job: Counters: 49
	File System Counters
		FILE: Number of bytes read=71
		FILE: Number of bytes written=223593
		FILE: Number of read operations=0
		FILE: Number of large read operations=0
		FILE: Number of write operations=0
		HDFS: Number of bytes read=145
		HDFS: Number of bytes written=66
		HDFS: Number of read operations=6
		HDFS: Number of large read operations=0
		HDFS: Number of write operations=2
	Job Counters
		Launched map tasks=1
		Launched reduce tasks=1
		Data-local map tasks=1
		Total time spent by all maps in occupied slots (ms)=2983
		Total time spent by all reduces in occupied slots (ms)=3594
		Total time spent by all map tasks (ms)=2983
		Total time spent by all reduce tasks (ms)=3594
		Total vcore-seconds taken by all map tasks=2983
		Total vcore-seconds taken by all reduce tasks=3594
		Total megabyte-seconds taken by all map tasks=3054592
		Total megabyte-seconds taken by all reduce tasks=3680256
	Map-Reduce Framework
		Map input records=3
		Map output records=9
		Map output bytes=80
		Map output materialized bytes=67
		Input split bytes=101
		Combine input records=9
		Combine output records=5
		Reduce input groups=5
		Reduce shuffle bytes=67
		Reduce input records=5
		Reduce output records=5
		Spilled Records=10
		Shuffled Maps =1
		Failed Shuffles=0
		Merged Map outputs=1
		GC time elapsed (ms)=54
		CPU time spent (ms)=1460
		Physical memory (bytes) snapshot=461029376
		Virtual memory (bytes) snapshot=3206344704
		Total committed heap usage (bytes)=319291392
	Shuffle Errors
		BAD_ID=0
		CONNECTION=0
		IO_ERROR=0
		WRONG_LENGTH=0
		WRONG_MAP=0
		WRONG_REDUCE=0
    File Input Format Counters
		Bytes Read=44
	File Output Format Counters
		Bytes Written=66

hadoop@hadoop004 sbin]$ hdfs dfs -ls /data/wc/output/par*
-rw-r--r--   1 hadoop supergroup         66 2019-04-18 20:14 /data/wc/output/part-r-00000.bz2
[hadoop@hadoop004 hadoop]$ hdfs dfs -text /data/wc/output/par*
19/04/18 20:20:15 INFO bzip2.Bzip2Factory: Successfully loaded & initialized native-bzip2 library system-native
19/04/18 20:20:15 INFO compress.CodecPool: Got brand-new decompressor [.bz2]
boy	2
hello	4
hi	1
son	1
word	1

再来试试snappy输出

[hadoop@hadoop004 hadoop-2.6.0-cdh5.7.0]$ hdfs dfs -rm -r /data/wc/output
Deleted /data/wc/output

[hadoop@hadoop004 hadoop-2.6.0-cdh5.7.0]$ hdfs dfs -ls /data/wc/output/par*
-rw-r--r--   1 hadoop supergroup         42 2019-04-18 20:25 /data/wc/output/part-r-00000.snappy
[hadoop@hadoop004 hadoop-2.6.0-cdh5.7.0]$ hdfs dfs -text /data/wc/output/par*
19/04/18 20:27:29 INFO compress.CodecPool: Got brand-new decompressor [.snappy]
boy	2
hello	4
hi	1
son	1
word	1

上面是以WordCount为例子，下面再以一个小文件为例子通过hive来实现压缩

[hadoop@hadoop004 data]$ hdfs dfs -put page_views.dat /data/click/

[hadoop@hadoop004 data]$ hdfs dfs -ls /data/click/
Found 1 items
-rw-r--r--   1 hadoop supergroup   19014993 2019-04-19 12:26 /data/click/page_views.dat

[hadoop@hadoop004 data]$ hdfs dfs -du -s -h /data/click/page_views.dat
18.1 M  18.1 M  /data/click/page_views.dat
hive> SET hive.exec.compress.output;
hive.exec.compress.output=false


hive>
    > create table page_views(
    > track_times string,
    > url string,
    > session_id string,
    > referer string,
    > ip string,
    > end_user_id string,
    > city_id string
    > ) row format delimited fields terminated by '\t';
OK
Time taken: 0.741 seconds

hive> load data local inpath '/home/hadoop/data/page_views.dat' overwrite into table page_views;
Loading data to table default.page_views
Table default.page_views stats: [numFiles=1, numRows=0, totalSize=19014993, rawDataSize=0]
OK
Time taken: 0.595 seconds


[hadoop@hadoop004 data]$ hdfs dfs -du -s -h /user/hive/warehouse/page_views/page_views.dat
18.1 M  18.1 M  /user/hive/warehouse/page_views/page_views.dat

因为我在mapred-site.xml里面设置默认的解压缩格式为snappy

hive>
    > set hive.exec.compress.output=true;
hive> set hive.exec.compress.output
    > ;
hive.exec.compress.output=true


hive> set mapreduce.output.fileoutputformat.compress.codec
    > ;
mapreduce.output.fileoutputformat.compress.codec=org.apache.hadoop.io.compress.SnappyCodec

hive> create table page_views_snappy as select * from page_views;
Query ID = hadoop_20190419141818_561ede29-e964-4655-9e48-1e4f5d6eeb5c
Total jobs = 3
Launching Job 1 out of 3
Number of reduce tasks is set to 0 since there's no reduce operator
Starting Job = job_1555643336639_0002, Tracking URL = http://hadoop004:8088/proxy/application_1555643336639_0002/
Kill Command = /home/hadoop/app/hadoop-2.6.0-cdh5.7.0/bin/hadoop job  -kill job_1555643336639_0002
Hadoop job information for Stage-1: number of mappers: 1; number of reducers: 0
2019-04-19 14:30:50,994 Stage-1 map = 0%,  reduce = 0%
2019-04-19 14:30:57,450 Stage-1 map = 100%,  reduce = 0%, Cumulative CPU 2.7 sec
MapReduce Total cumulative CPU time: 2 seconds 700 msec
Ended Job = job_1555643336639_0002
Stage-4 is selected by condition resolver.
Stage-3 is filtered out by condition resolver.
Stage-5 is filtered out by condition resolver.
Moving data to: hdfs://hadoop004:9000/user/hive/warehouse/.hive-staging_hive_2019-04-19_14-30-44_413_8511625915872870114-1/-ext-10001
Moving data to: hdfs://hadoop004:9000/user/hive/warehouse/page_views_snappy
Table default.page_views_snappy stats: [numFiles=1, numRows=100000, totalSize=8814444, rawDataSize=18914993]
MapReduce Jobs Launched:
Stage-Stage-1: Map: 1   Cumulative CPU: 2.7 sec   HDFS Read: 19018292 HDFS Write: 8814535 SUCCESS
Total MapReduce CPU Time Spent: 2 seconds 700 msec
OK
Time taken: 15.354 seconds

 

[hadoop@hadoop004 data]$ hdfs dfs -ls /user/hive/warehouse/page_views_snappy
Found 1 items
-rwxr-xr-x   1 hadoop supergroup    8814444 2019-04-19 14:30 /user/hive/warehouse/page_views_snappy/000000_0.snappy


[hadoop@hadoop004 data]$ hdfs dfs -du -s -h /user/hive/warehouse/page_views_snappy
8.4 M  8.4 M  /user/hive/warehouse/page_views_snappy

下面采用bzip2瞅瞅

hive>
    > set mapreduce.output.fileoutputformat.compress.codec=org.apache.hadoop.io.compress.BZip2Codec;

hive> set mapreduce.output.fileoutputformat.compress.codec;
mapreduce.output.fileoutputformat.compress.codec=org.apache.hadoop.io.compress.BZip2Codec


hive> create table page_views_bzip2 as select * from page_views;
Query ID = hadoop_20190419143939_53d67293-92de-4697-a791-f9a1afe7be01
Total jobs = 3
Launching Job 1 out of 3
Number of reduce tasks is set to 0 since there's no reduce operator
Starting Job = job_1555643336639_0003, Tracking URL = http://hadoop004:8088/proxy/application_1555643336639_0003/
Kill Command = /home/hadoop/app/hadoop-2.6.0-cdh5.7.0/bin/hadoop job  -kill job_1555643336639_0003
Hadoop job information for Stage-1: number of mappers: 1; number of reducers: 0
2019-04-19 14:44:16,992 Stage-1 map = 0%,  reduce = 0%
2019-04-19 14:44:25,471 Stage-1 map = 100%,  reduce = 0%, Cumulative CPU 5.19 sec
MapReduce Total cumulative CPU time: 5 seconds 190 msec
Ended Job = job_1555643336639_0003
Stage-4 is selected by condition resolver.
Stage-3 is filtered out by condition resolver.
Stage-5 is filtered out by condition resolver.
Moving data to: hdfs://hadoop004:9000/user/hive/warehouse/.hive-staging_hive_2019-04-19_14-44-09_564_3458580979229190548-1/-ext-10001
Moving data to: hdfs://hadoop004:9000/user/hive/warehouse/page_views_bzip2
Table default.page_views_bzip2 stats: [numFiles=1, numRows=100000, totalSize=3815195, rawDataSize=18914993]
MapReduce Jobs Launched:
Stage-Stage-1: Map: 1   Cumulative CPU: 5.19 sec   HDFS Read: 19018291 HDFS Write: 3815285 SUCCESS
Total MapReduce CPU Time Spent: 5 seconds 190 msec
OK
Time taken: 17.289 seconds


[hadoop@hadoop004 hadoop]$ hdfs dfs -ls /user/hive/warehouse/page_views_bzip2
Found 1 items
-rwxr-xr-x   1 hadoop supergroup    3815195 2019-04-19 14:44 /user/hive/warehouse/page_views_bzip2/000000_0.bz2

[hadoop@hadoop004 hadoop]$ hdfs dfs -du -s -h /user/hive/warehouse/page_views_bzip2/*
3.6 M  3.6 M  /user/hive/warehouse/page_views_bzip2/000000_0.bz2

再来看看gzip的

hive>
    > set mapreduce.output.fileoutputformat.compress.codec=org.apache.hadoop.io.compress.GzipCodec;

hive> set mapreduce.output.fileoutputformat.compress.codec;
mapreduce.output.fileoutputformat.compress.codec=org.apache.hadoop.io.compress.GzipCodec

hive>
    > create table page_views_gzip as select * from page_views;

Query ID = hadoop_20190419143939_53d67293-92de-4697-a791-f9a1afe7be01
Total jobs = 3
Launching Job 1 out of 3
Number of reduce tasks is set to 0 since there's no reduce operator
Starting Job = job_1555643336639_0004, Tracking URL = http://hadoop004:8088/proxy/application_1555643336639_0004/
Kill Command = /home/hadoop/app/hadoop-2.6.0-cdh5.7.0/bin/hadoop job  -kill job_1555643336639_0004
Hadoop job information for Stage-1: number of mappers: 1; number of reducers: 0
2019-04-19 14:48:10,606 Stage-1 map = 0%,  reduce = 0%
2019-04-19 14:48:18,019 Stage-1 map = 100%,  reduce = 0%, Cumulative CPU 3.29 sec
MapReduce Total cumulative CPU time: 3 seconds 290 msec
Ended Job = job_1555643336639_0004
Stage-4 is selected by condition resolver.
Stage-3 is filtered out by condition resolver.
Stage-5 is filtered out by condition resolver.
Moving data to: hdfs://hadoop004:9000/user/hive/warehouse/.hive-staging_hive_2019-04-19_14-48-03_436_7531556390065383047-1/-ext-10001
Moving data to: hdfs://hadoop004:9000/user/hive/warehouse/page_views_gzip
Table default.page_views_gzip stats: [numFiles=1, numRows=100000, totalSize=5550655, rawDataSize=18914993]
MapReduce Jobs Launched:
Stage-Stage-1: Map: 1   Cumulative CPU: 3.29 sec   HDFS Read: 19018290 HDFS Write: 5550744 SUCCESS
Total MapReduce CPU Time Spent: 3 seconds 290 msec
OK
Time taken: 15.866 seconds
[hadoop@hadoop004 hadoop]$ hdfs dfs -ls /user/hive/warehouse/page_views_gzip
Found 1 items
-rwxr-xr-x   1 hadoop supergroup    5550655 2019-04-19 14:48 /user/hive/warehouse/page_views_gzip/000000_0.gz

[hadoop@hadoop004 hadoop]$ hdfs dfs -du -s -h /user/hive/warehouse/page_views_gzip/*
5.3 M  5.3 M  /user/hive/warehouse/page_views_gzip/000000_0.gz

 

下面再来个猛的，下面还做了一部分清洗操作

[hadoop@hadoop004 data]$ hdfs dfs -put /home/hadoop/data/login.log /g6/hadoop/accesslog/20180717/

[hadoop@hadoop004 data]$ hdfs dfs -ls /g6/hadoop/accesslog/20180717/
Found 1 items
-rw-r--r--   1 hadoop supergroup  476118692 2019-04-19 12:09 /g6/hadoop/accesslog/20180717/login.log

[hadoop@hadoop004 data]$ hdfs dfs -du -s -h /g6/hadoop/accesslog/20180717/login.log
454.1 M  454.1 M  /g6/hadoop/accesslog/20180717/login.log
[hadoop@hadoop004 data]$ hadoop jar /home/hadoop/lib/g6-hadoop-1.0.jar com.ruozedata.hadoop.mapreduce.driver.LogETLDriver /g6/hadoop/accesslog/20180717 /g6/hadoop/access/output
19/04/19 12:12:57 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8032
19/04/19 12:12:57 WARN mapreduce.JobResourceUploader: Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this.
19/04/19 12:12:58 INFO input.FileInputFormat: Total input paths to process : 1
19/04/19 12:12:58 INFO mapreduce.JobSubmitter: number of splits:4
19/04/19 12:12:58 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1555643336639_0001
19/04/19 12:12:58 INFO impl.YarnClientImpl: Submitted application application_1555643336639_0001
19/04/19 12:12:58 INFO mapreduce.Job: The url to track the job: http://hadoop004:8088/proxy/application_1555643336639_0001/
19/04/19 12:12:58 INFO mapreduce.Job: Running job: job_1555643336639_0001
19/04/19 12:13:06 INFO mapreduce.Job: Job job_1555643336639_0001 running in uber mode : false
19/04/19 12:13:06 INFO mapreduce.Job:  map 0% reduce 0%
19/04/19 12:13:19 INFO mapreduce.Job:  map 1% reduce 0%
19/04/19 12:13:22 INFO mapreduce.Job:  map 3% reduce 0%
19/04/19 12:13:23 INFO mapreduce.Job:  map 5% reduce 0%
19/04/19 12:13:24 INFO mapreduce.Job:  map 6% reduce 0%
19/04/19 12:13:25 INFO mapreduce.Job:  map 11% reduce 0%
19/04/19 12:13:26 INFO mapreduce.Job:  map 13% reduce 0%
19/04/19 12:13:27 INFO mapreduce.Job:  map 19% reduce 0%
19/04/19 12:13:28 INFO mapreduce.Job:  map 23% reduce 0%
19/04/19 12:13:30 INFO mapreduce.Job:  map 34% reduce 0%
19/04/19 12:13:32 INFO mapreduce.Job:  map 41% reduce 0%
19/04/19 12:13:34 INFO mapreduce.Job:  map 52% reduce 0%
19/04/19 12:13:34 INFO mapreduce.Job: Task Id : attempt_1555643336639_0001_m_000001_0, Status : FAILED
Container killed on request. Exit code is 137
Container exited with a non-zero exit code 137
Killed by external signal

19/04/19 12:13:35 INFO mapreduce.Job:  map 62% reduce 0%
19/04/19 12:13:36 INFO mapreduce.Job:  map 75% reduce 0%
19/04/19 12:13:50 INFO mapreduce.Job:  map 90% reduce 25%
19/04/19 12:13:51 INFO mapreduce.Job:  map 100% reduce 25%
19/04/19 12:13:53 INFO mapreduce.Job:  map 100% reduce 78%
19/04/19 12:13:54 INFO mapreduce.Job:  map 100% reduce 100%
19/04/19 12:13:54 INFO mapreduce.Job: Job job_1555643336639_0001 completed successfully
19/04/19 12:13:54 INFO mapreduce.Job: Counters: 52
	File System Counters
		FILE: Number of bytes read=39419386
		FILE: Number of bytes written=79392688
		FILE: Number of read operations=0
		FILE: Number of large read operations=0
		FILE: Number of write operations=0
		HDFS: Number of bytes read=476131480
		HDFS: Number of bytes written=36841185
		HDFS: Number of read operations=15
		HDFS: Number of large read operations=0
		HDFS: Number of write operations=2
	Job Counters
		Failed map tasks=1
		Killed map tasks=1
		Launched map tasks=6
		Launched reduce tasks=1
		Other local map tasks=2
		Data-local map tasks=4
		Total time spent by all maps in occupied slots (ms)=120819
		Total time spent by all reduces in occupied slots (ms)=14541
		Total time spent by all map tasks (ms)=120819
		Total time spent by all reduce tasks (ms)=14541
		Total vcore-seconds taken by all map tasks=120819
		Total vcore-seconds taken by all reduce tasks=14541
		Total megabyte-seconds taken by all map tasks=123718656
		Total megabyte-seconds taken by all reduce tasks=14889984
	Map-Reduce Framework
		Map input records=1100000
		Map output records=1100000
		Map output bytes=100291376
		Map output materialized bytes=39416687
		Input split bytes=500
		Combine input records=0
		Combine output records=0
		Reduce input groups=1
		Reduce shuffle bytes=39416687
		Reduce input records=1100000
		Reduce output records=1100000
		Spilled Records=2200000
		Shuffled Maps =4
		Failed Shuffles=0
		Merged Map outputs=4
		GC time elapsed (ms)=3969
		CPU time spent (ms)=35800
		Physical memory (bytes) snapshot=2299715584
		Virtual memory (bytes) snapshot=7979675648
		Total committed heap usage (bytes)=2030043136
	Shuffle Errors
		BAD_ID=0
		CONNECTION=0
		IO_ERROR=0
		WRONG_LENGTH=0
		WRONG_MAP=0
		WRONG_REDUCE=0
	File Input Format Counters
		Bytes Read=476130980
	File Output Format Counters
		Bytes Written=36841185


[hadoop@hadoop004 data]$ hdfs dfs -ls /g6/hadoop/access/output
Found 2 items
-rw-r--r--   1 hadoop supergroup          0 2019-04-19 12:13 /g6/hadoop/access/output/_SUCCESS
-rw-r--r--   1 hadoop supergroup   36841185 2019-04-19 12:13 /g6/hadoop/access/output/part-r-00000.snappy

[hadoop@hadoop004 data]$ hdfs dfs -du -s -h /g6/hadoop/access/output
35.1 M  35.1 M  /g6/hadoop/access/output

 

hive数据文件格式和压缩格式
Wflowerd的博客
05-19 841
.文件格式文件格式按面向的存储形式不同，分为面向行和面向列两大类文件格式。面向行、列类型 类型名称是否可切割计算优点缺点适用场景面向行文本文件（.txt)YES查看编辑简单无压缩占空间大，传输压力大，数据解析开销大学习练习使用面向行SequenceFile序列文件格式(.seq)YES自支持、二进制KV存储，支持行和块的压缩本地查看不方便，小文件合并成KV格式后不易查看内部数据生产环境使用map输...
Hive常见压缩格式和性能对比
11号车厢
08-12 2982
压缩能够减少存储磁盘空间，降低网络IO和磁盘IO，加快数据在磁盘和网络中的传输速度，从而提高系统的处理速度。 虽然由以上的好处，但是由于使用数据时，需要先将数据解压，加重了CPU负荷。根据我们的实际工作场景，我们要选择合适的压缩格式。 常见的压缩格式如下： 下面是一份源数据1.4G的文件，各类压缩格式的比率和时间对比 数据压缩后大小： 压缩和解压时间： 从上面对比可以看出： 在压缩数据比...
hive文件存储格式
weixin_34302798的博客
01-03 199
hive文件存储格式包括以下几类： TEXTFILE SEQUENCEFILE RCFILE 自定义格式 其中TEXTFILE为默认格式，建表时不指定默认为这个格式，导入数据时会直接把数据文件拷贝到hdfs上不进行处理。 SequenceFile,RCFile格式的表不能直接从本地文件导入数据，数据要先导入到textfile格式的表中，然后再从textfile表中用insert导入到S...
大数据领域 Hive 数据压缩与存储优化
最新发布
大数据洞察的博客
08-05 698
在数据量呈指数级增长的今天，Hive作为大数据生态系统中的数据仓库基石，其存储效率和查询性能直接影响企业的运营成本和决策速度。本文将深入探讨Hive数据压缩与存储优化的核心技术，从基础原理到高级实践，为您揭示如何在不牺牲查询性能的前提下，显著降低存储成本并提升处理效率。我们将系统比较各类压缩算法的优劣，详解ORC与Parquet等列式存储格式的内部机制，并通过真实案例展示如何构建高效的Hive数据存储架构。
Hive压缩格式
weixin_30824599的博客
08-18 221
TextFile Hive数据表的默认格式，存储方式：行存储。 可使用Gzip,Bzip2等压缩算法压缩,压缩后的文件不支持split 但在反序列化过程中，必须逐个字符判断是不是分隔符和行结束符，因此反序列化开销会比SequenceFile高几十倍。 --创建数据表：create table if not exists textfile_table( site string,...
Hive 中的压缩格式对比
qq_654603797
04-10 1162
TextFile 默认格式，存储方式为行存储，数据不做压缩，磁盘开销大，数据解析开销大。可结合Gzip、Bzip2使用(系统自动检查，执行查询时自动解压)，但使用这种方式，压缩后的文件不支持split，Hive不会对数据进行切分，从而无法对数据进行并行操作。并且在反序列化过程中，必须逐个字符判断是不是分隔符和行结束符，因此反序列化开销会比SequenceFile高几十倍。 SequenceFile SequenceFile是Hadoop API提供的一种二进制文件支持，存储方式为行存储，其具有使用方便
大数据Hive--文件格式和压缩
xd__xy的博客
02-28 1300
压缩格式算法文件扩展名是否可切分DEFLATEDEFLATE.deflate否GzipDEFLATE.gz否bzip2bzip2.bz2是LZOLZO.lzo是SnappySnappy.snappy否为了支持多种压缩/解压缩算法，Hadoop引入了编码/解码器，如下表所示：Hadoop查看支持压缩的方式hadoop checknative。Hadoop在driver端设置压缩。压缩格式对应的编码/解码器DEFLATEgzipbzip2LZO。
HIVE的压缩和存储格式
u013983161的博客
05-10 1996
在生产环境中，常用的HIVE存储格式：列式存储的orc和parquet HIVE压缩格式：冷数据-----gzip压缩（压缩比高，压缩解压缩速度高，不可切割）； 非冷数据------lzo（可切割）和snappy（不可切割） 案例：student表存储方式是ORC，压缩方式SNAPPY create table student (empno int,empname string) stored as orc tblproperties("orc...
Hive之文件格式
ly13607255628的博客
10-10 1256
Hive支持的存储数的格式主要有：TEXTFILE（默认格式） 、SEQUENCEFILE、RCFILE、ORCFILE、PARQUET。 textfile为默认格式，建表时没有指定文件格式，则使用TEXTFILE，导入数据时会直接把数据文件拷贝到hdfs上不进行处理； sequencefile，rcfile，orcfile格式的表不能直接从本地文件导入数据，数据要先导入到textfile格式的表中， 然后再从表中用insert导入sequencefile、rcfile、orcfile表中。 行存储与列
Hive数据仓库文件存储压缩格式探究及经验总结
石榴姐yyds
07-12 1846
数仓存储压缩格式的选择对于数仓存储及性能的优化具有重要参考意义，存储优化是一个重要的指标，他可以帮助节省磁盘存储空间，节约了成本。本文对常见的几种压缩存储方案进行了研究，并对其性能各方面进行了测试，给出了数仓中常用的存储压缩解决方案，并对实践中遇到的问题进行了总结，具有一定的借鉴意义。
Hive存储格式跟压缩对比(各种技术都在这里)
qq_32736999的博客
06-01 2469
存储: 1 TextFile 2 SequenceFile 3 RCFile 4 OrcFile 5 Parquet 6 Avro 6种性能测试 textfile 默认格式； 存储方式为行存储； 磁盘开销大 数据解析开销大； 但使用这种方式，hive不会对数据进行切分，从而无法对数据进行并行操作。 SequenceFile 二进制文件,以<key,va...
HIVE数据的压缩与存储格式
meng_xin_true的博客
07-29 2251
压缩技术能够有效减少底层存储系统（HDFS）读写字节数。压缩提高了网络带宽和磁盘空间的效率。在运行MR程序时，I/O操作、网络数据传输、Shuffle和Merge要花大量的时间，尤其是数据规模很大和工作负载密集的情况下，因此，使用数据压缩显得非常重要。鉴于磁盘I/O和网络带宽是Hadoop的宝贵资源，数据压缩对于节省资源、最小化磁盘I/O和网络传输非常有帮助。可以在任意MapReduce阶段启用压缩。不过，尽管压缩与解压操作的CPU开销不高，其性能的提升和资源的节省并非没有代价。...
Hive文件格式与压缩
04-24 1072
概念与实操相结合介绍Hive文件格式及压缩的实现方法
浅谈hive中数据的几种压缩方式
zcb_data的博客
05-11 3027
hive库中有个表，表名叫做user_info_base表创表的命令是：create table user_info_base(    id string,    name string,    age string)row format delimited fields terminated by '\t' LINES TERMINATED BY '\n'  stored as textfile...
HIve 常见数据压缩方式对比
youhuakongzhi的博客
03-06 1336
转自：https://blog.youkuaiyun.com/weixin_36714575/article/details/8009157 五.结论 1.在压缩存储时间上，除Sequencefile外基本都相差无几。 2.数据压缩比例上ORC最优，相比textfile节省了50倍磁盘空间，parquet压缩性能也较好。 3.SQL查询速度而言，ORC与parquet性能较好，远超其余存储格式。 综合上述各种性能指标，建议工作中原始日志写入hive的存储格式都采用ORC或者parquet格式，这和目前主...
Hive ---- 文件格式和压缩
是微风，是晚霞，是无可替代
05-30 3991
为了支持多种压缩/解压缩算法，Hadoop引入了编码/解码器，如下表所示：Hadoop查看支持压缩的方式hadoop checknative。Hadoop在driver端设置压缩。
【Hive】压缩格式
Just Do IT
06-29 4809
常见压缩格式 压缩方式 压缩比 压缩速度 解压缩速度 是否可分割 gzip 13.4% 21 MB/s 118 MB/s 否 bzip2 13.2% 2.4MB/s 9.5MB/s 是 lzo 20.5% 135 MB/s 410 MB/s 是 snappy 22.2% 172 MB/s 409 MB/s 否 压缩格式对应的编/解码器 压缩格式 对应的编/解码器 DEFLATE org.apache.hadoop.io.compress.DefaultCode
关于Hive中的存储格式及压缩格式详解
wenqi
06-07 6372
最近面试，遇到了关于Hive的数据存储格式的问题，回答不尽人意，抽时间总结多看看关于Hive存储格式和压缩格式的内容。TEXTFILE 是 Hive 默认文件存储方式，存储方式为行存储，数据不做压缩，磁盘开销大，数据解析开销大，数据不支持分片，数据加载导入方式可以通过LOAD和INSERT两种方式加载数据。可结合Gzip、Bzip2使用(系统自动检查，执行查询时自动解压) ，但使用gzip方式，hive不会对数据进行切分，从而无法对数据进行并行操作，但压缩后的文件不支持split。在反序列化过程中，必须逐个
Hive常见的压缩格式
延宝小白马的博客
08-02 489
完成对Hive表的压缩，有两种方式：配置MapReduce压缩、开启Hive表压缩功能。当然为了方便起见，Hive中的特定表支持压缩属性，自动完成压缩的功能。其中压缩比bzip2 > zlib > gzip > deflate > snappy > lzo > lz4，在不同的测试场景中，会有差异，这仅仅是一个大概的排名情况。从压缩性能上来看：lz4 > lzo > snappy > deflate > gzip > bzip2，其中lz4、lzo、snappy压缩和解压缩速度快，压缩比低。
spark创建hive表指定压缩格式
06-04
因此，在Spark中创建Hive表并指定压缩格式，我们通常需要做两件事：1.设置表的存储格式（如ORC、Parquet等）2.设置压缩相关的属性（如压缩编解码器）注意：压缩格式的指定通常是通过在SparkSQL中设置Hadoop的配置...
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值