Hive基础理论及表的类型

Spark DataFrame vector 类型存储到Hive表 Spark DataFrame vector 类型存储到Hive表 Spark DataFrame vector 类型存储到Hive表 软件版本 场景描述 问题的迂回解决方法 示例 1. 软件版本 软件 版本 Spark 1.6.0 Hive 1.2.1 2. 场景描述 在使用Spark时,有时需要存储DataFram... 阅读详情

什么是Hive?

Hive是建立在Hadoop上的,用来构建数据仓库的工具,里面有表的概念,可以使用SQL语句实现存储、查询和分析存储在 HDFS上的数据,这些SQL语句在Hive中称为HQL,语法和SQL语句基本一样。
由于数据是杂乱无章的,所以Hive需要一份关于这些数据的元数据来管理和操作这些数据。这份元数据包括:

元数据(
行的分隔符(在映射成表的时候知道在哪里分行显示)
字段分隔符(在映射成表的时候知道在哪里分列显示)
字段的类型
字段的名称
)

在Hive中,我们把数据存储在HDFS中,元数据默认存储在Hive自带的Derby数据库中,由于Derby不能实现并发访问,所以我们一般使用mysql进行替换。

Hive的原理

Hive 将用户的HQL 语句进行解析,优化,最终把一个个的HQL语句转换为MapReduce 作业提交到Hadoop 集群上,Hadoop进行作业的调度及监控,作业完成后将执行结果返回给用户。所以,Hive并不进行计算,只是把HQL解析为MapperReduce在HDFS集群中运行而已,所以Hive的效率并不高。

Hive表的类型

hive中有5种表:

1. 内部表

数据默认存储在/user/hive/warehouse,由hive自身管理,删除内部表会同时删除存储数据和元数据。

建表方式一:
create table t1(
	    id	INT,
	    name STRING,
	    age INT,
		gfs ARRAY<STRING>,
		address MAP<STRING,STRING>,
		info STRUCT<country:String,province:String,shi:String>
)
ROW FORMAT DELIMITED 
FIELDS TERMINATED BY ' ' 
COLLECTION ITEMS TERMINATED BY ','
MAP KEYS TERMINATED BY ':' 
LINES TERMINATED BY '\n'
LOCATION "/test";//可以设置源数据的位置,若不设置默认就在Hive的工作目录区
建表方式二:
  create table gfstbl1 like gfstbl;只是创建表结构
建表方式三:
 create table gfstbl2 AS SELECT id,name,gfs,address from gfstbl; 
  会创建相应的表结构,并且插入数据,相当于完整的赋值
加载数据:
load data local inpath '/root/gfs.txt' into table gfstbl;
查看表描述信息:
DESCRIBE [EXTENDED|FORMATTED] table_name
EXTENDED极简的方式显示
FORMATTED格式化方式来显示
DESCRIBE EXTENDED gfstbl;默认就是EXTENDED
DESCRIBE FORMATTED gfstbl;
插入数据的其他方式:
插入数据的方式:
	1、insert 新数据
	2、load
	3、查询其他表数据 insert 到新表中
	模板:
	
hive修改表操作与array类型 导入数据:load data local inpath '/home/wtk/data_for_array_type.txt' into table test_array;如: ALTER TABLE table_name SET TBLPROPERTIES ('comment' = new_comment);删除分区:ALTER TABLE tablename DROP PARTITION (month='201105');-- 查询 location 数组中元素的个数。 阅读详情

相关推荐

Hive之表类型解析

Hive的表在HDFS里面体现都是文件 1、内部表 内部表示Hive中默认的表类型,表数据默认存储在warehouse目录下,即使我们使用load data的方式, Hive也会在数据拷贝在warehouse目录下,当我们创建内部表时,Hive会在warehouse下创建表名的目录存储数据文件, 同时会在metastore维护元数据信息,当我们删除表的时候,表的数据会删除,同时metastore中的元数据信息也会同时删除。 2、外部表 外部表在加载数据的时候,实际数据并不会移...

bigdodo的博客 852

推荐系统公平性之流行度偏差(fairness in recommender systems -- popularity bias)

主要参考论文:《The Unfairness of Popularity Bias in Recommendation》 RMSE@RecSys 2019 流行度偏差是什么 先定义流行物品和非流行物品。下图是(Movielens 1M)数据集中物品的评分情况。 图的横坐标表示不同物品,纵坐标表示物品的评分次数。可以看出只有少部分物品得到了很多的评分,大部分曲线尾部的物品都只有少量的评分,我们也把这部分物品称为长尾物品。在本文中,我们取曲线的前20%对应的物品为流行物品,剩余的为非流行物品(长尾物品)。 然

beilizhang的博客 5314

手把手教你理解瑞萨RH850芯片的HSM:从硬件隔离到软件中断通信

本文深入解析瑞萨RH850芯片的HSM架构,从硬件隔离到软件中断通信的完整实现。通过详细讲解HSM的硬件组件、启动流程、核间通信机制及安全服务实现,帮助开发者掌握芯片级安全技术,提升汽车电子和工业控制领域的安全防护能力。

weixin_33925615的博客 382

PCRNet: Point Cloud Registration Network using PointNet Encoding

PCRNet算是在ModelNet40这种object-centric数据集上用deep-learning做配准最好入手的一篇文章了,整体架构涉及简洁明了,一句话描述就是简单粗暴,话不多说,直接上网络架构图: Work Flow: 首先对输入source (N×3N \times 3N×3)与template (M×3M \times 3M×3)进行PointNet操作,即MLP + max-pooling后,分别得到描述各自点云的维度为1024的global feature vector:Φ(PS).

dloading7的博客 1648

VGG16

VGG16学习笔记

一去不复返的通信er 1178

COCO数据集缺失文件补全方法

本文代码是以目标检测(object detection)和实例分割(instance segmentation)任务的标签文件为例,即文件。其他任务的标签文件内容略有不同,但是图片来源表示字段完全相同,因此代码可通用。另外如果是非2017版本COCO应该也通用。msgspec是一个高性能的序列化和验证库,支持 JSON、MessagePack、YAML 和 TOML 等多种格式。高性能:在常见协议的编码/解码中表现出色,通常比其他库快10-80倍。零成本的模式验证。

qq_46396470的博客 1996

eye-in-hand手眼标定系统

上一个博文说了一种手眼标定系统的特殊情况,就是相机随着机械手一起移动的情况。虽然前面简单介绍了一种办法,就是内次拍照时让相机回到之前标定的位置,但是这样处理机器人的工作效率就会大大降低,产生很多重复动作,所以这篇博文就来讲一种更简单的标定方案。 下面是halcon官方给的手眼标定系统的案例代码。 这个例子主要应用于相机固定在机械手末端的手眼系统中,而且标定板相对于机器人也是静止的。 在这...

二毛的博客 1万+

Authentication Assertion Tickets,SAP 系统到系统单点登录里的短命凭证

SAP 系统间的身份传递机制存在 Authentication Assertion Ticket 和 SAP Logon Ticket 两种易混淆的凭证。前者用于系统间短时身份委托(有效期2分钟,通过HTTP头传输),后者用于浏览器SSO(可配置有效期,通过Cookie传输)。Assertion Ticket依赖签发系统的数字签名和严格的时间同步,适用于ABAP/Java系统间的后台作业调用。现代SAP架构已转向OAuth 2.0等标准协议,但老系统中仍需注意两种票据的关键差异:传输方式、有效期、接收方限制

2007 年 ~ 2025 年,深耕 SAP 技术 18 年 604

【STK】手把手教你利用STK进行光电红外仿真04-STK/EOIR模块的实战操作1航天器星载相机对LEO卫星成像

今天引入一个实际的场景,帮助大家完成对于这个模块的理解与运用。

weixin_44184852的博客 2419

深度学习工具包 DeepLearningToolbox

深度学习工具包Deprecation notice.-----This toolbox is outdated and no longer maintained.There are much better tools available for deep learning than this toolbox, e.g. [Theano](http://deeplearning.net/software/theano/), [torch](http://torch.ch/) or [tensorflow](http://www.tensorflow.org/)I would suggest you use one of the tools mentioned above rather than use this toolbox.Best, Rasmus.DeepLearnToolbox================A Matlab toolbox for Deep Learning.Deep Learning is a new subfield of machine learning that focuses on learning deep hierarchical models of data.It is inspired by the human brain's apparent deep (layered, hierarchical) architecture.A good overview of the theory of Deep Learning theory is[Learning Deep Architectures for AI](http://www.iro.umontreal.ca/~bengioy/papers/ftml_book.pdf)For a more informal introduction, see the following videos by Geoffrey Hinton and Andrew Ng.* [The Next Generation of Neural Networks](http://www.youtube.com/watch?v=AyzOUbkUf3M) (Hinton, 2007)* [Recent Developments in Deep Learning](http://www.youtube.com/watch?v=VdIURAu1-aU) (Hinton, 2010)* [Unsupervised Feature Learning and Deep Learning](http://www.youtube.com/watch?v=ZmNOAtZIgIk) (Ng, 2011)If you use this toolbox in your research please cite [Prediction as a candidate for learning deep hierarchical models of data](http://www2.imm.dtu.dk/pubdb/views/publication_details.php?id=6284)```@MASTERSTHESIS\{IMM2012-06284, author = "R. B. Palm", title = "Prediction as a candidate for learning deep hierarchical models of data", year = "2012",}```Contact: rasmusbergpalm at gmail dot comDirectories included in the toolbox-----------------------------------`NN/` - A library for Feedforward Backpropagation Neural Networks`CNN/` - A library for Convolutional Neural Networks`DBN/` - A library for Deep Belief Networks`SAE/` - A library for Stacked Auto-Encoders`CAE/` - A library for Convolutional Auto-Encoders`util/` - Utility functions used by the libraries`data/` - Data used by the examples`tests/` - unit tests to verify toolbox is workingFor references on each library check REFS.mdSetup-----1. Download.2. addpath(genpath('DeepLearnToolbox'));Example: Deep Belief Network---------------------```matlabfunction test_example_DBNload mnist_uint8;train_x = double(train_x) / 255;test_x = double(test_x) / 255;train_y = double(train_y);test_y = double(test_y);%% ex1 train a 100 hidden unit RBM and visualize its weightsrand('state',0)dbn.sizes = [100];opts.numepochs = 1;opts.batchsize = 100;opts.momentum = 0;opts.alpha = 1;dbn = dbnsetup(dbn, train_x, opts);dbn = dbntrain(dbn, train_x, opts);figure; visualize(dbn.rbm{1}.W'); % Visualize the RBM weights%% ex2 train a 100-100 hidden unit DBN and use its weights to initialize a NNrand('state',0)%train dbndbn.sizes = [100 100];opts.numepochs = 1;opts.batchsize = 100;opts.momentum = 0;opts.alpha = 1;dbn = dbnsetup(dbn, train_x, opts);dbn = dbntrain(dbn, train_x, opts);%unfold dbn to nnnn = dbnunfoldtonn(dbn, 10);nn.activation_function = 'sigm';%train nnopts.numepochs = 1;opts.batchsize = 100;nn = nntrain(nn, train_x, train_y, opts);[er, bad] = nntest(nn, test_x, test_y);assert(er < 0.10, 'Too big error');```Example: Stacked Auto-Encoders---------------------```matlabfunction test_example_SAEload mnist_uint8;train_x = double(train_x)/255;test_x = double(test_x)/255;train_y = double(train_y);test_y = double(test_y);%% ex1 train a 100 hidden unit SDAE and use it to initialize a FFNN% Setup and train a stacked denoising autoencoder (SDAE)rand('state',0)sae = saesetup([784 100]);sae.ae{1}.activation_function = 'sigm';sae.ae{1}.learningRate = 1;sae.ae{1}.inputZeroMaskedFraction = 0.5;opts.numepochs = 1;opts.batchsize = 100;sae = saetrain(sae, train_x, opts);visualize(sae.ae{1}.W{1}(:,2:end)')% Use the SDAE to initialize a FFNNnn = nnsetup([784 100 10]);nn.activation_function = 'sigm';nn.learningRate = 1;nn.W{1} = sae.ae{1}.W{1};% Train the FFNNopts.numepochs = 1;opts.batchsize = 100;nn = nntrain(nn, train_x, train_y, opts);[er, bad] = nntest(nn, test_x, test_y);assert(er < 0.16, 'Too big error');```Example: Convolutional Neural Nets---------------------```matlabfunction test_example_CNNload mnist_uint8;train_x = double(reshape(train_x',28,28,60000))/255;test_x = double(reshape(test_x',28,28,10000))/255;train_y = double(train_y');test_y = double(test_y');%% ex1 Train a 6c-2s-12c-2s Convolutional neural network %will run 1 epoch in about 200 second and get around 11% error. %With 100 epochs you'll get around 1.2% errorrand('state',0)cnn.layers = { struct('type', 'i') %input layer struct('type', 'c', 'outputmaps', 6, 'kernelsize', 5) %convolution layer struct('type', 's', 'scale', 2) %sub sampling layer struct('type', 'c', 'outputmaps', 12, 'kernelsize', 5) %convolution layer struct('type', 's', 'scale', 2) %subsampling layer};cnn = cnnsetup(cnn, train_x, train_y);opts.alpha = 1;opts.batchsize = 50;opts.numepochs = 1;cnn = cnntrain(cnn, train_x, train_y, opts);[er, bad] = cnntest(cnn, test_x, test_y);%plot mean squared errorfigure; plot(cnn.rL);assert(er<0.12, 'Too big error');```Example: Neural Networks---------------------```matlabfunction test_example_NNload mnist_uint8;train_x = double(train_x) / 255;test_x = double(test_x) / 255;train_y = double(train_y);test_y = double(test_y);% normalize[train_x, mu, sigma] = zscore(train_x);test_x = normalize(test_x, mu, sigma);%% ex1 vanilla neural netrand('state',0)nn = nnsetup([784 100 10]);opts.numepochs = 1; % Number of full sweeps through dataopts.batchsize = 100; % Take a mean gradient step over this many samples[nn, L] = nntrain(nn, train_x, train_y, opts);[er, bad] = nntest(nn, test_x, test_y);assert(er < 0.08, 'Too big error');%% ex2 neural net with L2 weight decayrand('state',0)nn = nnsetup([784 100 10]);nn.weightPenaltyL2 = 1e-4; % L2 weight decayopts.numepochs = 1; % Number of full sweeps through dataopts.batchsize = 100; % Take a mean gradient step over this many samplesnn = nntrain(nn, train_x, train_y, opts);[er, bad] = nntest(nn, test_x, test_y);assert(er < 0.1, 'Too big error');%% ex3 neural net with dropoutrand('state',0)nn = nnsetup([784 100 10]);nn.dropoutFraction = 0.5; % Dropout fraction opts.numepochs = 1; % Number of full sweeps through dataopts.batchsize = 100; % Take a mean gradient step over this many samplesnn = nntrain(nn, train_x, train_y, opts);[er, bad] = nntest(nn, test_x, test_y);assert(er < 0.1, 'Too big error');%% ex4 neural net with sigmoid activation functionrand('state',0)nn = nnsetup([784 100 10]);nn.activation_function = 'sigm'; % Sigmoid activation functionnn.learningRate = 1; % Sigm require a lower learning rateopts.numepochs = 1; % Number of full sweeps through dataopts.batchsize = 100; % Take a mean gradient step over this many samplesnn = nntrain(nn, train_x, train_y, opts);[er, bad] = nntest(nn, test_x, test_y);assert(er < 0.1, 'Too big error');%% ex5 plotting functionalityrand('state',0)nn = nnsetup([784 20 10]);opts.numepochs = 5; % Number of full sweeps through datann.output = 'softmax'; % use softmax outputopts.batchsize = 1000; % Take a mean gradient step over this many samplesopts.plot = 1; % enable plottingnn = nntrain(nn, train_x, train_y, opts);[er, bad] = nntest(nn, test_x, test_y);assert(er < 0.1, 'Too big error');%% ex6 neural net with sigmoid activation and plotting of validation and training error% split training data into training and validation datavx = train_x(1:10000,:);tx = train_x(10001:end,:);vy = train_y(1:10000,:);ty = train_y(10001:end,:);rand('state',0)nn = nnsetup([784 20 10]); nn.output = 'softmax'; % use softmax outputopts.numepochs = 5; % Number of full sweeps through dataopts.batchsize = 1000; % Take a mean gradient step over this many samplesopts.plot = 1; % enable plottingnn = nntrain(nn, tx, ty, opts, vx, vy); % nntrain takes validation set as last two arguments (optionally)[er, bad] = nntest(nn, test_x, test_y);assert(er < 0.1, 'Too big error');```[![Bitdeli Badge](https://d2weczhvl823v0.cloudfront.net/rasmusbergpalm/deeplearntoolbox/trend.png)](https://bitdeli.com/free "Bitdeli Badge")

精雕软件(JDpaint) 5.5 绿色版

精雕软件(JDpaint) 5.5 绿色版精雕软件(JDpaint) 5.5 绿色版精雕软件(JDpaint) 5.5 绿色版

pscad IEEE标准系统搭建说明

pscad IEEE标准系统搭建说明,包含典型的IEEE 9节点、14节点、30节点、39节点和118节点系统。内含标准测试系统的原始数据和潮流结果。

假人有限元模型

假人模型,车辆碰撞假人有限元模型,适合做乘员安全的研究者利用

上一篇: Hive工具的配置与使用
下一篇: Spark概述
PowerBlogger
博客等级 码龄8年 10粉丝 17原创
评论
成就一亿技术人!
拼手气红包6.0元
还能输入1000个字符
 
 条评论被折叠 查看
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付:10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值