Spark-SQL

最新推荐文章于 2020-05-19 20:59:11 发布

原创最新推荐文章于 2020-05-19 20:59:11 发布 · 532 阅读

0 ·

CC 4.0 BY-SA版权

文章标签：

#spark #sql

机器学习同时被 2 个专栏收录

2 篇文章

订阅专栏

大数据

0 篇文章

订阅专栏

Spark-SQL

1.直接通过脚本执行

注意指定使用的资源情况。
spark-sql –executor-cores 1 –executor-memory 2g

2.采用Beeline工具执行(mr用户)

beeline -u jdbc:hive2://localhost:18000 -n mr

HiveQL官方语法文档
https://cwiki.apache.org/confluence/display/Hive/LanguageManual

Example 1

hadoop fs -put /sample_fpgrowth.txt /user/mr/fpgrowth
CREATE EXTERNAL TABLE sample_fpgrowth(
        x1        STRING)
        LOCATION '/user/mr/fpgrowth/';

Example 2

hadoop fs -mkdir /user/mr/airline

hadoop fs -put /international-airline-passengers.csv /user/mr/airline/
CREATE EXTERNAL TABLE international_airline_passenger (
        x1        INT,
        x2        DOUBLE)
row format delimited fields terminated by ',' LOCATION '/user/mr/airline/';

Example3 Hive 0.14以上可用

drop table if exists test_zl;
create table test_zl (x1 INT, x2 DOUBLE, x3 DOUBLE) row format delimited fields terminated by ',';
insert overwrite into table test_zl values (1,2,3); //hive 0.14起支持
insert into table test_zl values( 2,2,3);

Iris数据集

create table iris ( sepal_length  double, sepal_width double, petal_length double, petal_width double, species string) row format delimited fields terminated by ',' LOCATION '/user/mr/iris/';

Titanic数据集

create table titanic_train (age int,passenger_class string,sex string,  no_of_siblings_or_spouses_on_board int, no_of_parents_or_children_on_board int, passenger_fare double, survived string) row format delimited fields terminated by ';' LOCATION '/user/mr/titanic_train/';

create table titanic_unlabeled (age int,passenger_class string,sex string,  no_of_siblings_or_spouses_on_board int, no_of_parents_or_children_on_board int, passenger_fare double) row format delimited fields terminated by ',' LOCATION '/user/mr/titanic_test/';

basket数据集

create table basket (basket string) row format delimited fields terminated by ';' LOCATION '/user/mr/basket/';