some command of executing hadoop spark hive kafka

最新推荐文章于 2022-04-15 18:50:34 发布

liangxw1

最新推荐文章于 2022-04-15 18:50:34 发布

阅读量442

点赞数

CC 4.0 BY-SA版权

分类专栏： bigdata command

本文链接：https://blog.youkuaiyun.com/liangxw1/article/details/51093979

command 同时被 2 个专栏收录

3 篇文章

订阅专栏

bigdata

2 篇文章

订阅专栏

本文介绍了Hadoop和Spark的基本操作，包括使用Hive创建表、加载数据、执行插入操作及表的删除等过程。此外还涉及了Kafka主题的创建与消息的生产和消费流程。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

sudo service iptables stop

sudo service mysqld start

jobs -l
kill -9 4500

haddop fs -ls /

./nohup ./hive --service metastore &
./hive

//Hive
create table table_0406 (
id int,
name string,
age int)
row format delimited fields terminated by ','
stored as textfile;

load data local inpath '/home/hadoop/hadoop_home/apache-hive-1.2.1-bin/test.txt' into table table_0406;

insert into table_0406(id, name,age) values(200,'lisi',20);

hadoop fs - put test.log /hive/warehousedir/table_0506/

//drop table
drop table table_0406;

//Spark
stop-all.sh
start-all.sh

cd hadoop-home/bin

./nohup ./hive --service metastore &

./hive

//Kafka fen bu shi

// Active MQ,

[hadoop@teradata1 bin]$ ./kafka-topics.sh --create --zookeeper teradata1:2181 --replication-factor 1 --partitions 1 --topic kafka_test

[hadoop@teradata1 bin]$ ./kafka-topics.sh --list -zookeeper teradata1:2181
kafka_test

[hadoop@teradata1 bin]$ ./kafka-topics.sh --describe -zookeeper teradata1:2181
Topic:kafka_test PartitionCount:1 ReplicationFactor:1 Configs:
Topic: kafka_test Partition: 0 Leader: 0 Replicas: 0 Isr: 0

[hadoop@teradata1 bin]$ ./kafka-console-producer.sh --broker-list teradata1:9092 --topic kafka_test

[hadoop@teradata1 bin]$ ./kafka-console-consumer.sh --zookeeper teradata1:2181 --topic kafka_test --from-beginning

Centos2
cd /spark_home/spark-1.6.0...../

bin/run-example org.apache.spark.examples.streaming.KafkaWordCount teradata1 my-consumer-group kafka_test 1