spark2.2.1+hadoop2.7.1伪分布式安装

最新推荐文章于 2025-09-20 16:11:54 发布
原创 最新推荐文章于 2025-09-20 16:11:54 发布 · 1.2k 阅读 · 0 ·
CC 4.0 BY-SA版权
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。

本文详细记录了Spark2.2.1在伪分布式环境下的安装步骤，包括配置环境变量、启动Spark进程等，并分享了启动spark-sql的操作流程及日志解析，展示了Spark在无Hive环境下的兼容性。

环境说明:jdk1.8+scala.2.11.8+spark2.2.1+hadoop.2.7.1

spark2.2.1伪分布式安装说明： 首先要安装好jdk+hadoop+scala

下载。上传、解压

配置spark/conf/spark-env.sh   

记得先把原始文件copy一份  cp spark-env.sh.temple spark-env.sh

vim spark-env.sh 

export JAVA_HOME=/usr/local/jdk
export SCALA_HOME=/usr/local/scala
export SPARK_WORKER_MEMORY=1G    //根据自己的机器的情况，可以不用写在配置文件里面。
export HADOOP_HOME=/usr/local/hadoop
export HADOOP_CONF_DIR=/usr/local/hadoop/etc/hadoop
export SPARK_MASTER_IP=localhost

完毕之后。cd ../sbin  

启动spark进程： ./start-all.sh    

jps  查看

发现多了master 和worker两个进程

。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。。

接下来是尝试spark-sql操作

首先要启动spark-sql

cd ../bin   执行./spark-sql --master spark://172.16.0.37:7077 --executor-memory 1g

这里首先贴一下，启动的日志：

18/08/16 08:23:24 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
18/08/16 08:23:24 INFO metastore.HiveMetaStore: 0: Opening raw store with implemenation class:org.apache.hadoop.hive.metastore.ObjectStore
18/08/16 08:23:24 INFO metastore.ObjectStore: ObjectStore, initialize called
18/08/16 08:23:24 INFO DataNucleus.Persistence: Property hive.metastore.integral.jdo.pushdown unknown - will be ignored
18/08/16 08:23:24 INFO DataNucleus.Persistence: Property datanucleus.cache.level2 unknown - will be ignored
18/08/16 08:23:26 INFO metastore.ObjectStore: Setting MetaStore object pin classes with hive.metastore.cache.pinobjtypes="Table,StorageDescriptor,SerDeInfo,Partition,Database,Type,FieldSchema,Order"
18/08/16 08:23:27 INFO DataNucleus.Datastore: The class "org.apache.hadoop.hive.metastore.model.MFieldSchema" is tagged as "embedded-only" so does not have its own datastore table.
18/08/16 08:23:27 INFO DataNucleus.Datastore: The class "org.apache.hadoop.hive.metastore.model.MOrder" is tagged as "embedded-only" so does not have its own datastore table.
18/08/16 08:23:28 INFO DataNucleus.Datastore: The class "org.apache.hadoop.hive.metastore.model.MFieldSchema" is tagged as "embedded-only" so does not have its own datastore table.
18/08/16 08:23:28 INFO DataNucleus.Datastore: The class "org.apache.hadoop.hive.metastore.model.MOrder" is tagged as "embedded-only" so does not have its own datastore table.
18/08/16 08:23:29 INFO metastore.MetaStoreDirectSql: Using direct SQL, underlying DB is DERBY
18/08/16 08:23:29 INFO metastore.ObjectStore: Initialized ObjectStore
18/08/16 08:23:29 WARN metastore.ObjectStore: Version information not found in metastore. hive.metastore.schema.verification is not enabled so recording the schema version 1.2.0
18/08/16 08:23:29 WARN metastore.ObjectStore: Failed to get database default, returning NoSuchObjectException
18/08/16 08:23:30 INFO metastore.HiveMetaStore: Added admin role in metastore
18/08/16 08:23:30 INFO metastore.HiveMetaStore: Added public role in metastore
18/08/16 08:23:30 INFO metastore.HiveMetaStore: No user is added in admin role, since config is empty
18/08/16 08:23:30 INFO metastore.HiveMetaStore: 0: get_all_databases
18/08/16 08:23:30 INFO HiveMetaStore.audit: ugi=hadoop  ip=unknown-ip-addr      cmd=get_all_databases
18/08/16 08:23:30 INFO metastore.HiveMetaStore: 0: get_functions: db=default pat=*
18/08/16 08:23:30 INFO HiveMetaStore.audit: ugi=hadoop  ip=unknown-ip-addr      cmd=get_functions: db=default pat=*
18/08/16 08:23:30 INFO DataNucleus.Datastore: The class "org.apache.hadoop.hive.metastore.model.MResourceUri" is tagged as "embedded-only" so does not have its own datastore table.
18/08/16 08:23:30 INFO session.SessionState: Created local directory: /tmp/3455c021-8c84-49a1-ad3b-037d5c706f68_resources
18/08/16 08:23:30 INFO session.SessionState: Created HDFS directory: /tmp/hive/hadoop/3455c021-8c84-49a1-ad3b-037d5c706f68
18/08/16 08:23:30 INFO session.SessionState: Created local directory: /tmp/hadoop/3455c021-8c84-49a1-ad3b-037d5c706f68
18/08/16 08:23:30 INFO session.SessionState: Created HDFS directory: /tmp/hive/hadoop/3455c021-8c84-49a1-ad3b-037d5c706f68/_tmp_space.db
18/08/16 08:23:30 INFO spark.SparkContext: Running Spark version 2.2.1
18/08/16 08:23:30 INFO spark.SparkContext: Submitted application: SparkSQL::172.16.0.37
18/08/16 08:23:30 INFO spark.SecurityManager: Changing view acls to: hadoop
18/08/16 08:23:30 INFO spark.SecurityManager: Changing modify acls to: hadoop
18/08/16 08:23:30 INFO spark.SecurityManager: Changing view acls groups to: 
18/08/16 08:23:30 INFO spark.SecurityManager: Changing modify acls groups to: 
18/08/16 08:23:30 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(hadoop); groups with view permissions: Set(); users  with modify permissions: Set(hadoop); groups with modify permissions: Set()
18/08/16 08:23:31 INFO util.Utils: Successfully started service 'sparkDriver' on port 33900.
18/08/16 08:23:31 INFO spark.SparkEnv: Registering MapOutputTracker
18/08/16 08:23:31 INFO spark.SparkEnv: Registering BlockManagerMaster
18/08/16 08:23:31 INFO storage.BlockManagerMasterEndpoint: Using org.apache.spark.storage.DefaultTopologyMapper for getting topology information
18/08/16 08:23:31 INFO storage.BlockManagerMasterEndpoint: BlockManagerMasterEndpoint up
18/08/16 08:23:31 INFO storage.DiskBlockManager: Created local directory at /tmp/blockmgr-730e28b9-c02c-4d5c-8b65-d8a490c234cf
18/08/16 08:23:31 INFO memory.MemoryStore: MemoryStore started with capacity 366.3 MB
18/08/16 08:23:31 INFO spark.SparkEnv: Registering OutputCommitCoordinator
18/08/16 08:23:31 INFO util.log: Logging initialized @8995ms
18/08/16 08:23:31 INFO server.Server: jetty-9.3.z-SNAPSHOT
18/08/16 08:23:31 INFO server.Server: Started @9087ms
18/08/16 08:23:31 WARN util.Utils: Service 'SparkUI' could not bind on port 4040. Attempting port 4041.
18/08/16 08:23:31 INFO server.AbstractConnector: Started ServerConnector@6533927b{HTTP/1.1,[http/1.1]}{0.0.0.0:4041}
18/08/16 08:23:31 INFO util.Utils: Successfully started service 'SparkUI' on port 4041.
18/08/16 08:23:31 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@5e5a8718{/jobs,null,AVAILABLE,@Spark}
18/08/16 08:23:31 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@38f617f4{/jobs/json,null,AVAILABLE,@Spark}
18/08/16 08:23:31 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@3dea226b{/jobs/job,null,AVAILABLE,@Spark}
18/08/16 08:23:31 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@7fc7152e{/jobs/job/json,null,AVAILABLE,@Spark}
18/08/16 08:23:31 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@6704df84{/stages,null,AVAILABLE,@Spark}
18/08/16 08:23:31 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@53bb71e5{/stages/json,null,AVAILABLE,@Spark}
18/08/16 08:23:31 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@15994b0b{/stages/stage,null,AVAILABLE,@Spark}
18/08/16 08:23:31 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@13aed42b{/stages/stage/json,null,AVAILABLE,@Spark}
18/08/16 08:23:31 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@78065fcd{/stages/pool,null,AVAILABLE,@Spark}
18/08/16 08:23:31 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@51ed2f68{/stages/pool/json,null,AVAILABLE,@Spark}
18/08/16 08:23:31 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@19b9f903{/storage,null,AVAILABLE,@Spark}
18/08/16 08:23:31 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@28cb86b2{/storage/json,null,AVAILABLE,@Spark}
18/08/16 08:23:31 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@748904e8{/storage/rdd,null,AVAILABLE,@Spark}
18/08/16 08:23:31 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@3b3056a6{/storage/rdd/json,null,AVAILABLE,@Spark}
18/08/16 08:23:31 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@51d8f2f2{/environment,null,AVAILABLE,@Spark}
18/08/16 08:23:31 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@72fb989b{/environment/json,null,AVAILABLE,@Spark}
18/08/16 08:23:31 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@597a7afa{/executors,null,AVAILABLE,@Spark}
18/08/16 08:23:31 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@7cdb7fc{/executors/json,null,AVAILABLE,@Spark}
18/08/16 08:23:31 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@586843bc{/executors/threadDump,null,AVAILABLE,@Spark}
18/08/16 08:23:31 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@7d8b66d9{/executors/threadDump/json,null,AVAILABLE,@Spark}
18/08/16 08:23:31 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@4ff66917{/static,null,AVAILABLE,@Spark}
18/08/16 08:23:31 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@40c76f5a{/,null,AVAILABLE,@Spark}
18/08/16 08:23:31 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@5546e754{/api,null,AVAILABLE,@Spark}
18/08/16 08:23:31 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@43e3a390{/jobs/job/kill,null,AVAILABLE,@Spark}
18/08/16 08:23:31 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@50dc49e1{/stages/stage/kill,null,AVAILABLE,@Spark}
18/08/16 08:23:31 INFO ui.SparkUI: Bound SparkUI to 0.0.0.0, and started at http://172.16.0.37:4041
18/08/16 08:23:31 INFO client.StandaloneAppClient$ClientEndpoint: Connecting to master spark://172.16.0.37:7077...
18/08/16 08:23:31 INFO client.TransportClientFactory: Successfully created connection to /172.16.0.37:7077 after 27 ms (0 ms spent in bootstraps)
18/08/16 08:23:32 INFO cluster.StandaloneSchedulerBackend: Connected to Spark cluster with app ID app-20180816082331-0000
18/08/16 08:23:32 INFO util.Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 44012.
18/08/16 08:23:32 INFO netty.NettyBlockTransferService: Server created on 172.16.0.37:44012
18/08/16 08:23:32 INFO storage.BlockManager: Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy
18/08/16 08:23:32 INFO storage.BlockManagerMaster: Registering BlockManager BlockManagerId(driver, 172.16.0.37, 44012, None)
18/08/16 08:23:32 INFO storage.BlockManagerMasterEndpoint: Registering block manager 172.16.0.37:44012 with 366.3 MB RAM, BlockManagerId(driver, 172.16.0.37, 44012, None)
18/08/16 08:23:32 INFO storage.BlockManagerMaster: Registered BlockManager BlockManagerId(driver, 172.16.0.37, 44012, None)
18/08/16 08:23:32 INFO storage.BlockManager: Initialized BlockManager: BlockManagerId(driver, 172.16.0.37, 44012, None)
18/08/16 08:23:32 INFO client.StandaloneAppClient$ClientEndpoint: Executor added: app-20180816082331-0000/0 on worker-20180816081203-172.16.0.37-37180 (172.16.0.37:37180) with 3 cores
18/08/16 08:23:32 INFO cluster.StandaloneSchedulerBackend: Granted executor ID app-20180816082331-0000/0 on hostPort 172.16.0.37:37180 with 3 cores, 700.0 MB RAM
18/08/16 08:23:32 INFO client.StandaloneAppClient$ClientEndpoint: Executor updated: app-20180816082331-0000/0 is now RUNNING
18/08/16 08:23:32 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@45e11627{/metrics/json,null,AVAILABLE,@Spark}
18/08/16 08:23:32 INFO cluster.StandaloneSchedulerBackend: SchedulerBackend is ready for scheduling beginning after reached minRegisteredResourcesRatio: 0.0
18/08/16 08:23:32 INFO internal.SharedState: Setting hive.metastore.warehouse.dir ('null') to the value of spark.sql.warehouse.dir ('file:/home/hadoop/apps/spark-2.2.1-bin-hadoop2.7/bin/spark-warehouse').
18/08/16 08:23:32 INFO internal.SharedState: Warehouse path is 'file:/home/hadoop/apps/spark-2.2.1-bin-hadoop2.7/bin/spark-warehouse'.
18/08/16 08:23:32 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@5d27b4c1{/SQL,null,AVAILABLE,@Spark}
18/08/16 08:23:32 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@4b508371{/SQL/json,null,AVAILABLE,@Spark}
18/08/16 08:23:32 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@270f28cf{/SQL/execution,null,AVAILABLE,@Spark}
18/08/16 08:23:32 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@467af68c{/SQL/execution/json,null,AVAILABLE,@Spark}
18/08/16 08:23:32 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@12c76d6e{/static/sql,null,AVAILABLE,@Spark}
18/08/16 08:23:32 INFO hive.HiveUtils: Initializing HiveMetastoreConnection version 1.2.1 using Spark classes.
18/08/16 08:23:33 INFO metastore.HiveMetaStore: 0: Opening raw store with implemenation class:org.apache.hadoop.hive.metastore.ObjectStore
18/08/16 08:23:33 INFO metastore.ObjectStore: ObjectStore, initialize called
18/08/16 08:23:33 INFO DataNucleus.Persistence: Property hive.metastore.integral.jdo.pushdown unknown - will be ignored
18/08/16 08:23:33 INFO DataNucleus.Persistence: Property datanucleus.cache.level2 unknown - will be ignored
18/08/16 08:23:35 INFO cluster.CoarseGrainedSchedulerBackend$DriverEndpoint: Registered executor NettyRpcEndpointRef(spark-client://Executor) (172.16.0.37:49208) with ID 0
18/08/16 08:23:35 INFO storage.BlockManagerMasterEndpoint: Registering block manager 172.16.0.37:42821 with 193.5 MB RAM, BlockManagerId(0, 172.16.0.37, 42821, None)
18/08/16 08:23:36 INFO metastore.ObjectStore: Setting MetaStore object pin classes with hive.metastore.cache.pinobjtypes="Table,StorageDescriptor,SerDeInfo,Partition,Database,Type,FieldSchema,Order"
18/08/16 08:23:37 INFO DataNucleus.Datastore: The class "org.apache.hadoop.hive.metastore.model.MFieldSchema" is tagged as "embedded-only" so does not have its own datastore table.
18/08/16 08:23:37 INFO DataNucleus.Datastore: The class "org.apache.hadoop.hive.metastore.model.MOrder" is tagged as "embedded-only" so does not have its own datastore table.
18/08/16 08:23:38 INFO DataNucleus.Datastore: The class "org.apache.hadoop.hive.metastore.model.MFieldSchema" is tagged as "embedded-only" so does not have its own datastore table.
18/08/16 08:23:38 INFO DataNucleus.Datastore: The class "org.apache.hadoop.hive.metastore.model.MOrder" is tagged as "embedded-only" so does not have its own datastore table.
18/08/16 08:23:38 INFO metastore.MetaStoreDirectSql: Using direct SQL, underlying DB is DERBY
18/08/16 08:23:38 INFO metastore.ObjectStore: Initialized ObjectStore
18/08/16 08:23:39 WARN metastore.ObjectStore: Version information not found in metastore. hive.metastore.schema.verification is not enabled so recording the schema version 1.2.0
18/08/16 08:23:39 WARN metastore.ObjectStore: Failed to get database default, returning NoSuchObjectException
18/08/16 08:23:39 INFO metastore.HiveMetaStore: Added admin role in metastore
18/08/16 08:23:39 INFO metastore.HiveMetaStore: Added public role in metastore
18/08/16 08:23:39 INFO metastore.HiveMetaStore: No user is added in admin role, since config is empty
18/08/16 08:23:39 INFO metastore.HiveMetaStore: 0: get_all_databases
18/08/16 08:23:39 INFO HiveMetaStore.audit: ugi=hadoop  ip=unknown-ip-addr      cmd=get_all_databases
18/08/16 08:23:39 INFO metastore.HiveMetaStore: 0: get_functions: db=default pat=*
18/08/16 08:23:39 INFO HiveMetaStore.audit: ugi=hadoop  ip=unknown-ip-addr      cmd=get_functions: db=default pat=*
18/08/16 08:23:39 INFO DataNucleus.Datastore: The class "org.apache.hadoop.hive.metastore.model.MResourceUri" is tagged as "embedded-only" so does not have its own datastore table.
18/08/16 08:23:39 INFO session.SessionState: Created local directory: /tmp/2cc14b21-0441-4d07-963e-2350d3148e9f_resources
18/08/16 08:23:39 INFO session.SessionState: Created HDFS directory: /tmp/hive/hadoop/2cc14b21-0441-4d07-963e-2350d3148e9f
18/08/16 08:23:39 INFO session.SessionState: Created local directory: /tmp/hadoop/2cc14b21-0441-4d07-963e-2350d3148e9f
18/08/16 08:23:39 INFO session.SessionState: Created HDFS directory: /tmp/hive/hadoop/2cc14b21-0441-4d07-963e-2350d3148e9f/_tmp_space.db
18/08/16 08:23:39 INFO client.HiveClientImpl: Warehouse location for Hive client (version 1.2.1) is file:/home/hadoop/apps/spark-2.2.1-bin-hadoop2.7/bin/spark-warehouse
18/08/16 08:23:39 INFO metastore.HiveMetaStore: 0: get_database: default
18/08/16 08:23:39 INFO HiveMetaStore.audit: ugi=hadoop  ip=unknown-ip-addr      cmd=get_database: default
18/08/16 08:23:40 INFO session.SessionState: Created local directory: /tmp/c96dfcc8-92ea-4fac-837b-a2de2344a4f3_resources
18/08/16 08:23:40 INFO session.SessionState: Created HDFS directory: /tmp/hive/hadoop/c96dfcc8-92ea-4fac-837b-a2de2344a4f3
18/08/16 08:23:40 INFO session.SessionState: Created local directory: /tmp/hadoop/c96dfcc8-92ea-4fac-837b-a2de2344a4f3
18/08/16 08:23:40 INFO session.SessionState: Created HDFS directory: /tmp/hive/hadoop/c96dfcc8-92ea-4fac-837b-a2de2344a4f3/_tmp_space.db
18/08/16 08:23:40 INFO client.HiveClientImpl: Warehouse location for Hive client (version 1.2.1) is file:/home/hadoop/apps/spark-2.2.1-bin-hadoop2.7/bin/spark-warehouse
18/08/16 08:23:40 INFO metastore.HiveMetaStore: 0: get_database: global_temp
18/08/16 08:23:40 INFO HiveMetaStore.audit: ugi=hadoop  ip=unknown-ip-addr      cmd=get_database: global_temp
18/08/16 08:23:40 WARN metastore.ObjectStore: Failed to get database global_temp, returning NoSuchObjectException
18/08/16 08:23:40 INFO session.SessionState: Created local directory: /tmp/6646afbf-c828-4682-8e77-e40b8181272b_resources
18/08/16 08:23:40 INFO session.SessionState: Created HDFS directory: /tmp/hive/hadoop/6646afbf-c828-4682-8e77-e40b8181272b
18/08/16 08:23:40 INFO session.SessionState: Created local directory: /tmp/hadoop/6646afbf-c828-4682-8e77-e40b8181272b
18/08/16 08:23:40 INFO session.SessionState: Created HDFS directory: /tmp/hive/hadoop/6646afbf-c828-4682-8e77-e40b8181272b/_tmp_space.db
18/08/16 08:23:40 INFO client.HiveClientImpl: Warehouse location for Hive client (version 1.2.1) is file:/home/hadoop/apps/spark-2.2.1-bin-hadoop2.7/bin/spark-warehouse
18/08/16 08:23:40 INFO state.StateStoreCoordinatorRef: Registered StateStoreCoordinator endpoint
spark-sql> 

说明一下：这里我还没有安装hive环境，所以上面启动的时候报的warn没有找到hive的metordata的信息，但是spark2.2.x会默认在没有hive的环境下面也是可以运行的，它会在本地的spark安装目录下面spark/bin/spark-warehouse下面创建临时目录保存hive元数据信息。目录如下：这个metastore_db是临时目录

 

该文件下面会有如下几个文件：

另外在hdfs上面的/tmp/hive/hadoop/下面产生临时文件

1.Hadoop的安装和使用（华为云学习笔记，Spark编程基础，大数据）
GCPOP的博客
05-24 2792
此笔记为第一篇，学校开放华为云平台，帮助我们学习有关大数据方面相关知识的学习笔记，因为是云平台，是已经搭建好linux环境了，使用的是Ubantu。精心整理，自学笔记，如有什么问题，请耐心指正 Hadoop的安装和使用 学习Hadoop的安装和使用 1 安装Hadoop前的准备工作 介绍安装Hadoop之前的一些准备工作，包括创建hadoop用户、安装SSH和安装Java环境等。 先配置hosts，输入以下命令，将本机的hostname添加到hosts文件中： 拷贝代码echo "127.0.0.1 "$H
自制Hadoop伪分布、集群安装详细过程（vmware）
Fortuna_i的博客
07-20 2428
Hadoop单机版、伪分布、集群安装教程推荐链接： http://dblab.xmu.edu.cn/blog/page/2/?s=Hadoop 注：vmware6.5及以上，执行命令基本相同，除了编辑文件时“vi”指令替换为“vim”指令 一、下载安装所需文件          1.vmware10中文版          2.centos6.4          3.jdk1.8-L...
spark-2.2.2-bin-hadoop2.7 安装
drl_blogs的博客
06-14 3549
1.上传spark-2.2.2-bin-hadoop2.7.tgz 2.解压文件 tar -zxvf spark-2.2.2-bin-hadoop2.7.tgz -C /usr/local/ 3.进入conf/下把spark-env.sh.template改为spark-env.sh cd /usr/local/spark-2.2.2-bin-hadoop2.7/conf/ mv s...
Hadoop2.7.5+Spark2.2.1分布式集群搭建
cmgx1986的博客
03-24 294
一、运行环境 CentOS 6.5 Spark 2.2.1 Hadoop 2.7.5 Java JDK 1.8 Scala 2.12.5 二、节点IP及角色对应关系 节点名 ...
Hadoop伪分布式环境搭建全攻略
最新发布
2401_87046623的博客
09-20 1014
本文详细介绍了在Ubuntu系统上搭建Hadoop伪分布式环境的完整流程。主要内容包括：1）更新apt并安装必要工具；2）配置SSH免密登录；3）安装配置Java环境；4）下载解压Hadoop并配置环境变量；5）修改core-site.xml和hdfs-site.xml配置文件；6）初始化HDFS并启动集群；7）运行WordCount示例验证集群；8）配置Hadoop全局环境变量。整个过程涵盖了从基础环境准备到Hadoop集群部署的关键步骤，并解释了各项配置参数的具体含义，为大数据学习提供了实践指导。
spark-2.2.0 集群安装部署以及hadoop集群部署
怀素的专栏
08-05 5217
Spark在生产环境中，主要部署在安装Linux系统的集群中。在linux系统中安装Spark需要预先安装JDK、Scala等所需要的依赖。 由于Spark是计算框架，所以需要预先在集群内有搭建好存储数据的持久化层，如HDFS、Hive、Cassandra等，最后可以通过启动脚本运行应用。 1.安装JDK OracleJDK下载地址：http://www.oracle.com/techne...
spark with hive
zphyy1988的博客
03-08 429
标题标题前提-已搭建完成： hive3.1 安装配置（一） 1.将hive-site.xml 拷贝到spark目录下 cp $HIVE_HOME/conf/hive-site.xml $SPARK_HOME/conf/hive-site.xml 2.编写测试案例 val sparkSession = SparkSession.builder. .appName("spark sql ...
Spark 和 Hadoop 本地【window】环境搭建
山鬼谣的专栏
06-20 1859
环境 Spark：2.3.1 Hadoop：2.7.6 java：1.8 前言 最近主要是在学习Spark，根据官网的文档，想本地跑个小程序看看效果； https://spark.apache.org/docs/latest/quick-start.html#self-contained-applications 具体想执行的官方代码如下： /* SimpleApp.java ...
基于Docker搭建hdfs分布式实验环境
u013242054的博客
10-20 2075
HDFS（Hadoop Distributed File System），作为Google File System（GFS）的实现，是Hadoop项目的核心子项目，是分布式计算中数据存储管理的基础，是基于流数据模式访问和处理超大文件的需求而开发的，可以运行于廉价的商用服务器上。导入成功后，源tar文件可以删除。/usr/local/hadoop/bin/hdfs dfs -put /usr/local/hadoop/spark.tar.gz /test # 将刚压缩的文件上传到/test目录上。
/home/hadoopmaster/jdk1.8.0_161/bin/java -javaagent:/home/hadoopmaster/idea-IC-221.6008.13/lib/idea_rt.jar=34515:/home/hadoopmaster/idea-IC-221.6008.13/bin -Dfile.encoding=UTF-8 -classpath /home/hadoopmaster/jdk1.8.0_161/jre/lib/charsets.jar:/home/hadoopmaster/jdk1.8.0_161/jre/lib/deploy.jar:/home/hadoopmaster/jdk1.8.0_161/jre/lib/ext/cldrdata.jar:/home/hadoopmaster/jdk1.8.0_161/jre/lib/ext/dnsns.jar:/home/hadoopmaster/jdk1.8.0_161/jre/lib/ext/jaccess.jar:/home/hadoopmaster/jdk1.8.0_161/jre/lib/ext/jfxrt.jar:/home/hadoopmaster/jdk1.8.0_161/jre/lib/ext/localedata.jar:/home/hadoopmaster/jdk1.8.0_161/jre/lib/ext/nashorn.jar:/home/hadoopmaster/jdk1.8.0_161/jre/lib/ext/sunec.jar:/home/hadoopmaster/jdk1.8.0_161/jre/lib/ext/sunjce_provider.jar:/home/hadoopmaster/jdk1.8.0_161/jre/lib/ext/sunpkcs11.jar:/home/hadoopmaster/jdk1.8.0_161/jre/lib/ext/zipfs.jar:/home/hadoopmaster/jdk1.8.0_161/jre/lib/javaws.jar:/home/hadoopmaster/jdk1.8.0_161/jre/lib/jce.jar:/home/hadoopmaster/jdk1.8.0_161/jre/lib/jfr.jar:/home/hadoopmaster/jdk1.8.0_161/jre/lib/jfxswt.jar:/home/hadoopmaster/jdk1.8.0_161/jre/lib/jsse.jar:/home/hadoopmaster/jdk1.8.0_161/jre/lib/management-agent.jar:/home/hadoopmaster/jdk1.8.0_161/jre/lib/plugin.jar:/home/hadoopmaster/jdk1.8.0_161/jre/lib/resources.jar:/home/hadoopmaster/jdk1.8.0_161/jre/lib/rt.jar:/root/IdeaProjects/kkk/out/production/kkk:/home/hadoopmaster/scala-2.12.15/lib/scala-reflect.jar:/home/hadoopmaster/scala-2.12.15/lib/scala-xml_2.12-1.0.6.jar:/home/hadoopmaster/scala-2.12.15/lib/scala-parser-combinators_2.12-1.0.7.jar:/home/hadoopmaster/scala-2.12.15/lib/scala-swing_2.12-2.0.3.jar:/home/hadoopmaster/scala-2.12.15/lib/scala-library.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/xz-1.8.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/jta-1.1.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/jpam-1.1.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/json-1.8.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/ST4-4.0.4.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/guice-3.0.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/ivy-2.5.0.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/oro-2.0.8.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/blas-2.2.1.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/core-1.1.2.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/gson-2.2.4.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/tink-1.6.0.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/avro-1.10.2.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/jsp-api-2.1.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/okio-1.14.0.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/opencsv-2.3.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/shims-0.9.0.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/xmlenc-0.52.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/arpack-2.2.1.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/guava-14.0.1.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/jetty-6.1.26.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/jline-2.14.6.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/jsr305-3.0.0.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/lapack-2.2.1.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/log4j-1.2.17.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/minlog-1.3.0.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/stream-2.9.6.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/velocity-1.5.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/generex-1.0.2.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/hk2-api-2.6.1.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/janino-3.0.16.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/jdo-api-3.0.1.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/objenesis-2.6.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/paranamer-2.8.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/py4j-0.10.9.3.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/pyrolite-4.30.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/HikariCP-2.5.1.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/commons-io-2.4.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/hive-cli-2.3.9.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/javax.inject-1.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/libfb303-0.9.3.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/lz4-java-1.7.1.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/okhttp-3.12.12.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/snakeyaml-1.27.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/stax-api-1.0.1.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/JTransforms-3.1.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/aopalliance-1.0.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/avro-ipc-1.10.2.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/breeze_2.12-1.2.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/commons-cli-1.2.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/commons-net-3.1.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/derby-10.14.2.0.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/hive-jdbc-2.3.9.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/hk2-utils-2.6.1.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/httpcore-4.4.14.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/jaxb-api-2.2.11.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/jersey-hk2-2.34.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/jodd-core-3.5.2.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/orc-core-1.6.12.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/super-csv-2.2.0.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/xml-apis-1.4.01.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/zookeeper-3.6.2.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/JLargeArrays-1.5.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/activation-1.1.1.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/automaton-1.11-8.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/commons-dbcp-1.4.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/commons-lang-2.6.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/commons-text-1.6.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/hive-serde-2.3.9.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/hive-shims-2.3.9.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/javolution-5.5.1.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/libthrift-0.12.0.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/orc-shims-1.6.12.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/slf4j-api-1.7.30.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/zjsonpatch-0.3.0.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/zstd-jni-1.5.0-4.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/chill-java-0.10.0.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/chill_2.12-0.10.0.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/guice-servlet-3.0.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/hadoop-auth-2.7.4.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/hadoop-hdfs-2.7.4.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/hive-common-2.3.9.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/hk2-locator-2.6.1.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/httpclient-4.5.13.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/jackson-xc-1.9.13.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/jetty-util-6.1.26.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/joda-time-2.10.10.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/kryo-shaded-4.0.2.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/metrics-jmx-4.2.0.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/metrics-jvm-4.2.0.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/rocksdbjni-6.20.3.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/spire_2.12-0.17.0.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/xercesImpl-2.12.0.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/aircompressor-0.21.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/algebra_2.12-2.0.1.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/annotations-17.0.0.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/antlr4-runtime-4.8.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/api-util-1.0.0-M20.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/arrow-format-2.0.0.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/arrow-vector-2.0.0.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/avro-mapred-1.10.2.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/commons-codec-1.15.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/commons-pool-1.5.4.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/compress-lzf-1.0.3.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/hive-beeline-2.3.9.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/javax.jdo-3.2.0-m3.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/jaxb-runtime-2.3.2.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/jersey-client-2.34.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/jersey-common-2.34.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/jersey-server-2.34.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/leveldbjni-all-1.8.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/metrics-core-4.2.0.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/metrics-json-4.2.0.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/RoaringBitmap-0.9.0.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/antlr-runtime-3.5.2.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/commons-math3-3.4.1.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/hadoop-client-2.7.4.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/hadoop-common-2.7.4.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/jackson-core-2.12.3.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/javassist-3.25.0-GA.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/jul-to-slf4j-1.7.30.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/protobuf-java-2.5.0.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/snappy-java-1.1.8.4.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/transaction-api-1.1.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/bonecp-0.8.0.RELEASE.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/commons-crypto-1.1.0.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/commons-digester-1.8.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/commons-lang3-3.12.0.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/curator-client-2.7.1.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/hive-exec-2.3.9-core.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/hive-metastore-2.3.9.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/jackson-jaxrs-1.9.13.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/jakarta.inject-2.6.1.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/orc-mapreduce-1.6.12.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/scala-xml_2.12-1.2.0.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/shapeless_2.12-2.3.3.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/slf4j-log4j12-1.7.30.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/spark-sql_2.12-3.2.1.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/threeten-extra-1.5.0.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/zookeeper-jute-3.6.2.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/commons-compress-1.21.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/commons-logging-1.1.3.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/curator-recipes-2.7.1.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/hadoop-yarn-api-2.7.4.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/hive-shims-0.23-2.3.9.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/jcl-over-slf4j-1.7.30.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/parquet-column-1.12.2.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/parquet-common-1.12.2.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/parquet-hadoop-1.12.2.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/scala-library-2.12.15.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/scala-reflect-2.12.15.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/spark-core_2.12-3.2.1.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/spark-hive_2.12-3.2.1.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/spark-repl_2.12-3.2.1.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/spark-tags_2.12-3.2.1.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/spark-yarn_2.12-3.2.1.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/api-asn1-api-1.0.0-M20.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/breeze-macros_2.12-1.2.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/cats-kernel_2.12-2.1.1.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/commons-httpclient-3.1.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/flatbuffers-java-1.9.0.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/hive-llap-common-2.3.9.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/hive-service-rpc-3.1.2.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/hive-storage-api-2.7.2.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/jetty-sslengine-6.1.26.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/metrics-graphite-4.2.0.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/netty-all-4.1.68.Final.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/parquet-jackson-1.12.2.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/scala-compiler-2.12.15.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/spark-mesos_2.12-3.2.1.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/spark-mllib_2.12-3.2.1.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/spire-util_2.12-0.17.0.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/xbean-asm9-shaded-4.20.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/apacheds-i18n-2.0.0-M15.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/arpack_combined_all-0.1.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/arrow-memory-core-2.0.0.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/commons-beanutils-1.9.4.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/commons-compiler-3.0.16.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/curator-framework-2.7.1.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/datanucleus-core-4.1.17.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/hive-shims-common-2.3.9.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/jackson-core-asl-1.9.13.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/jackson-databind-2.12.3.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/jakarta.ws.rs-api-2.1.6.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/kubernetes-client-5.4.1.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/macro-compat_2.12-1.1.1.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/parquet-encoding-1.12.2.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/spark-graphx_2.12-3.2.1.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/spark-sketch_2.12-3.2.1.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/spark-unsafe_2.12-3.2.1.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/univocity-parsers-2.9.1.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/arrow-memory-netty-2.0.0.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/datanucleus-rdbms-4.1.19.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/hadoop-annotations-2.7.4.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/hadoop-yarn-client-2.7.4.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/hadoop-yarn-common-2.7.4.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/spark-kvstore_2.12-3.2.1.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/spire-macros_2.12-0.17.0.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/commons-collections-3.2.2.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/commons-configuration-1.6.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/datanucleus-api-jdo-4.2.4.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/jackson-mapper-asl-1.9.13.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/jakarta.servlet-api-4.0.3.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/json4s-ast_2.12-3.7.0-M11.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/spark-catalyst_2.12-3.2.1.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/spark-launcher_2.12-3.2.1.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/audience-annotations-0.5.0.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/hive-shims-scheduler-2.3.9.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/hive-vector-code-gen-2.3.9.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/jackson-annotations-2.12.3.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/jakarta.xml.bind-api-2.3.2.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/json4s-core_2.12-3.7.0-M11.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/spark-streaming_2.12-3.2.1.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/spire-platform_2.12-0.17.0.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/kubernetes-model-apps-5.4.1.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/kubernetes-model-core-5.4.1.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/kubernetes-model-node-5.4.1.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/kubernetes-model-rbac-5.4.1.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/logging-interceptor-3.12.12.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/mesos-1.4.0-shaded-protobuf.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/osgi-resource-locator-1.0.3.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/spark-kubernetes_2.12-3.2.1.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/spark-tags_2.12-3.2.1-tests.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/aopalliance-repackaged-2.6.1.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/htrace-core-3.1.0-incubating.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/istack-commons-runtime-3.0.8.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/jakarta.annotation-api-1.3.5.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/jakarta.validation-api-2.0.2.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/json4s-scalap_2.12-3.7.0-M11.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/kubernetes-model-batch-5.4.1.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/spark-mllib-local_2.12-3.2.1.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/jersey-container-servlet-2.34.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/json4s-jackson_2.12-3.7.0-M11.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/kubernetes-model-common-5.4.1.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/kubernetes-model-events-5.4.1.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/kubernetes-model-policy-5.4.1.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/jackson-dataformat-yaml-2.12.3.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/jackson-datatype-jsr310-2.11.2.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/kubernetes-model-metrics-5.4.1.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/hadoop-yarn-server-common-2.7.4.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/spark-network-common_2.12-3.2.1.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/jackson-module-scala_2.12-2.12.3.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/kubernetes-model-discovery-5.4.1.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/parquet-format-structures-1.12.2.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/spark-network-shuffle_2.12-3.2.1.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/apacheds-kerberos-codec-2.0.0-M15.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/hadoop-mapreduce-client-app-2.7.4.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/kubernetes-model-extensions-5.4.1.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/kubernetes-model-networking-5.4.1.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/kubernetes-model-scheduling-5.4.1.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/hadoop-mapreduce-client-core-2.7.4.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/hadoop-yarn-server-web-proxy-2.7.4.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/jersey-container-servlet-core-2.34.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/kubernetes-model-autoscaling-5.4.1.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/kubernetes-model-flowcontrol-5.4.1.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/scala-collection-compat_2.12-2.1.1.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/spark-hive-thriftserver_2.12-3.2.1.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/kubernetes-model-certificates-5.4.1.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/kubernetes-model-coordination-5.4.1.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/kubernetes-model-storageclass-5.4.1.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/scala-parser-combinators_2.12-1.1.2.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/hadoop-mapreduce-client-common-2.7.4.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/kubernetes-model-apiextensions-5.4.1.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/hadoop-mapreduce-client-shuffle-2.7.4.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/hadoop-mapreduce-client-jobclient-2.7.4.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/kubernetes-model-admissionregistration-5.4.1.jar:/home/hadoopmaster/spark-3.2.1-bin-hadoop2.7/jars/dropwizard-metrics-hadoop-metrics2-reporter-0.1.2.jar kkk.WordCount Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties 25/06/02 20:17:03 INFO SparkContext: Running Spark version 3.2.1 25/06/02 20:17:04 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 25/06/02 20:17:04 INFO ResourceUtils: ============================================================== 25/06/02 20:17:04 INFO ResourceUtils: No custom resources configured for spark.driver. 25/06/02 20:17:04 INFO ResourceUtils: ============================================================== 25/06/02 20:17:04 INFO SparkContext: Submitted application: WordCount 25/06/02 20:17:04 INFO ResourceProfile: Default ResourceProfile created, executor resources: Map(cores -> name: cores, amount: 1, script: , vendor: , memory -> name: memory, amount: 1024, script: , vendor: , offHeap -> name: offHeap, amount: 0, script: , vendor: ), task resources: Map(cpus -> name: cpus, amount: 1.0) 25/06/02 20:17:04 INFO ResourceProfile: Limiting resource is cpu 25/06/02 20:17:04 INFO ResourceProfileManager: Added ResourceProfile id: 0 25/06/02 20:17:04 INFO SecurityManager: Changing view acls to: root 25/06/02 20:17:04 INFO SecurityManager: Changing modify acls to: root 25/06/02 20:17:04 INFO SecurityManager: Changing view acls groups to: 25/06/02 20:17:04 INFO SecurityManager: Changing modify acls groups to: 25/06/02 20:17:04 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(root); groups with view permissions: Set(); users with modify permissions: Set(root); groups with modify permissions: Set() 25/06/02 20:17:05 INFO Utils: Successfully started service 'sparkDriver' on port 41615. 25/06/02 20:17:05 INFO SparkEnv: Registering MapOutputTracker 25/06/02 20:17:05 INFO SparkEnv: Registering BlockManagerMaster 25/06/02 20:17:05 INFO BlockManagerMasterEndpoint: Using org.apache.spark.storage.DefaultTopologyMapper for getting topology information 25/06/02 20:17:05 INFO BlockManagerMasterEndpoint: BlockManagerMasterEndpoint up 25/06/02 20:17:05 INFO SparkEnv: Registering BlockManagerMasterHeartbeat 25/06/02 20:17:05 INFO DiskBlockManager: Created local directory at /tmp/blockmgr-68486311-3f69-48e8-8f69-e7afffcf5979 25/06/02 20:17:05 INFO MemoryStore: MemoryStore started with capacity 258.5 MiB 25/06/02 20:17:05 INFO SparkEnv: Registering OutputCommitCoordinator 25/06/02 20:17:06 INFO Utils: Successfully started service 'SparkUI' on port 4040. 25/06/02 20:17:06 INFO SparkUI: Bound SparkUI to 0.0.0.0, and started at http://hadoopmaster:4040 25/06/02 20:17:06 INFO Executor: Starting executor ID driver on host hadoopmaster 25/06/02 20:17:06 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 41907. 25/06/02 20:17:06 INFO NettyBlockTransferService: Server created on hadoopmaster:41907 25/06/02 20:17:06 INFO BlockManager: Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy 25/06/02 20:17:06 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(driver, hadoopmaster, 41907, None) 25/06/02 20:17:06 INFO BlockManagerMasterEndpoint: Registering block manager hadoopmaster:41907 with 258.5 MiB RAM, BlockManagerId(driver, hadoopmaster, 41907, None) 25/06/02 20:17:06 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(driver, hadoopmaster, 41907, None) 25/06/02 20:17:06 INFO BlockManager: Initialized BlockManager: BlockManagerId(driver, hadoopmaster, 41907, None) 25/06/02 20:17:08 INFO MemoryStore: Block broadcast_0 stored as values in memory (estimated size 244.0 KiB, free 258.2 MiB) 25/06/02 20:17:09 INFO MemoryStore: Block broadcast_0_piece0 stored as bytes in memory (estimated size 23.4 KiB, free 258.2 MiB) 25/06/02 20:17:09 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on hadoopmaster:41907 (size: 23.4 KiB, free: 258.5 MiB) 25/06/02 20:17:09 INFO SparkContext: Created broadcast 0 from textFile at WordCount.scala:9 Exception in thread "main" org.apache.hadoop.mapred.InvalidInputException: Input path does not exist: file:/home/hadoopmaster/words.txt at org.apache.hadoop.mapred.FileInputFormat.singleThreadedListStatus(FileInputFormat.java:287) at org.apache.hadoop.mapred.FileInputFormat.listStatus(FileInputFormat.java:229) at org.apache.hadoop.mapred.FileInputFormat.getSplits(FileInputFormat.java:315) at org.apache.spark.rdd.HadoopRDD.getPartitions(HadoopRDD.scala:205) at org.apache.spark.rdd.RDD.$anonfun$partitions$2(RDD.scala:300) at scala.Option.getOrElse(Option.scala:189) at org.apache.spark.rdd.RDD.partitions(RDD.scala:296) at org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:49) at org.apache.spark.rdd.RDD.$anonfun$partitions$2(RDD.scala:300) at scala.Option.getOrElse(Option.scala:189) at org.apache.spark.rdd.RDD.partitions(RDD.scala:296) at org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:49) at org.apache.spark.rdd.RDD.$anonfun$partitions$2(RDD.scala:300) at scala.Option.getOrElse(Option.scala:189) at org.apache.spark.rdd.RDD.partitions(RDD.scala:296) at org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:49) at org.apache.spark.rdd.RDD.$anonfun$partitions$2(RDD.scala:300) at scala.Option.getOrElse(Option.scala:189) at org.apache.spark.rdd.RDD.partitions(RDD.scala:296) at org.apache.spark.Partitioner$.$anonfun$defaultPartitioner$4(Partitioner.scala:78) at org.apache.spark.Partitioner$.$anonfun$defaultPartitioner$4$adapted(Partitioner.scala:78) at scala.collection.immutable.List.map(List.scala:293) at org.apache.spark.Partitioner$.defaultPartitioner(Partitioner.scala:78) at org.apache.spark.rdd.PairRDDFunctions.$anonfun$reduceByKey$4(PairRDDFunctions.scala:322) at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151) at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112) at org.apache.spark.rdd.RDD.withScope(RDD.scala:414) at org.apache.spark.rdd.PairRDDFunctions.reduceByKey(PairRDDFunctions.scala:322) at kkk.WordCount$.main(WordCount.scala:10) at kkk.WordCount.main(WordCount.scala) 25/06/02 20:17:09 INFO SparkContext: Invoking stop() from shutdown hook 25/06/02 20:17:09 INFO SparkUI: Stopped Spark web UI at http://hadoopmaster:4040 25/06/02 20:17:09 INFO MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped! 25/06/02 20:17:09 INFO MemoryStore: MemoryStore cleared 25/06/02 20:17:09 INFO BlockManager: BlockManager stopped 25/06/02 20:17:09 INFO BlockManagerMaster: BlockManagerMaster stopped 25/06/02 20:17:09 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped! 25/06/02 20:17:10 INFO SparkContext: Successfully stopped SparkContext 25/06/02 20:17:10 INFO ShutdownHookManager: Shutdown hook called 25/06/02 20:17:10 INFO ShutdownHookManager: Deleting directory /tmp/spark-213e3a91-227c-4e0a-8254-ab5c0f00786d Process finished with exit code 1
06-04
深层需求其实是理解Spark文件读取机制的核心差异——HDFS分布式存储vs本地文件系统。需要特别强调spark-shell默认行为这个关键点，因为很多新手会忽略shell环境预置的HDFS配置。注意到用户引用了5个相关资料，其中[1...
【图书推荐】《Spark入门与大数据分析实战》
夏天又到了的专栏
06-14 1602
本书基于Spark 3.3.1框架展开，系统介绍Spark生态系统各组件的操作，以及相应的大数据分析方法。本书各章节均提供丰富的示例及其详细的操作步骤，并配套示例源码、PPT课件和教学大纲。本书共分11章，内容包括Scala编程基础、Spark框架全生态体验、Spark RDD、Spark SQL、Kafka、Spark Streaming、Spark ML、Spark GraphX、Redis等技术框架和应用，并通过广告点击实时大数据分析和电影影评大数据分析两个综合项目进行实战提升。
Spark-shell上运行sql命令报错 WARN metastore.ObjectStore
DataAlgo
08-28 3582
在spark-shell上运行sql语句，报错 WARN metastore.ObjectStore: Failed to get database default, returning NoSuchObjectException 刚开始在自己的IDEA上都可以编译成功sql部分的代码， 没想到在client上翻了车。 google了好一会才发现是因为spark上conf里面缺少hive配置文件。 很简单，找到hive里conf的下hive-site.xml文件 拷贝到spark的conf目录下 cp h
Spark对接Hive：整合Hive操作及函数
muyingmiao的专栏
03-12 984
1.拷贝hive-site.xml文件到spark的conf目录下 2.[hadoop@hadoop002 bin]$ ./spark-shell --master local[2] --jars ~/software/mysql-connector-java-5.1.47.jar 注意用5版本的mysql-connector-java scala> spark.sql("sh...
spark整合hive后spark-sql启动报错
天龙八部
12-14 2719
通过spark-sql 来访问hive的元数据，hive元数据信息存储在mysql但是启动的时候是报如下的错误，检查过配置,metastore已经配置，并且启动，用hive启动可以正常使用，但是spark-sql不行，报错信息 Hive: Failed to access metastore. This class should not accessed in runtime. org.apac...
Spark的java开发环境实战
rishengcsdn的专栏
09-11 8622
看网上文档要用Spark开发搭建环境是挺麻烦的，需要多台机器安装Hadoop，还要建立和修改许多配置文件，有没有简单一点的方法呢？ 作者研究了一下，发现了一种非常简便的方法，能让小盆友们快速上手Spark又避免复杂安装命令。 研究了Spark的支持方面，我发现Spark2.2和mongodb3.4这两个家伙已经宣布了互相支持。这下子狼狈为奸，一脚踢开复杂难用的Hadoop了。
Spark--SparkSql--spark.sql.warehouse.dir使用详解
数据科学 爱好分享 快乐就完了
07-15 6955
对spark。sql中spark.sql.warehouse的详解
Spark(2.4.3) 快速开始
shareing
05-26 1253
使用Spark Shell进行交互式分析 基本 Spark的shell提供了一种学习API的简单方法，以及一种以交互方式分析数据的强大工具。它可以在Scala（在Java VM上运行，因此是使用现有Java库的好方法）或Python中使用。通过在Spark目录中运行以下命令来启动它： ./bin/spark-shell Spark的主要抽象是一个名为Dataset的分布式项目集合。可以从Hado...
spark:spark-shell启动出现文件找不到（file does not exist)
weixin_74181935的博客
04-16 706
需要注意的一个点就是mkdir 后面开始就是要根据自己的报错路径进行修改。出现上面的错误的解决办法就是创建一个路径。
Hadoop2.7.4 & Spark2.2.1 集群环境搭建指南
该资源提供了一种在三台服务器上构建Hadoop 2.7.4和Spark 2.2.1集群环境的方法，适用于学习环境搭建。在Linux系统中，使用JDK 1.8作为运行环境。文档涵盖了用户添加、文件权限设置以及Hadoop和Scala的安装步骤。 ...
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值