spark连接问题

18/04/19 23:27:58 INFO master.Master: Registering worker 192.168.72.143:41196 with 4 cores, 2.7 GB RAM
18/04/19 23:28:00 INFO master.Master: Registering worker 192.168.72.141:38108 with 4 cores, 2.7 GB RAM
18/04/19 23:28:06 INFO master.Master: Registering worker 192.168.72.141:38108 with 4 cores, 2.7 GB RAM
18/04/19 23:28:06 INFO master.Master: Registering worker 192.168.72.143:41196 with 4 cores, 2.7 GB RAM
18/04/19 23:28:11 INFO master.Master: Registering worker 192.168.72.142:42803 with 4 cores, 2.7 GB RAM
18/04/19 23:28:12 INFO master.Master: Registering worker 192.168.72.141:38108 with 4 cores, 2.7 GB RAM
18/04/19 23:28:15 INFO master.ZooKeeperLeaderElectionAgent: We have gained leadership
18/04/19 23:28:15 INFO master.Master: I have been elected leader! New state: RECOVERING
18/04/19 23:28:15 INFO master.Master: Trying to recover worker: worker-20180419223807-192.168.72.141-43776
18/04/19 23:28:15 INFO master.Master: Trying to recover worker: worker-20180419223753-192.168.72.143-38431
18/04/19 23:28:15 WARN netty.OneWayOutboxMessage: Failed to send one-way RPC.
java.io.IOException: Failed to connect to /192.168.72.143:38431
at org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:216)
at org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:167)
at org.apache.spark.rpc.netty.NettyRpcEnv.createClient(NettyRpcEnv.scala:200)
at org.apache.spark.rpc.netty.Outbox$$anon$1.call(Outbox.scala:187)
at org.apache.spark.rpc.netty.Outbox$$anon$1.call(Outbox.scala:183)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.net.ConnectException: 拒绝连接: /192.168.72.143:38431
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:224)
at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:289)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:528)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)
at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
... 1 more
18/04/19 23:28:15 WARN netty.OneWayOutboxMessage: Failed to send one-way RPC.
java.io.IOException: Failed to connect to /192.168.72.141:43776
at org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:216)
at org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:167)
at org.apache.spark.rpc.netty.NettyRpcEnv.createClient(NettyRpcEnv.scala:200)
at org.apache.spark.rpc.netty.Outbox$$anon$1.call(Outbox.scala:187)
at org.apache.spark.rpc.netty.Outbox$$anon$1.call(Outbox.scala:183)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.net.ConnectException: 拒绝连接: /192.168.72.141:43776
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:224)
at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:289)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:528)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)
at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111)
... 1 more
18/04/19 23:28:15 INFO master.Master: Registering worker 192.168.72.143:41196 with 4 cores, 2.7 GB RAM
18/04/19 23:28:18 INFO master.Master: Registering worker 192.168.72.141:38108 with 4 cores, 2.7 GB RAM
18/04/19 23:28:25 INFO master.Master: Registering worker 192.168.72.142:42803 with 4 cores, 2.7 GB RAM
18/04/19 23:29:15 INFO master.Master: Removing worker worker-20180419223753-192.168.72.143-38431 on 192.168.72.143:38431
18/04/19 23:29:15 INFO master.Master: Removing worker worker-20180419223807-192.168.72.141-43776 on 192.168.72.141:43776
18/04/19 23:29:15 INFO master.Master: Recovery complete - resuming operations!
### 如何使用 Spark 连接 MySQL 数据库 #### 创建 SparkSession 并配置连接属性 为了使 Spark 应用程序能够与 MySQL 数据库通信,需要创建 `SparkSession` 实例并配置必要的参数: ```scala import org.apache.spark.sql.SparkSession val spark = SparkSession.builder() .appName("Spark MySQL Example") .config("spark.master", "local") .config("spark.sql.warehouse.dir", "/user/hive/warehouse") .config("spark.sql.catalogImplementation", "hive") .getOrCreate() // 设置 MySQL 驱动加载 Class.forName("com.mysql.jdbc.Driver").newInstance()[^3] // 定义连接属性 val connectionProperties = new java.util.Properties() connectionProperties.put("user", "your_username") connectionProperties.put("password", "your_password")[^2] ``` #### 构建 JDBC URL 和读取数据 构建用于描述目标数据库位置的 JDBC URL 字符串,并利用它来执行 SQL 查询从而获取所需的数据集。 ```scala val jdbcUrl = "jdbc:mysql://localhost:3306/database_name" // 使用 jdbc 方法从 MySQL 中读取 'bank' 表的数据至 DataFrame val dfBank = spark.read.jdbc(jdbcUrl, "bank", connectionProperties) dfBank.show() // 展示部分记录 ``` #### 处理和分析数据 一旦获得了来自 MySQL 的数据帧对象之后就可以对其进行各种转换操作了。比如下面这段代码实现了基于职业字段(`job`)对员工数量进行计数的功能: ```scala val jobCountsDF = dfBank.groupBy("job").count().orderBy($"count".desc) jobCountsDF.show(10) // 只展示前十项结果 ``` #### 关闭 SparkSession 当所有的处理工作完成后应当记得终止会话以释放占用着的各种资源。 ```scala spark.stop() ```
评论 8
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值