第一个坑：无法指定被请求的地址: Service 'sparkDriver' failed after 16 retries (starting from 0)!...

最新推荐文章于 2021-12-09 21:35:43 发布
最新推荐文章于 2021-12-09 21:35:43 发布
阅读量961 收藏 1
点赞数 1
CC 4.0 BY-SA版权
文章标签： 大数据 开发工具 java
原文链接：http://www.cnblogs.com/ChristianKula/p/9349055.html
本文分享了一个关于Spark应用程序在不同操作系统（Mac与Linux）间迁移时遇到的问题及解决方案。核心在于修改配置文件中的一项设置：添加conf.set('spark.driver.host','localhost')这一行代码，以确保在IntelliJ IDEA环境下程序可以正常运行。

摘要生成于 C知道 ，由 DeepSeek-R1 满血版支持， 前往体验 >

 这个代码是我从mac上编译通过的，但是换到了linux上，再用intellij idea跑，挂了。。。

问题很简单，记住就行了，加一句就好

conf.set("spark.driver.host", "localhost");（就是注释掉那段）

先记下来，我再找找这个有没有配置：

 

转载于:https://www.cnblogs.com/ChristianKula/p/9349055.html

云平台发放的虚拟机无法提交spark任务到大数据集群上（双IP）：Service ‘sparkDriver‘ failed after 16 retries...correct binding ...
lukabruce的博客
05-28 579
1 现象描述 部分异常： Service 'sparkDriver' failed after 16 retries.......correct binding address. 云平台发放的虚拟机包含两个IP，包含虚拟机内部实际的网络地址（ifconfig查看到的内网IP：192.168.xx.xx）和对方访问的IP（绑定的外网虚拟IP：11.11.xx.xx）,想要在云平台外访问虚拟机需要使用外网IP经过一次网络地址转换，直接使用内部IP是无法访问的。 所以在当任务提交的时候实际使用的是内部
本地Idea运行Spark任务错误：Cannot assign requested address: bind: Service 'sparkDriver'
FYDragonFYL的博客
12-17 3178
18/12/17 10:37:20 ERROR SparkContext: Error initializing SparkContext. java.net.BindException: Cannot assign requested address: bind: Service 'sparkDriver' failed after 16 retries (starting from 0)! ...
java.net.BindException: 无法指定被请求的地址: Service 'sparkDriver' failed after 16 retries!
走向程序的康庄大道
07-21 7923
在Centos7 中 使用 Intellij IDEA 来 Spark 本地模式启动时，报错： java.net.BindException: 无法指定被请求的地址: Service 'sparkDriver' failed after 16 retries! 16/06/27 19:36:34 WARN Utils: Service ‘sparkDriver’ could not bind...
pyspark spark-shell无法指定被请求的地址: Service 'sparkDriver' failed after 16 retries (on a random free port)
haojiliang
12-23 4173
新地址：https://blog.iaiot.com/pyspark-spark-shell-sparkDriver-failed.html 原文：https://stackoverflow.com/questions/49654050/could-not-bind-on-a-random-free-port-error-while-trying-to-connect-to-spark-mast...
spark 配置问题 无法指定被请求的地址: Service ‘sparkDriver‘ failed after 16 retries (starting from 0)
程序员的博客
04-01 6970
问题描述 今天在回顾spark配置的时候，按照课本进行，测试spark 运行，但是在运行 bin/run-example SparkPi 的时候出现以下问题 Exception in thread "main" java.net.BindException: 无法指定被请求的地址: Service 'sparkDriver' failed after 16 retries (starting from 0)! Consider explicitly setting the appropriate p
ERROR SparkContext:无法指定请求的地址: bind: Service 'sparkDriver' failed after 16 retries!
aweiy0515的博客
10-25 3842
18/10/25 13:07:58 WARN Utils: Service 'sparkDriver' could not bind on port 0. Attempting port 1. 18/10/25 13:07:58 WARN Utils: Service 'sparkDriver' could not bind on port 0. Attempting port 1. 18/10...
Exception in thread "main" java.net.BindException: 无法指定被请求的地址: Service 'sparkMaster' failed 错误处理
ToChangetheWorld的博客
05-13 3702
1、搭建 Spark 的HA环境时报如下错误Spark Command: /root/training/jdk1.8.0_144/bin/java -cp /root/training/spark-2.1.0-bin-hadoop2.7/conf/:/root/training/spark-2.1.0-bin-hadoop2.7/jars/* -Xmx1g org.apache.spark.dep...
启动spark下的master异常报错， Service ‘sparkMaster‘ failed after 16 retries (starting from 7077)绑定7077端口失败问题
weixin_42098250的博客
12-09 2832
启动spark下的master异常报错， Service 'sparkMaster' failed after 16 retries (starting from 7077)!绑定7077端口失败问题
【error】SparkUI端口被占用
drl_blogs的博客
07-02 1885
ERROR ui.SparkUI: Failed to bind SparkUI java.net.BindException: Address already in use: bind: Service 'SparkUI' failed after 16 retries (starting from 4040)! Consider explicitly setting the appropri...
Scala学习之 java.net.BindException: 无法指定被请求的地址: Service 'sparkDriver' failed
qq_38603049的博客
11-05 1238
在Scala本地运行时，出现无法指定被请求地址，通过朋友帮助终于解决问题` java.net.BindException: 无法指定被请求的地址: Service ‘sparkDriver’ failed after 16 retries! 19/011/5 20:10:34 WARN Utils: Service ‘sparkDriver’ could not bind on port 0. A...
Cannot assign requested address: Service 'sparkMaster' failed after 16 retries (starting from 7077)!
chenxb1986的博客
11-20 5759
spark-2.1.1-bin-hadoop2.7安装完成在启动时出现：“ Exception in thread “main” java.net.BindException: Cannot assign requested address: Service ‘sparkMaster’ failed after 16 retries (starting from 7077)! Consider e...
Can't assign requested address: Service 'sparkDriver' failed（pyspark启动失败）
bjjoy2009的博客
12-12 4342
bug描述 py4j.protocol.Py4JJavaError: An error occurred while calling None.org.apache.spark.api.java.JavaSparkContext. java.net.BindException: Can’t assign requested address: Service ‘sparkDriver’ faile...
spark启动时Cannot assign requested address: Service 'sparkDriver'问题分析
weixin_43267910的博客
12-24 5526
问题经过 当使用bin/spark-shell命令或使用spark-submit提交任务时，会遇到如下报错。 Setting default log level to “WARN”. To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel). 19/12/24 09:58...
java.net.BindException: 无法指定被请求的地址: Service ‘sparkDriver‘ failed after 16 retries (on a random free
OhGiraffe的博客
05-11 3027
启动spark-shell的时候出现的错误： **原因：**该虚拟机是其他虚拟机的副本，复制过来的时候没有修改etc/hosts的地址，导致主机地址与etc/hosts地址不一致。 方法： 查看本机地址命令：ifconfig 修改etc/hosts下的地址：sudo vi /etc/hosts 改完重启shell即可 ...
Spark Yarn Client模式spark.driver.host导致 Cannot assign requested address: Service ‘sparkDriver‘ failed
九师兄
08-29 2794
1.背景 [deploy@fideploy cheetah-node]$ /opt/cloudera/parcels/SPARK2/bin/spark2-submit --keytab /etc/security/keytabs/cluster_user.keytab --principal cluster_user@HADOOP.COM --class com.dtwave.cheetah....
java.net.BindException: Cannot assign requested address: Service 'sparkDriver' failed
一杯派蒙的博客
05-02 4811
记一次Spark Local 模式下的错误：报错日记：[root@steve-lidiliang /user/coding/Conclusions/JavaBasic]# spark-shell 2018-05-02 13:51:20 WARN NativeCodeLoader:62 - Unable to load native-hadoop library for your platfor...
session.connectAndBind("localhost", 2775, bindParam);返回Test client error: Connection refused: connect
最新发布
08-07
<think>问题分析： 测试客户端连接被拒绝，通常意味着服务器没有在指定的端口上监听，或者防火墙阻止了连接。 在我们的场景中，可能是由于SMPP服务器启动失败或启动延迟导致客户端在服务器准备好之前就尝试连接。 解决方案： 1. 确保服务器成功启动并监听端口。 2. 在客户端连接之前，确保服务器已经完成启动并开始接受连接。 3. 在测试客户端中添加重试机制。 我们将修改测试客户端，使其在连接被拒绝时重试多次，并增加适当的延迟。 修改 `SmppServerApplication.java` 中的 `sendTestMessage` 方法，加入重试逻辑。 同时，确保服务器启动后，等待几秒钟再启动测试客户端（通过调整初始延迟）。 另外，检查服务器是否成功绑定到端口，并打印绑定成功的消息。 具体修改如下：</think>您的测试客户端遇到 "Connection refused: connect" 错误，这通常表示客户端无法连接到服务器。以下是完整的解决方案和调试步骤： ### 问题诊断与解决方案 ```java // src/main/java/com/example/smppserver/SmppServerApplication.java package com.example.smppserver; import org.jsmpp.bean.*; import org.jsmpp.session.BindParameter; import org.jsmpp.session.BindType; import org.jsmpp.session.SMPPSession; import org.springframework.boot.SpringApplication; import org.springframework.boot.autoconfigure.SpringBootApplication; import org.springframework.context.ConfigurableApplicationContext; import org.springframework.context.annotation.Bean; import org.springframework.scheduling.annotation.EnableScheduling; import org.springframework.scheduling.annotation.Scheduled; import javax.annotation.PostConstruct; import java.util.concurrent.ExecutorService; import java.util.concurrent.Executors; import java.util.concurrent.TimeUnit; import java.util.concurrent.atomic.AtomicInteger; import java.util.concurrent.atomic.AtomicLong; @SpringBootApplication @EnableScheduling public class SmppServerApplication { private static final AtomicLong messageCount = new AtomicLong(0); private static final AtomicInteger serverReady = new AtomicInteger(0); public static void main(String[] args) { ConfigurableApplicationContext context = SpringApplication.run(SmppServerApplication.class, args); } @Bean public AtomicLong messageCounter() { return messageCount; } @Bean public AtomicInteger serverReadySignal() { return serverReady; } @PostConstruct public void init() { System.out.println("Starting test client in 5 seconds..."); ExecutorService executor = Executors.newSingleThreadExecutor(); executor.execute(this::delayedClientStart); executor.shutdown(); } private void delayedClientStart() { try { // 等待服务器启动信号 while (serverReady.get() == 0) { System.out.println("Waiting for server to be ready..."); Thread.sleep(1000); } Thread.sleep(2000); // 额外等待2秒确保服务器完全就绪 System.out.println("Server ready signal received. Starting test client..."); sendTestMessage(); } catch (Exception e) { e.printStackTrace(); } } // 增强版测试客户端 private void sendTestMessage() { System.out.println("Attempting to send test SMS..."); BindParameter bindParam = new BindParameter( BindType.BIND_TX, "systemid", "password", "cp", TypeOfNumber.INTERNATIONAL, NumberingPlanIndicator.ISDN, null ); SMPPSession session = null; int retryCount = 0; final int MAX_RETRIES = 5; final int RETRY_DELAY = 2000; // 2秒重试间隔 while (retryCount < MAX_RETRIES) { try { session = new SMPPSession(); System.out.println("Connecting to SMPP server... attempt " + (retryCount + 1)); // 设置连接超时为5秒 session.setConnectTimeout(5000); session.connectAndBind("localhost", 2775, bindParam); SubmitSmResult result = session.submitShortMessage( "CMT", TypeOfNumber.INTERNATIONAL, NumberingPlanIndicator.ISDN, "1613", TypeOfNumber.INTERNATIONAL, NumberingPlanIndicator.ISDN, "123456789", new ESMClass(), (byte)0, (byte)1, null, null, new RegisteredDelivery(SMSCDeliveryReceipt.SUCCESS_FAILURE), (byte)0, new GeneralDataCoding(Alphabet.ALPHA_UCS2), (byte)0, "测试消息: Hello from SMPP client!".getBytes("UTF-16BE") ); System.out.println("Test SMS sent, message ID: " + result.getMessageId()); messageCount.incrementAndGet(); return; // 成功发送后退出 } catch (Exception e) { System.err.println("Test client error (attempt " + (retryCount + 1) + "): " + e.getMessage()); // 特定错误处理 if (e.getMessage().contains("Connection refused")) { System.out.println("Server may not be ready yet. Retrying..."); } else if (e.getMessage().contains("timeout")) { System.out.println("Connection timeout. Retrying..."); } retryCount++; // 关闭会话（如果已创建） if (session != null) { try { if (session.getSessionState().isBound()) { session.unbind(); } session.close(); } catch (Exception ex) { System.err.println("Error closing session: " + ex.getMessage()); } session = null; } // 重试前等待 try { Thread.sleep(RETRY_DELAY); } catch (InterruptedException ie) { Thread.currentThread().interrupt(); } } } System.err.println("Failed to send test message after " + MAX_RETRIES + " attempts"); } @Scheduled(fixedRate = 10000) public void reportStats() { long count = messageCount.get(); System.out.println("Current message rate: " + count + " messages/10s"); messageCount.set(0); } } ``` ### 服务器端修改（添加就绪信号） ```java // src/main/java/com/example/smppserver/SmppServer.java // 在SmppServer类中添加以下方法 @Component public class SmppServer implements Runnable { // 添加对就绪信号的引用 private final AtomicInteger serverReady; @Autowired public SmppServer(AtomicInteger serverReadySignal) { this.serverReady = serverReadySignal; } @Override public void run() { try { serverSocket = new ServerSocket(port); System.out.println("SMPP Server started on port " + port); // 设置服务器就绪信号 serverReady.set(1); System.out.println("Server is now ready to accept connections"); while (running) { Socket socket = serverSocket.accept(); System.out.println("New client connected: " + socket.getInetAddress()); executor.execute(new SessionHandler(socket)); } } catch (IOException e) { if (running) { System.err.println("Server error: " + e.getMessage()); } } } // ... 其余代码保持不变 ... } ``` ### 调试步骤和常见问题解决方案 1. **端口冲突检查**： ```java // 在SmppServer中添加端口检查 private boolean isPortAvailable(int port) { try (ServerSocket socket = new ServerSocket(port)) { return true; } catch (IOException e) { return false; } } @Override public void run() { if (!isPortAvailable(port)) { System.err.println("Port " + port + " is already in use!"); return; } // ... 其余代码 ... } ``` 2. **防火墙检查**： ```bash # Linux/Mac sudo lsof -i :2775 sudo netstat -tuln | grep 2775 # Windows netstat -ano | findstr :2775 ``` 3. **绑定地址问题**： ```java // 显式绑定到所有接口 serverSocket = new ServerSocket(port, 50, InetAddress.getByName("0.0.0.0")); ``` 4. **详细日志记录**： ```java // 在SmppServer中添加详细日志 System.out.println("Server socket created: " + serverSocket); System.out.println("Local address: " + serverSocket.getInetAddress()); System.out.println("Local port: " + serverSocket.getLocalPort()); ``` 5. **网络连通性测试**： ```java // 添加简单的echo服务器测试 private void testConnectivity() { try (ServerSocket testSocket = new ServerSocket(2776); Socket client = testSocket.accept(); PrintWriter out = new PrintWriter(client.getOutputStream(), true); BufferedReader in = new BufferedReader(new InputStreamReader(client.getInputStream()))) { System.out.println("Echo server running on port 2776"); String inputLine; while ((inputLine = in.readLine()) != null) { out.println(inputLine); } } catch (IOException e) { System.err.println("Echo server error: " + e.getMessage()); } } ``` ### 完整的调试流程 1. **启动应用程序**： ```bash mvn spring-boot:run ``` 2. **预期输出顺序**： ``` SMPP Server started on port 2775 Server is now ready to accept connections Waiting for server to be ready... Server ready signal received. Starting test client... Attempting to send test SMS... Connecting to SMPP server... attempt 1 New client connected: /127.0.0.1 Client bound: systemid Test SMS sent, message ID: ID1684567890123 ===== Received SMS ===== From: 1613 To: 123456789 Text: 测试消息: Hello from SMPP client! Encoding: 8 Session: 1 ======================== ``` 3. **如果仍然失败**： ```java // 在SmppServerApplication中添加备用测试方法 @Scheduled(initialDelay = 15000, fixedRate = 30000) public void healthCheck() { try (Socket socket = new Socket("localhost", 2775)) { System.out.println("Health check: Successfully connected to SMPP port"); } catch (IOException e) { System.err.println("Health check FAILED: " + e.getMessage()); e.printStackTrace(); // 尝试重启服务器组件 System.out.println("Attempting to restart server component..."); // 这里添加重启逻辑 } } ``` ### 常见问题总结表 | 问题 | 原因 | 解决方案 | |------|------|----------| | Connection refused | 服务器未启动 | 添加启动就绪信号 | | 端口被占用 | 其他进程使用相同端口 | 添加端口检查逻辑 | | 绑定到错误接口 | 默认绑定到127.0.0.1 | 使用0.0.0.0绑定所有接口 | | 防火墙阻止 | 系统防火墙阻止连接 | 添加防火墙规则或检查配置 | | 启动顺序问题 | 客户端在服务器前启动 | 添加就绪信号和重试机制 | | DNS解析问题 | localhost解析失败 | 使用127.0.0.1替代localhost | ### 最终解决方案包结构 ``` src/main/java/com/example/smppserver/ ├── SmppServerApplication.java # 主应用，含增强测试客户端 ├── SmppServer.java # SMPP服务器实现 ├── MessageUtil.java # 消息解码工具 └── config/ └── SmppConfig.java # 配置类（可选） ``` **SmppConfig.java**（可选配置类）： ```java package com.example.smppserver.config; import org.springframework.context.annotation.Bean; import org.springframework.context.annotation.Configuration; import java.util.concurrent.atomic.AtomicInteger; import java.util.concurrent.atomic.AtomicLong; @Configuration public class SmppConfig { @Bean public AtomicLong messageCounter() { return new AtomicLong(0); } @Bean public AtomicInteger serverReadySignal() { return new AtomicInteger(0); } } ```
评论
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值