webmagic爬取https网站,报错javax.net.ssl.SSLHandshakeException: Received fatal alert: protocol_version

最新推荐文章于 2025-04-23 17:49:57 发布

彭大勃

最新推荐文章于 2025-04-23 17:49:57 发布

阅读量1k

点赞数 3

CC 4.0 BY-SA版权

分类专栏：经验分享文章标签： java https 经验分享数据挖掘

本文链接：https://blog.youkuaiyun.com/NullPointerExcept1on/article/details/108076326

经验分享专栏收录该内容

1 篇文章

订阅专栏

针对WebMagic在爬取仅支持TLS1.2的HTTPS站点时出现的SSLHandshakeException错误，本文提供了修改HttpClientDownloader以支持多种SSL/TLS版本的解决方案，通过自定义SSLConnectionSocketFactory实现，确保爬虫正常运行。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

目前webmagic最新版是0.7.3版本，在爬取只支持TLS1.2的https站点会报错,作者说会在0.7.4版本发布修复错误,但是等了3年了都没等到0.7.4版本的发布.

javax.net.ssl.SSLHandshakeException: Received fatal alert: protocol_version
	at sun.security.ssl.Alert.createSSLException(Alert.java:131)
	at sun.security.ssl.Alert.createSSLException(Alert.java:117)
	at sun.security.ssl.TransportContext.fatal(TransportContext.java:314)
	at sun.security.ssl.Alert$AlertConsumer.consume(Alert.java:293)
	at sun.security.ssl.TransportContext.dispatch(TransportContext.java:187)
	at sun.security.ssl.SSLTransport.decode(SSLTransport.java:154)
	at sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1198)
	at sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1107)
	at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:400)
	at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:372)
	at org.apache.http.conn.ssl.SSLConnectionSocketFactory.createLayeredSocket(SSLConnectionSocketFactory.java:436)
	at org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:384)
	at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142)
	at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:376)
	at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393)
	at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236)
	at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186)
	at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
	at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
	at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
	at us.codecraft.webmagic.downloader.HttpClientDownloader.download(HttpClientDownloader.java:85)
	at us.codecraft.webmagic.Spider.processRequest(Spider.java:404)
	at us.codecraft.webmagic.Spider.access$000(Spider.java:61)
	at us.codecraft.webmagic.Spider$1.run(Spider.java:320)
	at us.codecraft.webmagic.thread.CountableThreadPool$1.run(CountableThreadPool.java:74)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

目前的解决办法是：临时适配方式，修改HttpClientGenerator中的buildSSLConnectionSocketFactory方法，重写自己实现的HttpClientDownloader，并设置到Spider中。修改内容如下:

return new SSLConnectionSocketFactory(createIgnoreVerifySSL(), new String[]{"SSLv3", "TLSv1", "TLSv1.1", "TLSv1.2"},
                    null,
                    new DefaultHostnameVerifier())

目前我已经把源码修改好了,并编译为jar包,有被同样问题困扰的朋友可以直接下载使用.

webmagic-core-0.7.3.jar

提取码:webm

jar包使用方法:将下载好的 jar包,放在你的maven仓库的us\codecraft\webmagic-core\0.7.3路径下,替换原有jar包即可解决问题.