我自己写了一个爬虫程序,跑了半天后程序就卡死了,没有任何输出和动静。
先是使用jstatd
和VisualVM
,参考这篇文章:jstatd,VisualVM使用和报错解决
结果如下图:
看到内存毫无变化,应该是发生了死锁。
使用命令jstack
查看线程堆栈状态:
-> % jstack 26924
2019-12-23 09:53:38
Full thread dump OpenJDK 64-Bit Server VM (11.0.4+11-post-Ubuntu-1ubuntu218.04.3 mixed mode, sharing):
Threads class SMR info:
_java_thread_list=0x00007f35dc0035b0, length=20, elements={
0x00007f360c011000, 0x00007f360c0b0800, 0x00007f360c0b2800, 0x00007f360c0b8800,
0x00007f360c0ba800, 0x00007f360c0bd000, 0x00007f360c0bf000, 0x00007f360c0f9000,
0x00007f360c0fe800, 0x00007f360c20f800, 0x00007f360c249000, 0x00007f360c378800,
0x00007f360c38b000, 0x00007f360c389800, 0x00007f360c390000, 0x00007f360c391800,
0x00007f360c393000, 0x00007f360c395800, 0x00007f360c38e000, 0x00007f35dc001800
}
"main" #1 prio=5 os_prio=0 cpu=697257.22ms elapsed=65383.05s tid=0x00007f360c011000 nid=0x692d waiting on condition [0x00007f3612f97000]
java.lang.Thread.State: WAITING (parking)
at jdk.internal.misc.Unsafe.park(java.base@11.0.4/Native Method)
- parking to wait for <0x00000000e3c00020> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(java.base@11.0.4/LockSupport.java:194)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@11.0.4/AbstractQueuedSynchronizer.java:2081)
at org.apache.http.pool.AbstractConnPool.getPoolEntryBlocking(AbstractConnPool.java:379)
at org.apache.http.pool.AbstractConnPool.access$200(AbstractConnPool.java:69)
at org.apache.http.pool.AbstractConnPool$2.get(AbstractConnPool.java:245)
- locked <0x00000000e3c000b0> (a org.apache.http.pool.AbstractConnPool$2)
at org.apache.http.pool.AbstractConnPool$2.get(AbstractConnPool.java:193)
at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.leaseConnection(PoolingHttpClientConnectionManager.java:304)
at org.apache.http.impl.conn.PoolingHttpClientConnectionManager$1.get(PoolingHttpClientConnectionManager.java:280)
at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:190)
at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186)
at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:108)
at com.zhang.spider.getHtml(spider.java:219)
at com.zhang.spider.main(spider.java:145)
"pool-1-thread-1" #13 prio=5 os_prio=0 cpu=68909.14ms elapsed=65380.72s tid=0x00007f360c378800 nid=0x693f waiting on condition [0x00007f35f0c38000]
java.lang.Thread.State: WAITING (parking)
at jdk.internal.misc.Unsafe.park(java.base@11.0.4/Native Method)
- parking to wait for <0x00000000ecfae070> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(java.base@11.0.4/LockSupport.java:194)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@11.0.4/AbstractQueuedSynchronizer.java:2081)
at java.util.concurrent.LinkedBlockingQueue.take(java.base@11.0.4/LinkedBlockingQueue.java:433)
at java.util.concurrent.ThreadPoolExecutor.getTask(java.base@11.0.4/ThreadPoolExecutor.java:1054)
at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.4/ThreadPoolExecutor.java:1114)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.4/ThreadPoolExecutor.java:628)
at java.lang.Thread.run(java.base@11.0.4/Thread.java:834)
JNI global refs: 17, weak refs: 0
看到线程池中的线程状态是java.lang.Thread.State: WAITING (parking)
和parking to wait for <0x00000000ecfae070> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
,这是正常的线程等待状态。
发现主线程有问题,虽然同样是java.lang.Thread.State: WAITING (parking)
,
但在等锁parking to wait for <0x00000000e3c00020> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
。
发现问题是由org.apache.http.impl.client.CloseableHttpClient
造成的。
然后看到了这篇文章:记一次 HttpClient 死锁问题 | Mr.Kail’s Blog
一个解决办法是升级库的版本:
<dependency>
<groupId>org.apache.httpcomponents</groupId>
<artifactId>httpclient</artifactId>
<version>4.5.10</version>
</dependency>
另一个办法是记得关闭连接:
try (CloseableHttpClient httpclient = HttpClients.createDefault()) {
CloseableHttpResponse response = httpclient.execute(url);
response.close()