记一次tomcat CPU利用率高达380%+的问题排查思路与处理过程

夜雨胭脂凝夜紫

已于 2024-11-25 15:26:13 修改

阅读量1.6k

点赞数 9

分类专栏：运维文章标签： linux 运维服务器 java tomcat

于 2024-11-25 15:19:03 首次发布

本文链接：https://blog.youkuaiyun.com/yeyuningzi/article/details/143595738

版权

运维专栏收录该内容

1 篇文章

订阅专栏

4.1 top -H -p PID 查看线程情况

4.2 jstack PID 生成线程快照

4.3查看线程快照文件，查看其线程情况并分析线程状态

4.4线程状态基础知识

4.5 结合日志分析

查看正在运行的sql语句

1、前言

这个问题困扰了我很久很久，最直观的感受就是：服务器上的应用无法正常使用了，好在是内部演示使用的应用，并不是大规模应用的存在，否则我估计嘿嘿。。。

下定决心，想要找出具体问题原因，想办法搞定他！！！

2、问题现象

服务器资源描述：这是一个4核CPU 的服务器通过lscpu命令可查看 cpu数量：

cat /etc/os-release 查看系统详情： Debian 12

top 查看CPU占用率，直接给我干到了 387% 相当于每个CPU 占用到了 96%+ 了我的天啊！

他到底干了什么啊！！！

由于我是在同一个服务器上部署了两个tomcat ，一个tomcat部署了A应用，由于B应用与A应用有冲突，所以此前同事喊我分开部署，于是就成了在tomcatA 里的A应用，需要调用在Tomcat B里的B。奈何不遂人愿，经常性的CPU飙升。

为了图方便，此前我也直接会直接kill -9 PID 干掉这个进程，但是会发现：屁用没有，他还是会给你报错，应用日志会说：

org.apache.http.conn.HttpHostConnectException: Connect to 127.0.0.1:8081 [/127.0.0.1] failed: 拒绝连接 (Connection refused)
。。。。

Caused by: java.net.ConnectException: 拒绝连接 (Connection refused)
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:476)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:218)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:200)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:394)
at java.net.Socket.connect(Socket.java:606)
at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:74)
at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:134)

啥玩意就跟我说拒绝连接啊。。。。。

只能把两个tomcat 同时重启，才能撑一段时间，一段时间之后又双叕爆了，应用又不行了.....

几乎每每这是这个tomcat上对应的B应用的日志就会提示： java.lang.OutOfMemoryError: Java heap space

java.lang.OutOfMemoryError: Java heap space

呵呵呵呵，麻了...... 同事喊我设定tomcat自动重启。。。。然而在很久之前我试图配置了自动重启，然而似乎配置的有问题，然后就没有然后了。。。。。

一直报错，一直重启，又不知道他什么时候报错，没找到规律。。。醉人

3、排查思路

一直在报错，一直没有找到很好的解决办法，直到看到： Tomcat进程占用CPU过高怎么办？_tomcat cpu占用率暴涨-优快云博客

思路总结：在出现CPU占用率高时，

1、通过命令top -H -p PID进程号查看该进程中各线程的情况，看是哪个线程占用cpu比较高，如threadA；

2、通过命令jstack PID 生成线程快照，生成文件如：jstack PID>PID.log ；

3、查看线程快照文件PID.log，找到threadA，查看其线程栈；

4、结合线程状态分析具体情况

大致就是这样一个思路。知道思路之后，等到下次问题重现时，就可以尝试根据这个思路去找问题了。

4、排查过程记录

4.1 top -H -p PID 查看线程情况

top -H -p 进程ID -P - 按照CPU利用率从高到低排序， -H 所有线程 ;即该命令意在查看指定进程ID的线程情况，这些线程按照CPU利用率从高到低排序。

终于被我抓到了：

top -H -p PID：

可以看到以下几个线程

进程号 USER PR NI VIRT RES SHR %CPU %MEM TIME+ COMMAND
395365 wl 20 0 9033908 3.9g 22488 S 99.9 25.3 5d+23h java
395368 wl 20 0 9033908 3.9g 22488 R 94.0 25.3 35,15 java
395370 wl 20 0 9033908 3.9g 22488 R 94.0 25.3 35,15 java
395369 wl 20 0 9033908 3.9g 22488 R 90.7 25.3 35,15 java
395367 wl 20 0 9033908 3.9g 22488 R 90.0 25.3 35,15 java

4.2 jstack PID 生成线程快照

这里我遇到了这样的问题：

Unable to open socket file: target process not responding or HotSpot VM not loaded
The -F option can be used when the target process is not responding
这里需要做个很重要的事情：如果是 jstack 直接使用需要跟进程号；而如果是跟线程号则是要跟线程号的十六进制

jstack 进程号：

在linux环境下，有一个简单的方法将十进制数据直接转成十六进制数据，即使用命令： printf "%x\n" data 转换即可：

转换后的数据真正要使用时需要加上0x的前缀，表示其是十六进制数据，不然就会出现如下图所示的：错误的文件描述符的问题

应是：

XX@debian12:~$ printf "%x\n" 395369
60869
XX@debian12:~$ jstack 395365 |grep -A 50 0x60869>tid395369_60869.txt

jstack 395365 |grep -A 50 0x60869>tid395369_60869.txt 表示：用jstack工具查询进程号为 395365的进程，并输入线程号为395369（十六进制：0x60869）线程的后50行堆栈并写入当前目录下的xxx.txt 文件。

将这些占用cpu较高的线程全部进行快照文件，然后再针对分析。

4.3查看线程快照文件，查看其线程情况并分析线程状态

打开抓取到的jstack日志，根据日志的数据去分析线程情况。

由于我这个问题分了好几次查看，所以后来我又重新抓取了。步骤重复上述过程，得到：

抓取出来的进程日志如下：

2024-11-21 11:29:53
Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.281-b09 mixed mode):

"Attach Listener" #58989 daemon prio=9 os_prio=0 tid=0x00007f46f4006800 nid=0xd68b8 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"HikariPool-2 connection closer" #58988 daemon prio=5 os_prio=0 tid=0x00005612835a1800 nid=0xd68a8 runnable [0x00007f471b6fe000]
   java.lang.Thread.State: RUNNABLE
	at com.mysql.jdbc.ConnectionImpl.setNetworkTimeout(ConnectionImpl.java:5465)
	- locked <0x000000075abdb1c0> (a com.mysql.jdbc.JDBC4Connection)
	at com.zaxxer.hikari.pool.PoolBase.setNetworkTimeout(PoolBase.java:566)
	at com.zaxxer.hikari.pool.PoolBase.quietlyCloseConnection(PoolBase.java:137)
	at com.zaxxer.hikari.pool.HikariPool.lambda$closeConnection$1(HikariPool.java:451)
	at com.zaxxer.hikari.pool.HikariPool$$Lambda$49/1082899297.run(Unknown Source)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

"HikariPool-2 connection adder" #57279 daemon prio=5 os_prio=0 tid=0x000056128359d000 nid=0xd5432 runnable [0x00007f4719bfd000]
   java.lang.Thread.State: RUNNABLE
	at java.net.SocketOutputStream.socketWrite0(Native Method)
	at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:111)
	at java.net.SocketOutputStream.write(SocketOutputStream.java:155)
	at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
	at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
	- locked <0x000000075ace2230> (a java.io.BufferedOutputStream)
	at com.mysql.jdbc.MysqlIO.send(MysqlIO.java:3688)
	at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2506)
	at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2675)
	at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2465)
	- locked <0x000000075acdb4c0> (a com.mysql.jdbc.JDBC4Connection)
	at com.mysql.jdbc.ConnectionImpl.configureClientCharacterSet(ConnectionImpl.java:1716)
	at com.mysql.jdbc.ConnectionImpl.initializePropsFromServer(ConnectionImpl.java:3286)
	at com.mysql.jdbc.ConnectionImpl.connectOneTryOnly(ConnectionImpl.java:2233)
	at com.mysql.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:2015)
	- locked <0x000000075acdb4c0> (a com.mysql.jdbc.JDBC4Connection)
	at com.mysql.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:768)
	at com.mysql.jdbc.JDBC4Connection.<init>(JDBC4Connection.java:47)
	at sun.reflect.GeneratedConstructorAccessor133.newInstance(Unknown Source)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
	at com.mysql.jdbc.Util.handleNewInstance(Util.java:403)
	at com.mysql.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:385)
	at com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:323)
	at com.zaxxer.hikari.util.DriverDataSource.getConnection(DriverDataSource.java:138)
	at com.zaxxer.hikari.pool.PoolBase.newConnection(PoolBase.java:364)
	at com.zaxxer.hikari.pool.PoolBase.newPoolEntry(PoolBase.java:206)
	at com.zaxxer.hikari.pool.HikariPool.createPoolEntry(HikariPool.java:476)
	at com.zaxxer.hikari.pool.HikariPool.access$100(HikariPool.java:71)
	at com.zaxxer.hikari.pool.HikariPool$PoolEntryCreator.call(HikariPool.java:726)
	at com.zaxxer.hikari.pool.HikariPool$PoolEntryCreator.call(HikariPool.java:712)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

"HikariPool-1 connection adder" #56415 daemon prio=5 os_prio=0 tid=0x00007f46bc008800 nid=0xd4a67 runnable [0x00007f47193fe000]
   java.lang.Thread.State: RUNNABLE
	at java.lang.String.substring(String.java:1969)
	at java.util.StringTokenizer.nextToken(StringTokenizer.java:352)
	at com.mysql.jdbc.NonRegisteringDriver.parseURL(NonRegisteringDriver.java:621)
	at com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:314)
	at com.zaxxer.hikari.util.DriverDataSource.getConnection(DriverDataSource.java:138)
	at com.zaxxer.hikari.pool.PoolBase.newConnection(PoolBase.java:364)
	at com.zaxxer.hikari.pool.PoolBase.newPoolEntry(PoolBase.java:206)
	at com.zaxxer.hikari.pool.HikariPool.createPoolEntry(HikariPool.java:476)
	at com.zaxxer.hikari.pool.HikariPool.access$100(HikariPool.java:71)
	at com.zaxxer.hikari.pool.HikariPool$PoolEntryCreator.call(HikariPool.java:726)
	at com.zaxxer.hikari.pool.HikariPool$PoolEntryCreator.call(HikariPool.java:712)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

"Java2D Disposer" #95 daemon prio=10 os_prio=0 tid=0x00007f46dca4b000 nid=0xbcc9c in Object.wait() [0x00007f471b7fe000]
   java.lang.Thread.State: WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:144)
	- locked <0x000000068f843ba0> (a java.lang.ref.ReferenceQueue$Lock)
	at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:165)
	at sun.java2d.Disposer.run(Disposer.java:148)
	at java.lang.Thread.run(Thread.java:748)

"http-nio-8081-Acceptor" #59 daemon prio=5 os_prio=0 tid=0x00007f47408fd000 nid=0xbcc14 runnable [0x00007f471b8fe000]
   java.lang.Thread.State: RUNNABLE
	at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
	at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:424)
	at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:252)
	- locked <0x000000068050bfb0> (a java.lang.Object)
	at org.apache.tomcat.util.net.NioEndpoint.serverSocketAccept(NioEndpoint.java:469)
	at org.apache.tomcat.util.net.NioEndpoint.serverSocketAccept(NioEndpoint.java:71)
	at org.apache.tomcat.util.net.Acceptor.run(Acceptor.java:106)
	at java.lang.Thread.run(Thread.java:748)

"http-nio-8081-ClientPoller" #58 daemon prio=5 os_prio=0 tid=0x00007f47405a3800 nid=0xbcc13 runnable [0x00007f471b9fe000]
   java.lang.Thread.State: RUNNABLE
	at java.util.HashMap$KeySet.iterator(HashMap.java:917)
	at java.util.HashSet.iterator(HashSet.java:173)
	at java.util.Collections$UnmodifiableCollection$1.<init>(Collections.java:1041)
	at java.util.Collections$UnmodifiableCollection.iterator(Collections.java:1040)
	at org.apache.tomcat.util.net.NioEndpoint$Poller.timeout(NioEndpoint.java:939)
	at org.apache.tomcat.util.net.NioEndpoint$Poller.run(NioEndpoint.java:751)
	at java.lang.Thread.run(Thread.java:748)

"http-nio-8081-exec-10" #57 daemon prio=5 os_prio=0 tid=0x00007f47409bf000 nid=0xbcc12 waiting on condition [0x00007f471bafe000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x000000068e5120c0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
	at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:108)
	at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:33)
	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
	at java.lang.Thread.run(Thread.java:748)

"http-nio-8081-exec-9" #56 daemon prio=5 os_prio=0 tid=0x00007f47408bc000 nid=0xbcc11 waiting on condition [0x00007f471bbfe000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x000000068e5120c0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
	at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:108)
	at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:33)
	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
	at java.lang.Thread.run(Thread.java:748)

"http-nio-8081-exec-8" #55 daemon prio=5 os_prio=0 tid=0x00007f47400ca000 nid=0xbcc10 waiting on condition [0x00007f471bcfe000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x000000068e5120c0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
	at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:108)
	at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:33)
	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
	at java.lang.Thread.run(Thread.java:748)

"http-nio-8081-exec-7" #54 daemon prio=5 os_prio=0 tid=0x00007f47407b4000 nid=0xbcc0f waiting on condition [0x00007f47291fe000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x000000068e5120c0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
	at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:108)
	at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:33)
	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
	at java.lang.Thread.run(Thread.java:748)

"http-nio-8081-exec-6" #53 daemon prio=5 os_prio=0 tid=0x00007f4740887800 nid=0xbcc0e waiting on condition [0x00007f47292fe000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x000000068e5120c0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
	at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:108)
	at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:33)
	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
	at java.lang.Thread.run(Thread.java:748)

"http-nio-8081-exec-5" #52 daemon prio=5 os_prio=0 tid=0x00007f47409be800 nid=0xbcc0d waiting on condition [0x00007f47293fe000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x000000068e5120c0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
	at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:108)
	at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:33)
	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
	at java.lang.Thread.run(Thread.java:748)

"http-nio-8081-exec-4" #51 daemon prio=5 os_prio=0 tid=0x00007f474079f800 nid=0xbcc0c waiting on condition [0x00007f47294fe000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x000000068e5120c0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
	at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:108)
	at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:33)
	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
	at java.lang.Thread.run(Thread.java:748)

"http-nio-8081-exec-3" #50 daemon prio=5 os_prio=0 tid=0x00007f4740565800 nid=0xbcc0b waiting on condition [0x00007f47295fe000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x000000068e5120c0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
	at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:108)
	at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:33)
	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
	at java.lang.Thread.run(Thread.java:748)

"http-nio-8081-exec-2" #49 daemon prio=5 os_prio=0 tid=0x00007f47400c9000 nid=0xbcc0a waiting on condition [0x00007f47296fe000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x000000068e5120c0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
	at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:108)
	at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:33)
	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
	at java.lang.Thread.run(Thread.java:748)

"http-nio-8081-exec-1" #48 daemon prio=5 os_prio=0 tid=0x00007f47409b9800 nid=0xbcc09 waiting on condition [0x00007f47297fe000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x000000068e5120c0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
	at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:108)
	at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:33)
	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
	at java.lang.Thread.run(Thread.java:748)

"Catalina-utility-2" #47 prio=1 os_prio=0 tid=0x00007f46cc002800 nid=0xbcc08 runnable [0x00007f47201fd000]
   java.lang.Thread.State: RUNNABLE
	at org.apache.catalina.util.LifecycleBase.fireLifecycleEvent(LifecycleBase.java:121)
	at org.apache.catalina.core.StandardServer.access$000(StandardServer.java:71)
	at org.apache.catalina.core.StandardServer$2.run(StandardServer.java:960)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
	at java.lang.Thread.run(Thread.java:748)

"Catalina-utility-1" #46 prio=1 os_prio=0 tid=0x00007f47405b4800 nid=0xbcc07 waiting on condition [0x00007f471bffe000]
   java.lang.Thread.State: TIMED_WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x0000000682755380> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
	at java.lang.Thread.run(Thread.java:748)

"pool-8-thread-1" #45 prio=5 os_prio=0 tid=0x00007f46c0007800 nid=0xbcc06 runnable [0x00007f47203fd000]
   java.lang.Thread.State: RUNNABLE
	at java.lang.Throwable.toString(Throwable.java:482)
	at com.mysql.jdbc.Util.handleNewInstance(Util.java:421)
	at com.mysql.jdbc.PreparedStatement.getInstance(PreparedStatement.java:824)
	at com.mysql.jdbc.ConnectionImpl.clientPrepareStatement(ConnectionImpl.java:1399)
	at com.mysql.jdbc.ConnectionImpl.prepareStatement(ConnectionImpl.java:4102)
	- locked <0x000000068ac8e2b0> (a com.mysql.jdbc.JDBC4Connection)
	at com.mysql.jdbc.ConnectionImpl.prepareStatement(ConnectionImpl.java:4006)
	at com.zaxxer.hikari.pool.ProxyConnection.prepareStatement(ProxyConnection.java:337)
	at com.zaxxer.hikari.pool.HikariProxyConnection.prepareStatement(HikariProxyConnection.java)
	at com.ntko.dao.impl.BatchSettingDaoImpl.getPoolingBatchSettings(BatchSettingDaoImpl.java:336)
	at com.ntko.init.PoolingThread.getSourceFile(PoolingThread.java:43)
	at com.ntko.init.PoolingThread.lambda$0(PoolingThread.java:95)
	at com.ntko.init.PoolingThread$$Lambda$47/149694545.run(Unknown Source)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

"AFileConvert" #44 prio=5 os_prio=0 tid=0x00007f46ac002000 nid=0xbcc05 waiting on condition [0x00007f47202fd000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x000000068e5959a8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
	at com.ntko.convert.thread.AConvert.run(AConvert.java:1268)

"SignThread" #43 prio=5 os_prio=0 tid=0x00007f46b4008800 nid=0xbcc04 waiting on condition [0x00007f471bdfe000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x000000068e595c58> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
	at com.ntko.sign.thread.SignThread.run(SignThread.java:230)

"pool-7-thread-1" #42 prio=5 os_prio=0 tid=0x00007f46b8005000 nid=0xbcc03 waiting for monitor entry [0x00007f471befe000]
   java.lang.Thread.State: BLOCKED (on object monitor)
	at com.mysql.jdbc.ConnectionImpl.prepareStatement(ConnectionImpl.java:4036)
	- waiting to lock <0x000000068ac8e2b0> (a com.mysql.jdbc.JDBC4Connection)
	at com.mysql.jdbc.ConnectionImpl.prepareStatement(ConnectionImpl.java:4006)
	at com.zaxxer.hikari.pool.ProxyConnection.prepareStatement(ProxyConnection.java:337)
	at com.zaxxer.hikari.pool.HikariProxyConnection.prepareStatement(HikariProxyConnection.java)
	at com.ntko.dao.impl.FileConvertInfoDaoImpl.getAllConvertInfo(FileConvertInfoDaoImpl.java:26)
	at com.ntko.sign.thread.BatchSignThread.lambda$0(BatchSignThread.java:149)
	at com.ntko.sign.thread.BatchSignThread$$Lambda$46/329650903.run(Unknown Source)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

"Thread-13" #37 daemon prio=5 os_prio=0 tid=0x00007f46a4005800 nid=0xbcbfe runnable [0x00007f47204fd000]
   java.lang.Thread.State: RUNNABLE
	at sun.nio.fs.LinuxWatchService.poll(Native Method)
	at sun.nio.fs.LinuxWatchService.access$600(LinuxWatchService.java:47)
	at sun.nio.fs.LinuxWatchService$Poller.run(LinuxWatchService.java:314)
	at java.lang.Thread.run(Thread.java:748)

"MonitorThread" #36 prio=5 os_prio=0 tid=0x00007f4740fd4000 nid=0xbcbfd waiting on condition [0x00007f47207fd000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x000000068e59f588> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
	at java.util.concurrent.LinkedBlockingDeque.takeFirst(LinkedBlockingDeque.java:492)
	at java.util.concurrent.LinkedBlockingDeque.take(LinkedBlockingDeque.java:680)
	at sun.nio.fs.AbstractWatchService.take(AbstractWatchService.java:118)
	at com.ntko.init.MonitorThread.run(MonitorThread.java:389)

"Timer-1" #34 prio=5 os_prio=0 tid=0x00007f4740f02000 nid=0xbcbfb in Object.wait() [0x00007f47208fd000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	at java.util.TimerThread.mainLoop(Timer.java:552)
	- locked <0x000000068ab2d540> (a java.util.TaskQueue)
	at java.util.TimerThread.run(Timer.java:505)

"HikariPool-2 housekeeper" #33 daemon prio=5 os_prio=0 tid=0x00007f4740420800 nid=0xbcbfa runnable [0x00007f47209fd000]
   java.lang.Thread.State: RUNNABLE
	at com.zaxxer.hikari.pool.HikariPool$HouseKeeper.run(HikariPool.java:808)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

"mysql-cj-abandoned-connection-cleanup" #32 daemon prio=5 os_prio=0 tid=0x00007f474134a000 nid=0xbcbf9 in Object.wait() [0x00007f4728211000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:144)
	- locked <0x000000068ac207b8> (a java.lang.ref.ReferenceQueue$Lock)
	at com.mysql.jdbc.AbandonedConnectionCleanupThread.run(AbandonedConnectionCleanupThread.java:80)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

"pool-4-thread-1" #31 prio=5 os_prio=0 tid=0x00007f46b0003800 nid=0xbcbf8 runnable [0x00007f4728111000]
   java.lang.Thread.State: RUNNABLE
	at java.util.Arrays.copyOf(Arrays.java:3181)
	at java.util.concurrent.CopyOnWriteArrayList.addIfAbsent(CopyOnWriteArrayList.java:637)
	at java.util.concurrent.CopyOnWriteArrayList.addIfAbsent(CopyOnWriteArrayList.java:615)
	at com.mysql.jdbc.ConnectionImpl.registerStatement(ConnectionImpl.java:4297)
	at com.mysql.jdbc.StatementImpl.<init>(StatementImpl.java:319)
	at com.mysql.jdbc.PreparedStatement.<init>(PreparedStatement.java:905)
	at com.mysql.jdbc.JDBC4PreparedStatement.<init>(JDBC4PreparedStatement.java:49)
	at com.mysql.jdbc.JDBC42PreparedStatement.<init>(JDBC42PreparedStatement.java:43)
	at sun.reflect.GeneratedConstructorAccessor128.newInstance(Unknown Source)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
	at com.mysql.jdbc.Util.handleNewInstance(Util.java:403)
	at com.mysql.jdbc.PreparedStatement.getInstance(PreparedStatement.java:824)
	at com.mysql.jdbc.ConnectionImpl.clientPrepareStatement(ConnectionImpl.java:1399)
	at com.mysql.jdbc.ConnectionImpl.prepareStatement(ConnectionImpl.java:4102)
	- locked <0x0000000683c7cb00> (a com.mysql.jdbc.JDBC4Connection)
	at com.mysql.jdbc.ConnectionImpl.prepareStatement(ConnectionImpl.java:4006)
	at com.zaxxer.hikari.pool.ProxyConnection.prepareStatement(ProxyConnection.java:337)
	at com.zaxxer.hikari.pool.HikariProxyConnection.prepareStatement(HikariProxyConnection.java)
	at com.ntko.dao.impl.FileConvertInfoDaoImpl.getAllConvertInfo(FileConvertInfoDaoImpl.java:26)
	at com.ntko.sign.thread.BatchSignThread.lambda$0(BatchSignThread.java:149)
	at com.ntko.sign.thread.BatchSignThread$$Lambda$39/182093380.run(Unknown Source)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

"SignThread" #30 prio=5 os_prio=0 tid=0x00007f46a4001800 nid=0xbcbf7 waiting on condition [0x00007f4720efd000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x0000000686dbcf10> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
	at com.ntko.sign.thread.SignThread.run(SignThread.java:230)

"pool-3-thread-1" #29 prio=5 os_prio=0 tid=0x00007f469c001800 nid=0xbcbf6 waiting for monitor entry [0x00007f4728411000]
   java.lang.Thread.State: BLOCKED (on object monitor)
	at com.mysql.jdbc.ConnectionImpl.prepareStatement(ConnectionImpl.java:4036)
	- waiting to lock <0x0000000683c7cb00> (a com.mysql.jdbc.JDBC4Connection)
	at com.mysql.jdbc.ConnectionImpl.prepareStatement(ConnectionImpl.java:4006)
	at com.zaxxer.hikari.pool.ProxyConnection.prepareStatement(ProxyConnection.java:337)
	at com.zaxxer.hikari.pool.HikariProxyConnection.prepareStatement(HikariProxyConnection.java)
	at com.ntko.dao.impl.BatchSettingDaoImpl.getPoolingBatchSettings(BatchSettingDaoImpl.java:336)
	at com.ntko.init.PoolingThread.getSourceFile(PoolingThread.java:43)
	at com.ntko.init.PoolingThread.lambda$0(PoolingThread.java:91)
	at com.ntko.init.PoolingThread$$Lambda$38/1378276112.run(Unknown Source)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

"AFileConvert" #27 prio=5 os_prio=0 tid=0x00007f46b8003800 nid=0xbcbf4 waiting on condition [0x00007f4720ffd000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x0000000686dc11f0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
	at com.ntko.convert.thread.AConvert.run(AConvert.java:1355)

"Thread-6" #24 daemon prio=5 os_prio=0 tid=0x00007f46b4007800 nid=0xbcbf1 runnable [0x00007f4728311000]
   java.lang.Thread.State: RUNNABLE
	at sun.nio.fs.LinuxWatchService.poll(Native Method)
	at sun.nio.fs.LinuxWatchService.access$600(LinuxWatchService.java:47)
	at sun.nio.fs.LinuxWatchService$Poller.run(LinuxWatchService.java:314)
	at java.lang.Thread.run(Thread.java:748)

"MonitorThread" #22 prio=5 os_prio=0 tid=0x00007f474143f000 nid=0xbcbef waiting on condition [0x00007f4728711000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x0000000686dc1918> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
	at java.util.concurrent.LinkedBlockingDeque.takeFirst(LinkedBlockingDeque.java:492)
	at java.util.concurrent.LinkedBlockingDeque.take(LinkedBlockingDeque.java:680)
	at sun.nio.fs.AbstractWatchService.take(AbstractWatchService.java:118)
	at com.ntko.init.MonitorThread.run(MonitorThread.java:389)

"Timer-0" #20 prio=5 os_prio=0 tid=0x00007f47415db800 nid=0xbcbed in Object.wait() [0x00007f4728811000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	at java.util.TimerThread.mainLoop(Timer.java:552)
	- locked <0x0000000683e44de0> (a java.util.TaskQueue)
	at java.util.TimerThread.run(Timer.java:505)

"HikariPool-1 housekeeper" #19 daemon prio=5 os_prio=0 tid=0x00007f47414e8000 nid=0xbcbec waiting on condition [0x00007f4728b11000]
   java.lang.Thread.State: TIMED_WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x0000000683c95f38> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

"mysql-cj-abandoned-connection-cleanup" #18 daemon prio=5 os_prio=0 tid=0x00007f47413fe000 nid=0xbcbeb in Object.wait() [0x00007f4728e11000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:144)
	- locked <0x0000000683e3ee68> (a java.lang.ref.ReferenceQueue$Lock)
	at com.mysql.jdbc.AbandonedConnectionCleanupThread.run(AbandonedConnectionCleanupThread.java:80)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

"http-nio-8081-BlockPoller" #17 daemon prio=5 os_prio=0 tid=0x00007f47406f1000 nid=0xbcbe4 runnable [0x00007f4729dfd000]
   java.lang.Thread.State: RUNNABLE
	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
	- locked <0x000000068275d168> (a sun.nio.ch.Util$3)
	- locked <0x000000068275d158> (a java.util.Collections$UnmodifiableSet)
	- locked <0x000000068275d030> (a sun.nio.ch.EPollSelectorImpl)
	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
	at org.apache.tomcat.util.net.NioBlockingSelector$BlockPoller.run(NioBlockingSelector.java:313)

"GC Daemon" #16 daemon prio=2 os_prio=0 tid=0x00007f4740638800 nid=0xbcbe3 in Object.wait() [0x00007f472a0fd000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	- waiting on <0x0000000682f1ca40> (a sun.misc.GC$LatencyLock)
	at sun.misc.GC$Daemon.run(GC.java:117)
	- locked <0x0000000682f1ca40> (a sun.misc.GC$LatencyLock)

"RMI TCP Accept-0" #15 daemon prio=5 os_prio=0 tid=0x00007f47402b3000 nid=0xbcbe1 runnable [0x00007f472a9fe000]
   java.lang.Thread.State: RUNNABLE
	at java.net.PlainSocketImpl.socketAccept(Native Method)
	at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:535)
	at java.net.ServerSocket.implAccept(ServerSocket.java:545)
	at java.net.ServerSocket.accept(ServerSocket.java:513)
	at sun.management.jmxremote.LocalRMIServerSocketFactory$1.accept(LocalRMIServerSocketFactory.java:52)
	at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:405)
	at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:377)
	at java.lang.Thread.run(Thread.java:748)

"RMI TCP Accept-18999" #14 daemon prio=5 os_prio=0 tid=0x00007f4740289000 nid=0xbcbe0 runnable [0x00007f472adfe000]
   java.lang.Thread.State: RUNNABLE
	at java.net.PlainSocketImpl.socketAccept(Native Method)
	at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:535)
	at java.net.ServerSocket.implAccept(ServerSocket.java:545)
	at java.net.ServerSocket.accept(ServerSocket.java:513)
	at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:405)
	at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:377)
	at java.lang.Thread.run(Thread.java:748)

"RMI TCP Accept-0" #13 daemon prio=5 os_prio=0 tid=0x00007f474027a800 nid=0xbcbdf runnable [0x00007f472b1fb000]
   java.lang.Thread.State: RUNNABLE
	at java.net.PlainSocketImpl.socketAccept(Native Method)
	at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:535)
	at java.net.ServerSocket.implAccept(ServerSocket.java:545)
	at java.net.ServerSocket.accept(ServerSocket.java:513)
	at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:405)
	at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:377)
	at java.lang.Thread.run(Thread.java:748)

"FileHandlerLogFilesCleaner-1" #12 daemon prio=5 os_prio=0 tid=0x00007f4740215000 nid=0xbcbde waiting on condition [0x00007f472b2fb000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x000000068275d8b0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)

"AsyncFileHandlerWriter-1625635731" #11 daemon prio=5 os_prio=0 tid=0x00007f4740199800 nid=0xbcbdd runnable [0x00007f472b3fb000]
   java.lang.Thread.State: RUNNABLE
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.addConditionWaiter(AbstractQueuedSynchronizer.java:1855)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2068)
	at java.util.concurrent.LinkedBlockingDeque.pollFirst(LinkedBlockingDeque.java:522)
	at java.util.concurrent.LinkedBlockingDeque.poll(LinkedBlockingDeque.java:684)
	at org.apache.juli.AsyncFileHandler$LoggerThread.run(AsyncFileHandler.java:159)

"Service Thread" #8 daemon prio=9 os_prio=0 tid=0x00007f47400cc800 nid=0xbcbdc runnable [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"C1 CompilerThread2" #7 daemon prio=9 os_prio=0 tid=0x00007f47400c1800 nid=0xbcbdb waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"C2 CompilerThread1" #6 daemon prio=9 os_prio=0 tid=0x00007f47400bf800 nid=0xbcbda waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"C2 CompilerThread0" #5 daemon prio=9 os_prio=0 tid=0x00007f47400bc800 nid=0xbcbd9 runnable [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"Signal Dispatcher" #4 daemon prio=9 os_prio=0 tid=0x00007f47400bb000 nid=0xbcbd8 runnable [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"Finalizer" #3 daemon prio=8 os_prio=0 tid=0x00007f4740088800 nid=0xbcbd7 in Object.wait() [0x00007f472bdfd000]
   java.lang.Thread.State: WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:144)
	- locked <0x0000000682b61430> (a java.lang.ref.ReferenceQueue$Lock)
	at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:165)
	at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:216)

"Reference Handler" #2 daemon prio=10 os_prio=0 tid=0x00007f4740084000 nid=0xbcbd6 in Object.wait() [0x00007f472befd000]
   java.lang.Thread.State: WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	at java.lang.Object.wait(Object.java:502)
	at java.lang.ref.Reference.tryHandlePending(Reference.java:191)
	- locked <0x0000000682f1d068> (a java.lang.ref.Reference$Lock)
	at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:153)

"main" #1 prio=5 os_prio=0 tid=0x00007f474000a800 nid=0xbcbd0 runnable [0x00007f474751e000]
   java.lang.Thread.State: RUNNABLE
	at java.net.PlainSocketImpl.socketAccept(Native Method)
	at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:535)
	at java.net.ServerSocket.implAccept(ServerSocket.java:545)
	at java.net.ServerSocket.accept(ServerSocket.java:513)
	at org.apache.catalina.core.StandardServer.await(StandardServer.java:609)
	at org.apache.catalina.startup.Catalina.await(Catalina.java:864)
	at org.apache.catalina.startup.Catalina.start(Catalina.java:810)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:342)
	at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:473)

"VM Thread" os_prio=0 tid=0x00007f474007a000 nid=0xbcbd5 runnable 

"GC task thread#0 (ParallelGC)" os_prio=0 tid=0x00007f4740021800 nid=0xbcbd1 runnable 

"GC task thread#1 (ParallelGC)" os_prio=0 tid=0x00007f4740023000 nid=0xbcbd2 runnable 

"GC task thread#2 (ParallelGC)" os_prio=0 tid=0x00007f4740025000 nid=0xbcbd3 runnable 

"GC task thread#3 (ParallelGC)" os_prio=0 tid=0x00007f4740026800 nid=0xbcbd4 runnable 

"VM Periodic Task Thread" os_prio=0 tid=0x00007f47402b5800 nid=0xbcbe2 waiting on condition 

JNI global references: 308

4.4线程状态基础知识

要分析线程的状态，首先需要知道线程有哪几种状态，在日志里怎么呈现？

线程有6种状态：

1、 NEW：当线程被创建出来还没有被调用start()时候的状态。

2、RUNNABLE：当线程被调用了start()，且处于等待操作系统分配资源（如CPU）、等待IO连接、正在运行状态，即表示Running状态和Ready状态。

注：不一定被调用了start()立刻会改变状态，还有一些准备工作，这个时候的状态是不确定的。

3、BLOCKED:等待监视器锁而被阻塞的线程的线程状态，当进入synchronized块/方法或者在调用wait()被唤醒/超时之后重新进入synchronized块/方法，但是锁被其它线程占有，这个时候被操作系统挂起，状态为阻塞状态。阻塞状态的线程，即使调用interrupt()方法也不会改变其状态。

4、WAITING ：无条件等待，当线程调用wait()、join()、LockSupport.park()不加超时时间的方法之后所处的状态，如果没有被唤醒或等待的线程没有结束，那么将一直等待，当前状态的线程不会被分配CPU资源和持有锁.可理解为：无限期等待。

5、TIMED_WAITING：有条件的等待，当线程调用sleep(睡眠时间)/wait(等待时间)/join(等待时间)/ LockSupport.parkNanos(等待时间)/LockSupport.parkUntil(等待时间)方法之后所处的状态，在指定的时间没有被唤醒或者等待线程没有结束，会被系统自动唤醒，正常退出。

6、TERMINATED ：执行完了run()方法。其实这只是Java语言级别的一种状态，在操作系统内部可能已经注销了相应的线程，或者将它复用给其他需要使用线程的请求，而在Java语言级别只是通过Java代码看到的线程状态而已。

Thread的状态使用java.lang.Thread.State枚举表示。

线程状态状态转换图：

更多线程状态说明以及相关代码详解，可以参考线程6种状态_线程状态-优快云博客，这里我也是用了大神文章里的图以及解释的。

根据线程状态，可知：当线程获取不到需要的锁时就会进入BLOCKED状态即阻塞状态。

而当线程为WAITING、TIMED_WAITING时是不会占用CPU资源以及持有锁的。当线程状态为阻塞时是否释放CPU,则取决于阻塞的原因以及操作系统的行为。

1、资源等待引起的阻塞：

当线程因为等待某个资源（如锁、信号量、条件变量等）而阻塞时，它通常会释放CPU。会让出CPU:阻塞的线程不会占用CPU资源，直到它等待的资源变得可用。

2、等待I/O操作引起的阻塞：

等待I/O操作（如读取文件、网络通信等）的线程也会释放CPU，操作系统将线程挂起，直到I/O操作完成。

3、等待用户输入引起的阻塞：

等待用户输入（如从控制台读取数据）的线程会释放CPU。

4、主动阻塞：

调用Thread.sleep()或Object.wait()等方法主动阻塞的线程会释放CPU 。

5、就绪队列阻塞

阻塞在就绪队列中的线程（等待条件变量的信号）也会释放CPU。

6、特殊情况

自旋锁不释放CPU：在等待锁时“自旋”，即不断检查锁是否可用，而不立即释放CPU。
忙等待不释放CPU：不断检查某个条件是否满足，而不释放CPU。

线程阻塞状态下，有自旋锁以及忙等待引起人阻塞会占用CPU。

4.5 结合日志分析

根据线程不同状态时找到日志里对应的线程状态信息 ,尝试在抓取到的日志里找：BLOCKED可找到如下信息：

可以看到提示说：  java.lang.Thread.State: BLOCKED (on object monitor)
   at com.mysql.jdbc.ConnectionImpl.prepareStatement(ConnectionImpl.java:4036)
   - waiting to lock <0x000000068ac8e2b0> (a com.mysql.jdbc.JDBC4Connection)
   at com.mysql.jdbc.ConnectionImpl.prepareStatement(ConnectionImpl.java:4006)

在等待锁住0x000000068ac8e2b0 ，是一个数据库的JDBC连接情况

再尝试找这个0x000000068ac8e2b0，可以找到：

at com.mysql.jdbc.ConnectionImpl.prepareStatement(ConnectionImpl.java:4102)
- locked <0x000000068ac8e2b0> (a com.mysql.jdbc.JDBC4Connection)

上一个线程做了一个 com.mysql.jdbc.JDBC4Connection 然后把这个0x000000068ac8e2b0 锁住了。

结合前面分析的：

可以猜想是由于不断检查这个锁 0x000000068ac8e2b0 是否可用，出现的自旋锁？？？反正就是不释放CPU了。。。。。。

到这里大致就可以分析出是由于数据库连接的问题引起了这个线程阻塞。那么就还需要找到这个具体执行的sql语句是哪一句。

查看正在运行的sql语句

我这里用的是mariadb ,可以参考msyql 查看正在运行的sql的方法：

1、select * from information_schema.processlist; 查看当前正在执行的processlist

通过processlist表，可以查看到：

ID: 线程的id
USER: 线程属于哪一个用户
HOST：客户端的host信息：hostname+端口
DB：线程在哪一个数据库下
COMMAND：线程使用哪一种命令在执行，空闲的线程状态为sleep
TIME:线程已经运行的时间，秒为单位
STATE：线程正在做什么：当前的状态，行为，或者事件
INFO：线程正在执行的语句。

但是这个并不是很准确，所以需要使用其他的方式来完成我们的目标。大抵可以通过这个 select * from information_schema.processlist; 确认哪一种命令正在执行。

2、 select * from performance_schema.threads；查询正在运行的线程信息

threads 表位于performance_schema库中，每一行记录的是一条服务器线程。当performance_schema初始化的时候，它会根据当时存在的线程填充线程表，之后每当服务器创建线程时，都会添加一条新数据。当线程结束线程表中也会删除这条数据。使用 desc performance_schema.threads 来查看表结构：

THREAD_ID：线程唯一id
NAME:与服务器中的线程监测代码相关联的名称
TYPE：线程类型。分为前台和后台。用户连线程是前台线程，与内部服务器活动相关的线程是后端线程。例如InnoDB内部线程。
PROCESSLIST_ID：上边的processlist的id
PROCESSLIST_USER：上边的processlist的user
PROCESSLIST_HOST:上边的processlist的host
PROCESSLIST_DB:上边的processlist的DB
PROCESSLIST_COMMAND:上边的processlist的COMMAND
PROCESSLIST_TIME:上边的processlist的TIME
PROCESSLIST_STATE:上边的processlist的STATE
PROCESSLIST_INFO:上边的processlist的INFO
PARENT_THREAD_ID:衍生线程的id值
ROLE：未使用
INSTRUMENTED：线程执行的事件是否插装，（YES or ON)
HISTORY:是否为线程记录历史事件
CONNECTION_TYPE：用于建立连接的协议，或者NULL用于后台线程。
THREAD_OS_ID：MySQL线程对应于操作系统的线程id

未完待续........