今天工作中遇到一个问题,用户程序无法启动,报连接数据库异常:
错误日志分析如下:
|2014-07-09 16:58:08|WARN |ThreadPoolAsynchronousRunner$DeadlockDetector:608|com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector@df0092 -- APPARENT DEADLOCK!!! Creating emergency threads for unassigned pending tasks! |
||2014-07-09 16:58:08|WARN |ThreadPoolAsynchronousRunner$DeadlockDetector:624|com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector@df0092 -- APPARENT DEADLOCK!!! Complete Status:
Managed Threads: 3
Active Threads: 3
Active Tasks:
com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@18a3ddc (com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#2)
com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@e371c6 (com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#0)
com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@42bc2b (com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#1)
Pending Tasks:
com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@14fb98b
com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@11e8696
com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@115ff26
com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@122195f
com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@1afb8dd
Pool thread stack traces:
Thread[com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#2,5,main]
java.net.Inet4AddressImpl.getHostByAddr(Native Method)
java.net.InetAddress$1.getHostByAddr(InetAddress.java:842)
java.net.InetAddress.getHostFromNameService(InetAddress.java:532)
java.net.InetAddress.getHostName(InetAddress.java:475)
java.net.InetAddress.getHostName(InetAddress.java:447)
java.net.InetSocketAddress.getHostName(InetSocketAddress.java:210)
java.net.SocksSocketImpl.connect(SocksSocketImpl.java:341)
java.net.Socket.connect(Socket.java:507)
java.net.Socket.connect(Socket.java:457)
java.net.Socket.<init>(Socket.java:365)
java.net.Socket.<init>(Socket.java:178)
oracle.net.nt.TcpNTAdapter.connect(Unknown Source)
oracle.net.nt.ConnOption.connect(Unknown Source)
oracle.net.nt.ConnStrategy.execute(Unknown Source)
oracle.net.resolver.AddrResolution.resolveAndExecute(Unknown Source)
oracle.net.ns.NSProtocol.establishConnection(Unknown Source)
oracle.net.ns.NSProtocol.connect(Unknown Source)
oracle.jdbc.ttc7.TTC7Protocol.connect(TTC7Protocol.java:1687)
oracle.jdbc.ttc7.TTC7Protocol.logon(TTC7Protocol.java:214)
oracle.jdbc.driver.OracleConnection.<init>(OracleConnection.java:346)
oracle.jdbc.driver.OracleDriver.getConnectionInstance(OracleDriver.java:468)
oracle.jdbc.driver.OracleDriver.connect(OracleDriver.java:314)
com.mchange.v2.c3p0.DriverManagerDataSource.getConnection(DriverManagerDataSource.java:134)
com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:182)
com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:171)
com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool$1PooledConnectionResourcePoolManager.acquireResource(C3P0PooledConnectionPool.java:137)
com.mchange.v2.resourcepool.BasicResourcePool.doAcquire(BasicResourcePool.java:1014)
com.mchange.v2.resourcepool.BasicResourcePool.access$800(BasicResourcePool.java:32)
com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask.run(BasicResourcePool.java:1810)
com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:547)
Thread[com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#0,5,main]
java.net.Inet4AddressImpl.getHostByAddr(Native Method)
java.net.InetAddress$1.getHostByAddr(InetAddress.java:842)
java.net.InetAddress.getHostFromNameService(InetAddress.java:532)
java.net.InetAddress.getHostName(InetAddress.java:475)
java.net.InetAddress.getHostName(InetAddress.java:447)
java.net.InetSocketAddress.getHostName(InetSocketAddress.java:210)
java.net.SocksSocketImpl.connect(SocksSocketImpl.java:341)
java.net.Socket.connect(Socket.java:507)
java.net.Socket.connect(Socket.java:457)
java.net.Socket.<init>(Socket.java:365)
java.net.Socket.<init>(Socket.java:178)
oracle.net.nt.TcpNTAdapter.connect(Unknown Source)
oracle.net.nt.ConnOption.connect(Unknown Source)
oracle.net.nt.ConnStrategy.execute(Unknown Source)
oracle.net.resolver.AddrResolution.resolveAndExecute(Unknown Source)
oracle.net.ns.NSProtocol.establishConnection(Unknown Source)
oracle.net.ns.NSProtocol.connect(Unknown Source)
oracle.jdbc.ttc7.TTC7Protocol.connect(TTC7Protocol.java:1687)
oracle.jdbc.ttc7.TTC7Protocol.logon(TTC7Protocol.java:214)
oracle.jdbc.driver.OracleConnection.<init>(OracleConnection.java:346)
oracle.jdbc.driver.OracleDriver.getConnectionInstance(OracleDriver.java:468)
oracle.jdbc.driver.OracleDriver.connect(OracleDriver.java:314)
com.mchange.v2.c3p0.DriverManagerDataSource.getConnection(DriverManagerDataSource.java:134)
com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:182)
com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:171)
com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool$1PooledConnectionResourcePoolManager.acquireResource(C3P0PooledConnectionPool.java:137)
com.mchange.v2.resourcepool.BasicResourcePool.doAcquire(BasicResourcePool.java:1014)
com.mchange.v2.resourcepool.BasicResourcePool.access$800(BasicResourcePool.java:32)
com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask.run(BasicResourcePool.java:1810)
com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:547)
Thread[com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#1,5,main]
java.net.Inet4AddressImpl.getHostByAddr(Native Method)
java.net.InetAddress$1.getHostByAddr(InetAddress.java:842)
java.net.InetAddress.getHostFromNameService(InetAddress.java:532)
java.net.InetAddress.getHostName(InetAddress.java:475)
java.net.InetAddress.getHostName(InetAddress.java:447)
java.net.InetSocketAddress.getHostName(InetSocketAddress.java:210)
java.net.SocksSocketImpl.connect(SocksSocketImpl.java:341)
java.net.Socket.connect(Socket.java:507)
java.net.Socket.connect(Socket.java:457)
java.net.Socket.<init>(Socket.java:365)
java.net.Socket.<init>(Socket.java:178)
oracle.net.nt.TcpNTAdapter.connect(Unknown Source)
oracle.net.nt.ConnOption.connect(Unknown Source)
oracle.net.nt.ConnStrategy.execute(Unknown Source)
oracle.net.resolver.AddrResolution.resolveAndExecute(Unknown Source)
oracle.net.ns.NSProtocol.establishConnection(Unknown Source)
oracle.net.ns.NSProtocol.connect(Unknown Source)
oracle.jdbc.ttc7.TTC7Protocol.connect(TTC7Protocol.java:1687)
oracle.jdbc.ttc7.TTC7Protocol.logon(TTC7Protocol.java:214)
oracle.jdbc.driver.OracleConnection.<init>(OracleConnection.java:346)
oracle.jdbc.driver.OracleDriver.getConnectionInstance(OracleDriver.java:468)
oracle.jdbc.driver.OracleDriver.connect(OracleDriver.java:314)
com.mchange.v2.c3p0.DriverManagerDataSource.getConnection(DriverManagerDataSource.java:134)
com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:182)
com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:171)
com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool$1PooledConnectionResourcePoolManager.acquireResource(C3P0PooledConnectionPool.java:137)
com.mchange.v2.resourcepool.BasicResourcePool.doAcquire(BasicResourcePool.java:1014)
com.mchange.v2.resourcepool.BasicResourcePool.access$800(BasicResourcePool.java:32)
com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask.run(BasicResourcePool.java:1810)
com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:547)
分析原因:
造成程序启动失败的根本原因是数据库
可能性:
1、数据库连接池异常---尝试重启程序所在服务器
2、数据库连接配置错误(之前程序一直运行稳定正常,从未修改过数据库连接配置文件)----排除
3、数据库容量不足
4、数据库服务相关异常
经与客户共同排查,排查方式如下:
1、重启服务器释放资源,重启程序----仍然报如上错误
2、数据库空间正常
3、有其他程序能正常连接数据库,并无异常发生
---------------------------由上推断把可能性1、3、4都排除了
那么下面只剩下数据库连接配置问题了,(这怎么可能!用户未修改过用户名和密码,况且他们连配置文件存放位置都不知道)
说道配置文件中的数据库配置参数:
DRIVER、dialect、URL、user、password
不会变的有:DRIVER、dialect
那么考虑一下其他三个可能性,用户向我咨询了配置数据库的文件存放位置,找到了配置文件中数据库相关的配置参数
最后通过用户调查,查出问题的根本原因是数据库将用户上锁了
至此问题排查结束,重启程序恢复正常
------------------------------------------------------
发表此博客就是为了勉励自己,遇到问题时有时候不仅仅要看经验或者是否遇到过
也不必急于查出问题,一一罗列出可能性(我取名叫散打法)虽然看是复杂但有的时候未免不是一种有效又条例清晰的方法,走的都是直线不会绕弯
不知道是否看过盗墓笔记中胖子曾多次使用这种方式解决了很多棘手问题。。。