1、异常描述
myeclipse2014运行web项目Tomcat崩溃,界面输出异常错误,并在Tomcat/bin文件夹下生成hs_err_pid****.log、replay_pid****.log文件。hs_err_pid****.log异常内容如下:
#
# A fatal error has been detected by the Java Runtime Environment:
#
# EXCEPTION_ACCESS_VIOLATION (0xc0000005) at pc=0x0000000075894686, pid=37796, tid=38616
#
# JRE version: Java(TM) SE Runtime Environment (8.0_31-b13) (build 1.8.0_31-b13)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (25.31-b07 mixed mode windows-amd64 compressed oops)
# Problematic frame:
# V [jvm.dll+0x4c4686]
#
# Failed to write core dump. Minidumps are not enabled by default on client versions of Windows
#
# If you would like to submit a bug report, please visit:
# http://bugreport.java.com/bugreport/crash.jsp
#
以下省略.....
tomcat/并生成的文件截图:
2、异常分析
异常内容:EXCEPTION_ACCESS_VIOLATION (0xc0000005) at pc=0x0000000075894686, pid=37796, tid=38616
异常含义:
表示jvm crash时正在执行jvm程序,jvm发生bug导致crash,代码内容:
Problematic frame:
V [jvm.dll+0x4c4686]
根据异常的tid找到crash的线程信息。线程类型为JavaThread,JavaThread表示执行的是java线程。线程名为C2 CompilerThread0
通过以上信息,可判断是由于程序本身出现了问题。
3、问题处理
通过排查服务器程序日志,发现异常发生前,曾产生以下错误:
五月 19, 2022 9:35:51 上午 com.mchange.v2.c3p0.impl.NewPooledConnection handleThrowable
警告: [c3p0] Another error has occurred [ java.sql.SQLException: No operations allowed after statement closed. ] which will not be reported to listeners!
java.sql.SQLException: No operations allowed after statement closed.
at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:129)
at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:97)
at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:89)
at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:63)
at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:73)
at com.mysql.cj.jdbc.exceptions.SQLExceptionsMapping.translateException(SQLExceptionsMapping.java:82)
at com.mysql.cj.jdbc.StatementImpl.clearWarnings(StatementImpl.java:410)
at com.mchange.v2.c3p0.impl.NewProxyPreparedStatement.clearWarnings(NewProxyPreparedStatement.java:1057)
at org.hibernate.engine.jdbc.spi.SqlExceptionHelper.handleAndClearWarnings(SqlExceptionHelper.java:329)
at org.hibernate.engine.jdbc.spi.SqlExceptionHelper.logAndClearWarnings(SqlExceptionHelper.java:273)
at org.hibernate.engine.jdbc.internal.JdbcCoordinatorImpl.close(JdbcCoordinatorImpl.java:529)
at org.hibernate.engine.jdbc.internal.JdbcCoordinatorImpl.release(JdbcCoordinatorImpl.java:421)
at org.hibernate.loader.Loader.doQuery(Loader.java:923)
at org.hibernate.loader.Loader.doQueryAndInitializeNonLazyCollections(Loader.java:354)
at org.hibernate.loader.Loader.doList(Loader.java:2553)
at org.hibernate.loader.Loader.doList(Loader.java:2539)
at org.hibernate.loader.Loader.listIgnoreQueryCache(Loader.java:2369)
at org.hibernate.loader.Loader.list(Loader.java:2364)
at org.hibernate.loader.hql.QueryLoader.list(QueryLoader.java:496)
at org.hibernate.hql.internal.ast.QueryTranslatorImpl.list(QueryTranslatorImpl.java:387)
at org.hibernate.engine.query.spi.HQLQueryPlan.performList(HQLQueryPlan.java:231)
at org.hibernate.internal.SessionImpl.list(SessionImpl.java:1264)
at org.hibernate.internal.QueryImpl.list(QueryImpl.java:103)
at com.cetc7.dao.BaseDAOImpl.find(BaseDAOImpl.java:149)
at com.cetc7.dao.Csr6930ReaderDAO.findReaderbyType(Csr6930ReaderDAO.java:166)
at com.cetc7.dao.Csr6930ReaderDAO$$FastClassBySpringCGLIB$$bd2c18fe.invoke(<generated>)
at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:649)
at com.cetc7.dao.Csr6930ReaderDAO$$EnhancerBySpringCGLIB$$7c03209b.findReaderbyType(<generated>)
at com.cetc7.action.service.ReaderService.getReadersByType(ReaderService.java:430)
at com.cetc7.action.service.ReaderService$$FastClassBySpringCGLIB$$b81ef96.invoke(<generated>)
at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:649)
at com.cetc7.action.service.ReaderService$$EnhancerBySpringCGLIB$$ea6e7c33.getReadersByType(<generated>)
at com.cetc7.action.device.ReaderAction.getReaderStateListbyType(ReaderAction.java:213)
at com.cetc7.action.device.ReaderAction.getRTISStateList(ReaderAction.java:170)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:483)
at ognl.OgnlRuntime.invokeMethodInsideSandbox(OgnlRuntime.java:1266)
at ognl.OgnlRuntime.invokeMethod(OgnlRuntime.java:1251)
at ognl.OgnlRuntime.callAppropriateMethod(OgnlRuntime.java:1969)
at ognl.ObjectMethodAccessor.callMethod(ObjectMethodAccessor.java:68)
at com.opensymphony.xwork2.ognl.accessor.XWorkMethodAccessor.callMethodWithDebugInfo(XWorkMethodAccessor.java:98)
at com.opensymphony.xwork2.ognl.accessor.XWorkMethodAccessor.callMethod(XWorkMethodAccessor.java:90)
at ognl.OgnlRuntime.callMethod(OgnlRuntime.java:2045)
at ognl.ASTMethod.getValueBody(ASTMethod.java:97)
at ognl.SimpleNode.evaluateGetValueBody(SimpleNode.java:212)
at ognl.SimpleNode.getValue(SimpleNode.java:258)
at ognl.Ognl.getValue(Ognl.java:537)
at ognl.Ognl.getValue(Ognl.java:501)
at com.opensymphony.xwork2.ognl.OgnlUtil$3.execute(OgnlUtil.java:492)
at com.opensymphony.xwork2.ognl.OgnlUtil.compileAndExecuteMethod(OgnlUtil.java:544)
at com.opensymphony.xwork2.ognl.OgnlUtil.callMethod(OgnlUtil.java:490)
at com.opensymphony.xwork2.DefaultActionInvocation.invokeAction(DefaultActionInvocation.java:438)
at com.opensymphony.xwork2.DefaultActionInvocation.invokeActionOnly(DefaultActionInvocation.java:293)
at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:254)
at org.apache.struts2.interceptor.debugging.DebuggingInterceptor.intercept(DebuggingInterceptor.java:250)
at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:249)
at com.opensymphony.xwork2.interceptor.DefaultWorkflowInterceptor.doIntercept(DefaultWorkflowInterceptor.java:179)
at com.opensymphony.xwork2.interceptor.MethodFilterInterceptor.intercept(MethodFilterInterceptor.java:99)
at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:249)
at com.opensymphony.xwork2.validator.ValidationInterceptor.doIntercept(ValidationInterceptor.java:263)
at org.apache.struts2.interceptor.validation.AnnotationValidationInterceptor.doIntercept(AnnotationValidationInterceptor.java:49)
at com.opensymphony.xwork2.interceptor.MethodFilterInterceptor.intercept(MethodFilterInterceptor.java:99)
at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:249)
at com.opensymphony.xwork2.interceptor.ConversionErrorInterceptor.doIntercept(ConversionErrorInterceptor.java:142)
at com.opensymphony.xwork2.interceptor.MethodFilterInterceptor.intercept(MethodFilterInterceptor.java:99)
at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:249)
at com.opensymphony.xwork2.interceptor.ParametersInterceptor.doIntercept(ParametersInterceptor.java:140)
at com.opensymphony.xwork2.interceptor.MethodFilterInterceptor.intercept(MethodFilterInterceptor.java:99)
at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:249)
at com.opensymphony.xwork2.interceptor.ParametersInterceptor.doIntercept(ParametersInterceptor.java:140)
at com.opensymphony.xwork2.interceptor.MethodFilterInterceptor.intercept(MethodFilterInterceptor.java:99)
at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:249)
at com.opensymphony.xwork2.interceptor.StaticParametersInterceptor.intercept(StaticParametersInterceptor.java:201)
at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:249)
at org.apache.struts2.interceptor.MultiselectInterceptor.intercept(MultiselectInterceptor.java:67)
at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:249)
at org.apache.struts2.interceptor.DateTextFieldInterceptor.intercept(DateTextFieldInterceptor.java:133)
at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:249)
at org.apache.struts2.interceptor.CheckboxInterceptor.intercept(CheckboxInterceptor.java:89)
at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:249)
at org.apache.struts2.interceptor.FileUploadInterceptor.intercept(FileUploadInterceptor.java:243)
at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:249)
at com.opensymphony.xwork2.interceptor.ModelDrivenInterceptor.intercept(ModelDrivenInterceptor.java:101)
at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:249)
at com.opensymphony.xwork2.interceptor.ScopedModelDrivenInterceptor.intercept(ScopedModelDrivenInterceptor.java:142)
at com.opensymphony.xwork2.DefaultAc
tionInvocation.invoke(DefaultActionInvocation.java:249)
at com.opensymphony.xwork2.interceptor.ChainingInterceptor.intercept(ChainingInterceptor.java:160)
at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:249)
at com.opensymphony.xwork2.interceptor.PrepareInterceptor.doIntercept(PrepareInterceptor.java:175)
at com.opensymphony.xwork2.interceptor.MethodFilterInterceptor.intercept(MethodFilterInterceptor.java:99)
at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:249)
at org.apache.struts2.interceptor.I18nInterceptor.intercept(I18nInterceptor.java:121)
at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:249)
at org.apache.struts2.interceptor.ServletConfigInterceptor.intercept(ServletConfigInterceptor.java:167)
at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:249)
at com.opensymphony.xwork2.interceptor.AliasInterceptor.intercept(AliasInterceptor.java:228)
at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:249)
at com.opensymphony.xwork2.interceptor.ExceptionMappingInterceptor.intercept(ExceptionMappingInterceptor.java:196)
at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:249)
at com.cetc7.interceptors.UserLoginInterceptor.intercept(UserLoginInterceptor.java:22)
at com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:249)
at org.apache.struts2.factory.StrutsActionProxy.execute(StrutsActionProxy.java:48)
at org.apache.struts2.dispatcher.Dispatcher.serviceAction(Dispatcher.java:574)
at org.apache.struts2.dispatcher.ExecuteOperations.executeAction(ExecuteOperations.java:79)
at org.apache.struts2.dispatcher.filter.StrutsPrepareAndExecuteFilter.doFilter(StrutsPrepareAndExecuteFilter.java:141)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.springframework.orm.hibernate4.support.OpenSessionInViewFilter.doFilterInternal(OpenSessionInViewFilter.java:152)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:199)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:543)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:139)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:81)
at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:688)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343)
at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:609)
at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65)
at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:818)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1623)
at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:745)
Caused by: com.mysql.cj.exceptions.StatementIsClosedException: No operations allowed after statement closed.
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:408)
at com.mysql.cj.exceptions.ExceptionFactory.createException(ExceptionFactory.java:61)
at com.mysql.cj.exceptions.ExceptionFactory.createException(ExceptionFactory.java:85)
at com.mysql.cj.jdbc.StatementImpl.checkClosed(StatementImpl.java:340)
at com.mysql.cj.jdbc.StatementImpl.clearWarnings(StatementImpl.java:405)
... 125 more
五月 19, 2022 9:35:51 上午 com.mchange.v2.c3p0.impl.NewPooledConnection handleThrowable
警告: [c3p0] A PooledConnection that has already signalled a Connection error is still in use!
...
The last packet sent successfully to the server was 15780 milliseconds ago. The driver has not received any packets from the server.
...
总结:
结合异常内容分析,此次Tomcat崩溃,主要由以下几个问题一起产生所导致的:
1、mysql数据库网络断开,导致数据库访问失败;
2、C3P0数据池配置的最大空闲时间(maxIdleTime)偏小,已由20秒改成30秒;
3、C3P0缺少检查所有连接池中的空闲连接配置,导致空闲连接增多,最终导致JVM 执行Crash,进而引起Tomcat崩溃,产生上面内容,C3P0增加配置,最终配置如下:
<!-- 连接池中保留的最大连接数。默认值: 15 -->
<property name="maxPoolSize" value="40"></property>
<!-- 连接池中保留的最小连接数,默认为:3 -->
<property name="minPoolSize" value="5"></property>
<!-- 初始化时获取连接数,取值应在minPoolSize与maxPoolSize之间。Default: 3 -->
<property name="initialPoolSize" value="5"></property>
<!-- 最大空闲时间,多少秒内未使用则连接被丢弃。若为0则永不丢弃。默认值: 0,改为30秒 -->
<property name="maxIdleTime" value="30"></property>
<!--每60秒检查所有连接池中的空闲连接。Default: 0 -->
<property name="idleConnectionTestPeriod" value="60"></property>
复测
1、运行Tomcat项目,正常访问网站;
2、断开数据库服务器连接,复现上述异常;
3、由于C3P0已配置完成,数据库服务器断开后,C3P0自动检查连接池的空闲连接进行处理,不会再导致Tomcat崩溃,异常处理完成。
4、延伸
jvm.dll+0x4c0ddf异常处理,参考此链接:jvm 0x4c0ddf异常处理
需进行JDK包升级。