接口获取数据库连接阻塞问题

在并发200的压测中,接口URL的提交充值卡订单接口出现TPS骤降和响应时间飙升,疑似应用卡住。通过jstack日志发现大量线程在TIMED_WAITING状态,等待数据库连接。分析发现,数据库连接池最大连接数为50,当50个线程同时在事务中时,如果MysqlConfigService再尝试获取连接,会因无连接可用而等待,导致所有线程阻塞。解决方案是为MysqlConfigService单独配置dataSource并增大其缓存时间。

摘要生成于 C知道 ,由 DeepSeek-R1 满血版支持, 前往体验 >

问题描述

接口URL:http://192.168.10.200:20200/h5-customer/customer/card/consumerCard/submit?cardId=card_20171010210403659539&cityCode=110100&token=lOukBdwqvVBOpj9EUPiA

在压测提交充值卡订单接口,在并发数200情况下,jmeter 图表显示如下:

tps

响应时间

从上图可以看到,应用在某一个时间点,tps 急剧下降同时响应时间飙升,整个应用存在"卡住"的情况。这个和提交订单接口上次的情况非常相似。

 

分析过程

通过查看jstack 日志发现大量线程状态 TIMED_WAITING

"http-nio-20200-exec-132" daemon prio=10 tid=0x00007f3b8452e800 nid=0x266ce in Object.wait() [0x00007f3b7b255000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	at org.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:1112)
	- locked <0x0000000721c248f8> (a org.apache.commons.pool.impl.GenericObjectPool$Latch)
	at org.apache.commons.dbcp.AbandonedObjectPool.borrowObject(AbandonedObjectPool.java:79)
	at org.apache.commons.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:106)
	at org.apache.commons.dbcp.BasicDataSource.getConnection(BasicDataSource.java:1044)
	at org.springframework.jdbc.datasource.lookup.AbstractRoutingDataSource.getConnection(AbstractRoutingDataSource.java:164)
	at org.springframework.jdbc.datasource.DataSourceTransactionManager.doBegin(DataSourceTransactionManager.java:204)
	at com.helijia.framework.datasource.MasterSlavesDataSourceTransactionManager.doBegin(MasterSlavesDataSourceTransactionManager.java:18)
	at org.springframework.transaction.support.AbstractPlatformTransactionManager.getTransaction(AbstractPlatformTransactionManager.java:373)
	at org.springframework.transaction.interceptor.TransactionAspectSupport.createTransactionIfNecessary(TransactionAspectSupport.java:420)
	at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:257)
	at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:95)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
	at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:207)
	at com.sun.proxy.$Proxy255.submitCardOrder(Unknown Source)
	at com.helijia.customer.h5.modules.card.web.controller.CardController$4.doGetData(CardController.java:137)
	at com.helijia.customer.h5.modules.card.web.controller.CardController$4.doGetData(CardController.java:134)
	at com.helijia.customer.h5.template.H5APIReturnTemplate.getData(H5APIReturnTemplate.java:42)
	at com.helijia.customer.h5.modules.card.web.controller.CardController.consumerCardSubmit(CardController.java:134)
	at sun.reflect.GeneratedMethodAccessor90.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at org.springframework.web.method.support.InvocableHandlerMethod.invoke(InvocableHandlerMethod.java:215)
	at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:132)
	at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:104)
	at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandleMethod(RequestMappingHandlerAdapter.java:749)
	at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:689)
	at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:83)
	at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:938)
	at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:870)
	at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:961)
	at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:863)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:646)
	at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:837)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:727)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
	at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
	at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:88)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
	at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:220)
	at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:122)
	at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:503)
	at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:170)
	at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)
	at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
	at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:421)
	at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1070)
	at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:611)
	at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1736)
	at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1695)
	- locked <0x0000000746a82890> (a org.apache.tomcat.util.net.NioChannel)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
	at java.lang.Thread.run(Thread.java:745)
"http-nio-20200-exec-131" daemon prio=10 tid=0x00007f3b853d1800 nid=0x26595 in Object.wait() [0x00007f3b7c55f000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	at org.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:1112)
	- locked <0x0000000721ab96b8> (a org.apache.commons.pool.impl.GenericObjectPool$Latch)
	at org.apache.commons.dbcp.AbandonedObjectPool.borrowObject(AbandonedObjectPool.java:79)
	at org.apache.commons.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:106)
	at org.apache.commons.dbcp.BasicDataSource.getConnection(BasicDataSource.java:1044)
	at org.springframework.jdbc.datasource.lookup.AbstractRoutingDataSource.getConnection(AbstractRoutingDataSource.java:164)
	at com.helijia.framework.config.MysqlConfigService.getConfigFromRemote(MysqlConfigService.java:31)
	at com.helijia.framework.config.AbstractConfigServiceImpl.getConfig(AbstractConfigServiceImpl.java:80)
	at com.helijia.card.order.service.handler.purchase.PurchaseRequestHandler.closeCardOrder(PurchaseRequestHandler.java:113)
	at com.helijia.card.order.service.handler.purchase.PurchaseRequestHandler.execute(PurchaseRequestHandler.java:82)
	at com.helijia.card.order.service.handler.purchase.PurchaseRequestHandler.execute(PurchaseRequestHandler.java:49)
	at com.helijia.card.order.service.impl.StoreCardOrderServiceImpl.request(StoreCardOrderServiceImpl.java:39)
	at com.helijia.customer.h5.modules.card.service.impl.H5ConsumerCardServiceImpl.submitCardOrder(H5ConsumerCardServiceImpl.java:114)
	at sun.reflect.GeneratedMethodAccessor92.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
	at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:98)
	at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:262)
	at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:95)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
	at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:207)
	at com.sun.proxy.$Proxy255.submitCardOrder(Unknown Source)
	at com.helijia.customer.h5.modules.card.web.controller.CardController$4.doGetData(CardController.java:137)
	at com.helijia.customer.h5.modules.card.web.controller.CardController$4.doGetData(CardController.java:134)
	at com.helijia.customer.h5.template.H5APIReturnTemplate.getData(H5APIReturnTemplate.java:42)
	at com.helijia.customer.h5.modules.card.web.controller.CardController.consumerCardSubmit(CardController.java:134)
	at sun.reflect.GeneratedMethodAccessor90.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at org.springframework.web.method.support.InvocableHandlerMethod.invoke(InvocableHandlerMethod.java:215)
	at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:132)
	at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:104)
	at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandleMethod(RequestMappingHandlerAdapter.java:749)
	at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:689)
	at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:83)
	at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:938)
	at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:870)
	at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:961)
	at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:863)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:646)
	at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:837)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:727)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
	at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
	at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:88)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
	at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:220)
	at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:122)
	at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:503)
	at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:170)
	at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)
	at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
	at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:421)
	at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1070)
	at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:611)
	at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1736)
	at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1695)
	- locked <0x0000000746a827f0> (a org.apache.tomcat.util.net.NioChannel)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
	at java.lang.Thread.run(Thread.java:745)

jstack全部日志:jstack.log

从日志中可以看到,线程在获取数据库连接时候存在等待情况
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at org.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:1112)
GenericObjectPool.borrowObject 等待说明此时数据库连接池中已经没有可用连接。应用数据库连接池配置的最大连接数是50了。

tomcat日志中也存在大量获取不到连接异常

 

通过查看接口逻辑,发现逻辑中使用了MysqlConfigService 动态配置中心服务,那么数据库获取连接和释放连接会发生在下图两个地方,
*@Transactional 事务开启和结束时
*MysqlConfigService 访问数据库时

查看MysqlConfigService 逻辑如下,已经在finally里处理connection的释放。

通过MysqlConfigService代码可以看到,它获取连接是通过dataSource.getConnection(); 
spring配置文件中MysqlConfigService注入的dataSource和spring管理的dataSource是同一个

 

<beans xmlns="http://www.springframework.org/schema/beans" xmlns:mvc="http://www.springframework.org/schema/mvc"
    xmlns:context="http://www.springframework.org/schema/context" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xmlns:task="http://www.springframework.org/schema/task"
    xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-3.0.xsd
                        http://www.springframework.org/schema/context http://www.springframework.org/schema/context/spring-context-3.0.xsd
                        http://www.springframework.org/schema/mvc http://www.springframework.org/schema/mvc/spring-mvc-3.2.xsd
                        http://www.springframework.org/schema/task http://www.springframework.org/schema/task/spring-task-3.2.xsd">

   <bean id="mysqlConfigService" class="com.helijia.framework.config.MysqlConfigService" init-method="init" >
      <property name="dataSource" ref="masterSlaveDataSource" />
   </bean>

</beans>

 

这时如果在高并发情况下,会出现这样一种情况:
如果同时进入@Transactional事务线程数达到50个,每个线程进入事务都会获取一个连接,这时连接池最大连接数配置为50。此时
某一个线程在执行到MysqlConfigService 获取连接时池中已经没有连接了,这时线程会进入等待。同时进入事务中的这50个线程都会执行到MysqlConfigService,此时事务并没有结束,连接不会释放回池中 这50个线程同样会进入等待。
由于整个应用共用一份dataSource,这样系统中其它请求只要会访问到数据库的任何逻辑也会同样进入等待状态。整个应用就”卡住“了。

 

临时解决方案
为MysqlConfigService单独创建一个dataSource,调大MysqlConfigService的缓存时间

### 解决线程获取数据库连接时出现的无限期等待问题 当多个线程尝试访问有限数量的数据库连接池资源时,可能会发生无限期等待的现象。这种情况下,通常是因为某些线程长时间占用连接而未释放,导致其他线程无法获得所需的连接资源[^1]。 为了防止这种情况的发生,可以采取以下几种策略: #### 1. 设置合理的超时机制 通过设置获取数据库连接的最大等待时间来避免线程无休止地等待。一旦超过设定的时间阈值,则认为当前请求失败并抛出异常给调用方处理。这不仅能够有效减少不必要的阻塞情况,同时也便于上层业务逻辑针对不同场景做出相应的响应措施。 ```java // 假设使用的是 HikariCP 连接池配置 HikariConfig config = new HikariConfig(); config.setConnectionTimeout(30_000); // 单位毫秒, 超过此时间仍未取得连接则报错 ``` #### 2. 实施限流控制 对于高并发的应用环境来说,在短时间内大量创建新线程去争夺少量可用的数据库链接显然是不理智的行为。可以通过引入信号量(Semaphore)或者其他形式的流量控制器来限制同一时刻最多允许多少个活跃的工作单元同时向数据源发起读写请求。 ```java Semaphore semaphore = new Semaphore(maxConcurrentAccesses); public void accessDatabase() throws InterruptedException { try { semaphore.acquire(); // 尝试进入临界区前先申请许可 // 执行具体的数据库操作... } finally { semaphore.release(); // 不管成功与否都要记得释放许可证 } } ``` #### 3. 使用异步非阻塞模式 如果应用程序架构支持的话,考虑采用基于回调函数或Future/Promise风格的设计思路重构原有的同步流程。这样做的好处是可以让主线程不必一直挂起直到子任务完成为止;相反它可以在后台持续运行而不影响用户体验流畅度的同时利用事件驱动的方式接收来自底层I/O模块返回的结果集。 ```kotlin fun asyncQuery(sql:String): CompletableFuture<List<Map<String, Any>>> = CompletableFuture.supplyAsync{ val connection = dataSource.getConnection() try { val statement = connection.prepareStatement(sql) val resultSet = statement.executeQuery() mutableListOf<Map<String,Any>>().apply { while (resultSet.next()){ add(mapOf( "columnA" to resultSet.getString("Column_A"), "columnB" to resultSet.getInt("Column_B") )) } } }finally {connection.close()} } ``` 以上三种方式都可以有效地缓解乃至彻底消除因为争抢共享型持久化存储接口所引发的一系列连锁反应——即所谓的“饥饿”状态或是更严重的死锁状况[^4]。
评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值