微服务+rabbitmq+mysql 并发下遇到的问题

最新推荐文章于 2024-07-15 22:05:24 发布

原创最新推荐文章于 2024-07-15 22:05:24 发布 · 965 阅读

0 ·

CC 4.0 BY-SA版权

技术专栏收录该内容

15 篇文章

订阅专栏

本文探讨了微服务架构下负载均衡与熔断机制的配置，解决了服务注册失败与负载过大的问题。同时，针对MySQL性能瓶颈，提出了mycat分库方案，并解决了MQ消息堆积问题。最后，分享了微服务集群的硬件需求与优化经验。

一、服务撑不住

在做物联网模块的时候，用到微服务，200个设备5分钟向物联网模块发送数据一次数据，注册中心是单例模式，所有的微服务都开了两个服务，网关会报错，负载过大，服务请求的时候也会报错，提示连接注册中心拒绝，负载过大，连接不到注册中心、服务会失效。

com.sun.jersey.api.client.ClientHandlerException: org.apache.http.NoHttpResponseException: 127.0.0.1:0000failed to respond
   at com.sun.jersey.client.apache4.ApacheHttpClient4Handler.handle(ApacheHttpClient4Handler.java:187)
   at com.sun.jersey.api.client.filter.GZIPContentEncodingFilter.handle(GZIPContentEncodingFilter.java:123)
   at com.netflix.discovery.EurekaIdentityHeaderFilter.handle(EurekaIdentityHeaderFilter.java:27)
   at com.sun.jersey.api.client.Client.handle(Client.java:652)
   at com.sun.jersey.api.client.WebResource.handle(WebResource.java:682)
   at com.sun.jersey.api.client.WebResource.access$200(WebResource.java:74)
   at com.sun.jersey.api.client.WebResource$Builder.put(WebResource.java:529)
   at com.netflix.discovery.shared.transport.jersey.AbstractJerseyEurekaHttpClient.sendHeartBeat(AbstractJerseyEurekaHttpClient.java:102)
   at com.netflix.discovery.shared.transport.decorator.EurekaHttpClientDecorator$3.execute(EurekaHttpClientDecorator.java:92)
   at com.netflix.discovery.shared.transport.decorator.MetricsCollectingEurekaHttpClient.execute(MetricsCollectingEurekaHttpClient.java:73)
   at com.netflix.discovery.shared.transport.decorator.EurekaHttpClientDecorator.sendHeartBeat(EurekaHttpClientDecorator.java:89)
   at com.netflix.discovery.shared.transport.decorator.EurekaHttpClientDecorator$3.execute(EurekaHttpClientDecorator.java:92)
   at com.netflix.discovery.shared.transport.decorator.RedirectingEurekaHttpClient.execute(RedirectingEurekaHttpClient.java:89)
   at com.netflix.discovery.shared.transport.decorator.EurekaHttpClientDecorator.sendHeartBeat(EurekaHttpClientDecorator.java:89)
   at com.netflix.discovery.shared.transport.decorator.EurekaHttpClientDecorator$3.execute(EurekaHttpClientDecorator.java:92)
   at com.netflix.discovery.shared.transport.decorator.RetryableEurekaHttpClient.execute(RetryableEurekaHttpClient.java:120)
   at com.netflix.discovery.shared.transport.decorator.EurekaHttpClientDecorator.sendHeartBeat(EurekaHttpClientDecorator.java:89)
   at com.netflix.discovery.shared.transport.decorator.EurekaHttpClientDecorator$3.execute(EurekaHttpClientDecorator.java:92)
   at com.netflix.discovery.shared.transport.decorator.SessionedEurekaHttpClient.execute(SessionedEurekaHttpClient.java:77)
   at com.netflix.discovery.shared.transport.decorator.EurekaHttpClientDecorator.sendHeartBeat(EurekaHttpClientDecorator.java:89)
   at com.netflix.discovery.DiscoveryClient.renew(DiscoveryClient.java:846)
   at com.netflix.discovery.DiscoveryClient$HeartbeatThread.run(DiscoveryClient.java:1399)
   at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
   at java.util.concurrent.FutureTask.run(FutureTask.java:266)
   at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
   at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.http.NoHttpResponseException: 127.0.0.1:0000failed to respond
   at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:141)
   at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56)
   at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259)
   at org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:286)
   at org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:257)
   at org.apache.http.impl.conn.AbstractClientConnAdapter.receiveResponseHeader(AbstractClientConnAdapter.java:230)
   at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273)
   at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125)
   at org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:684)
   at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:486)
   at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:835)
   at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:118)
   at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
   at com.sun.jersey.client.apache4.ApacheHttpClient4Handler.handle(ApacheHttpClient4Handler.java:173)
   ... 26 common frames omitted

意思就是当服务器端由于负载过大等情况发生时，可能会导致在收到请求后无法处理(比如没有足够的线程资源)，会直接丢弃链接而不进行处理。此时客户端就回报错：NoHttpResponseException。
官方建议出现这种情况时，可以选择重试。但是重试一定要限制重试次数，避免雪崩。

at com.netflix.loadbalancer.LoadBalancerContext.getServerFromLoadBalancer(LoadBalancerContext.java:483)
   at com.netflix.loadbalancer.reactive.LoadBalancerCommand$1.call(LoadBalancerCommand.java:184)
   at com.netflix.loadbalancer.reactive.LoadBalancerCommand$1.call(LoadBalancerCommand.java:180)
   at rx.Observable.unsafeSubscribe(Observable.java:10327)
   at rx.internal.operators.OnSubscribeConcatMap.call(OnSubscribeConcatMap.java:94)
   at rx.internal.operators.OnSubscribeConcatMap.call(OnSubscribeConcatMap.java:42)
   at rx.Observable.unsafeSubscribe(Observable.java:10327)
   at rx.internal.operators.OperatorRetryWithPredicate$SourceSubscriber$1.call(OperatorRetryWithPredicate.java:127)
   at rx.internal.schedulers.TrampolineScheduler$InnerCurrentThreadScheduler.enqueue(TrampolineScheduler.java:73)
   at rx.internal.schedulers.TrampolineScheduler$InnerCurrentThreadScheduler.schedule(TrampolineScheduler.java:52)
   at rx.internal.operators.OperatorRetryWithPredicate$SourceSubscriber.onNext(OperatorRetryWithPredicate.java:79)
   at rx.internal.operators.OperatorRetryWithPredicate$SourceSubscriber.onNext(OperatorRetryWithPredicate.java:45)
   at rx.internal.util.ScalarSynchronousObservable$WeakSingleProducer.request(ScalarSynchronousObservable.java:276)
   at rx.Subscriber.setProducer(Subscriber.java:209)
   at rx.internal.util.ScalarSynchronousObservable$JustOnSubscribe.call(ScalarSynchronousObservable.java:138)
   at rx.internal.util.ScalarSynchronousObservable$JustOnSubscribe.call(ScalarSynchronousObservable.java:129)
   at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:48)
   at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30)
   at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:48)
   at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30)
   at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:48)
   at rx.internal.operators.OnSubscribeLift.call(OnSubscribeLift.java:30)
   at rx.Observable.subscribe(Observable.java:10423)
   at rx.Observable.subscribe(Observable.java:10390)
   at rx.observables.BlockingObservable.blockForSingle(BlockingObservable.java:443)
   at rx.observables.BlockingObservable.single(BlockingObservable.java:340)
   at com.netflix.client.AbstractLoadBalancerAwareClient.executeWithLoadBalancer(AbstractLoadBalancerAwareClient.java:112)
   at org.springframework.cloud.openfeign.ribbon.LoadBalancerFeignClient.execute(LoadBalancerFeignClient.java:63)

二、mysql 性能瓶颈

单表数据表数据量大概达到125000条的时候，mysql 性能达到瓶颈，读写IO达到百分之九十九到一百，写入极慢，导致数据堆在mq 里几万条。

三、mq 消费不及时导致消息堆积

因为mysql 性能的问题，出现了mq 消息堆积问题，如果消费速度小于消息产生速度，不断产生消息堆积，该如何解决？

四，总结

注册拒绝、负载过大的情况我的解决方案是搭建微服务集群，通过开多个服务来负载均衡，数据库瓶颈的问题采用mycat分库，做一个负载均衡，缓解数据库压力，mq消息堆积问题采用mq 集群来解决。此时会产生新的问题，微服务集群对服务器性能要求，开多个服务后，因为耗内存，会出现服务起不起来的情况，8核32G服务器，同时开15个服务，包含redis、mq、mysql等应用，会有一两个起不起来，硬件方面需要一定的配置。内存加到64G后，同时开17个服务，目前没有问题。mycat分四个库，根据分片字段存储查询，mysql读写性能还好。

五、其他

ribbon+hystrix实现负载均衡和熔断的时候，ribbon的超时时间应该小于hystrix超时时间，如果大于，那么ribbon的负载将失去意义，hystrix的超时时间一到就会立马熔断，ribbon将无法重连，配置如下：

hystrix:
  command:
    default:
      execution:
        isolation:
          strategy: SEMAPHORE
          thread:
            timeoutInMilliseconds: 130000
          semaphore:
            maxConcurrentRequests: 1000 
#        circuitBreaker:
#          sleepWindowInMilliseconds:  5000
ribbon:
  ReadTimeout: 30000
  ConnectTimeout: 30000
  MaxAutoRetries: 1
  MaxAutoRetriesNextServer: 1

ribbon、hystrix、zuul 超时时间详细配置及计算后期整理再发。。。