cassandra节点down机(java.lang.OutOfMemoryError: unable to create new native thread)

本文针对在CentOS 6 64位系统上进行压力测试时遇到的Cassandra节点宕机问题进行了详细分析。问题根源在于Linux系统的maxuserprocesses(nproc)限制导致内存溢出。文中提供了具体的错误信息,并给出了临时解决方案和永久修复方法,包括调整当前会话的nproc限制和修改全局配置文件。

摘要生成于 C知道 ,由 DeepSeek-R1 满血版支持, 前往体验 >

在对集群做压力测试的时候,发现有节点down机,错误信息如下。google后查明原因,由于Linux " max user processes(nproc)"所致,我操作系统的是 CentOS 6 64bit,修改方法如下:
错误信息
ERROR [Thread-28] 2013-07-25 06:14:05,055 CassandraDaemon.java (line 175) Exception in thread Thread[Thread-28,5,main]
java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:640)
at java.util.concurrent.ThreadPoolExecutor.addThread(ThreadPoolExecutor.java:681)
at java.util.concurrent.ThreadPoolExecutor.addIfUnderMaximumPoolSize(ThreadPoolExecutor.java:727)
at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:655)
at org.apache.cassandra.thrift.CustomTThreadPoolServer.serve(CustomTThreadPoolServer.java:113)
at org.apache.cassandra.thrift.ThriftServer$ThriftServerThread.run(ThriftServer.java:111)

 

当前会话有效设置
ulimit -u # 查看nproc
ulimit -u 65535 # 设置nproc,仅当前会话有效

  

全局有效
cat /etc/security/limits.d/90-nproc.conf
* soft nproc 1024

vi /etc/security/limits.d/90-nproc.conf
* soft nproc 65535

 

参考:

java.lang.OutOfMemoryError: unable to create new native thread

关于RHEL6中ulimit的nproc限制

ulimit限制之nproc问题

 

  --end

 

 

 

1:26.963358059+08:00 stdout F java.lang.IllegalAccessException: access to public member failed: jdk.internal.ref.Cleaner.clean[Ljava.lang.Object;@71e693fa/invokeVirtual, from org.apache.cassandra.io.util.FileUtils (unnamed module @63d4e2ba) 2025-07-03T14:41:26.963358059+08:00 stdout F at java.base/java.lang.invoke.MemberName.makeAccessException(Unknown Source) 2025-07-03T14:41:26.963358059+08:00 stdout F at java.base/java.lang.invoke.MethodHandles$Lookup.checkAccess(Unknown Source) 2025-07-03T14:41:26.963358059+08:00 stdout F at java.base/java.lang.invoke.MethodHandles$Lookup.checkMethod(Unknown Source) 2025-07-03T14:41:26.963358059+08:00 stdout F at java.base/java.lang.invoke.MethodHandles$Lookup.getDirectMethodCommon(Unknown Source) 2025-07-03T14:41:26.963358059+08:00 stdout F at java.base/java.lang.invoke.MethodHandles$Lookup.getDirectMethodNoSecurityManager(Unknown Source) 2025-07-03T14:41:26.963358059+08:00 stdout F at java.base/java.lang.invoke.MethodHandles$Lookup.unreflect(Unknown Source) 2025-07-03T14:41:26.963358059+08:00 stdout F at org.apache.cassandra.io.util.FileUtils.<clinit>(FileUtils.java:99) 2025-07-03T14:41:26.963358059+08:00 stdout F at org.apache.cassandra.config.DatabaseDescriptor.guessFileStore(DatabaseDescriptor.java:1208) 2025-07-03T14:41:26.963358059+08:00 stdout F at org.apache.cassandra.config.DatabaseDescriptor.applySimpleConfig(DatabaseDescriptor.java:550) 2025-07-03T14:41:26.963358059+08:00 stdout F at org.apache.cassandra.config.DatabaseDescriptor.applyAll(DatabaseDescriptor.java:350) 2025-07-03T14:41:26.963358059+08:00 stdout F at org.apache.cassandra.config.DatabaseDescriptor.daemonInitialization(DatabaseDescriptor.java:178) 2025-07-03T14:41:26.963358059+08:00 stdout F at org.apache.cassandra.config.DatabaseDescriptor.daemonInitialization(DatabaseDescriptor.java:162) 2025-07-03T14:41:26.963358059+08:00 stdout F at org.apache.cassandra.service.CassandraDaemon.applyConfig(CassandraDaemon.java:812) 2025-07-03T14:41:26.963358059+08:00 stdout F at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:748) 2025-07-03T14:41:26.963358059+08:00 stdout F at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:893) 2025-07-03T14:41:26.963736562+08:00 stdout P Exception (java.lang.ExceptionInInitializerError) encountered during startup: null 2025-07-03T14:41:26.963776026+08:00 stdout F 2025-07-03T14:41:26.963961182+08:00 stderr P java.lang.ExceptionInInitializerError 2025-07-03T14:41:26.963973216+08:00 stderr F 2025-07-03T14:41:26.964038486+08:00 stderr P at org.apache.cassandra.config.DatabaseDescriptor.guessFileStore(DatabaseDescriptor.java:1208) 2025-07-03T14:41:26.964051773+08:00 stderr F 2025-07-03T14:41:26.964092671+08:00 stderr P at org.apache.cassandra.config.DatabaseDescriptor.applySimpleConfig(DatabaseDescriptor.java:550) 2025-07-03T14:41:26.964104133+08:00 stderr F 2025-07-03T14:41:26.964123091+08:00 stderr P at org.apache.cassandra.config.DatabaseDescriptor.applyAll(DatabaseDescriptor.java:350) 2025-07-03T14:41:26.964132994+08:00 stderr F 2025-07-03T14:41:26.964464325+08:00 stderr F at org.apache.cassandra.config.DatabaseDescriptor.daemonInitialization(DatabaseDescriptor.java:178) 2025-07-03T14:41:26.964464325+08:00 stderr F at org.apache.cassandra.config.DatabaseDescriptor.daemonInitialization(DatabaseDescriptor.java:162) 2025-07-03T14:41:26.964464325+08:00 stderr F at org.apache.cassandra.service.CassandraDaemon.applyConfig(CassandraDaemon.java:812) 2025-07-03T14:41:26.964464325+08:00 stderr F at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:748) 2025-07-03T14:41:26.964464325+08:00 stderr F at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:893) 2025-07-03T14:41:26.964464325+08:00 stderr P Caused by: java.lang.RuntimeException: java.lang.IllegalAccessException: access to public member failed: jdk.internal.ref.Cleaner.clean[Ljava.lang.Object;@71e693fa/invokeVirtual, from org.apache.cassandra.io.util.FileUtils (unnamed module @63d4e2ba) 2025-07-03T14:41:26.964487780+08:00 stderr F 2025-07-03T14:41:26.964497232+08:00 stderr P at org.apache.cassandra.io.util.FileUtils.<clinit>(FileUtils.java:110) 2025-07-03T14:41:26.964526594+08:00 stderr F 2025-07-03T14:41:26.964574438+08:00 stderr F ... 8 more 2025-07-03T14:41:26.964671211+08:00 stderr P Caused by: java.lang.IllegalAccessException: access to public member failed: jdk.internal.ref.Cleaner.clean[Ljava.lang.Object;@71e693fa/invokeVirtual, from org.apache.cassandra.io.util.FileUtils (unnamed module @63d4e2ba) 2025-07-03T14:41:26.964690467+08:00 stderr F 2025-07-03T14:41:26.964738798+08:00 stderr P at java.base/java.lang.invoke.MemberName.makeAccessException(Unknown Source) 2025-07-03T14:41:26.964769627+08:00 stderr F 2025-07-03T14:41:26.964824546+08:00 stderr P at java.base/java.lang.invoke.MethodHandles$Lookup.checkAccess(Unknown Source) 2025-07-03T14:41:26.964854586+08:00 stderr F 2025-07-03T14:41:26.964906265+08:00 stderr P at java.base/java.lang.invoke.MethodHandles$Lookup.checkMethod(Unknown Source) 2025-07-03T14:41:26.964936031+08:00 stderr F 2025-07-03T14:41:26.964977123+08:00 stderr P at java.base/java.lang.invoke.MethodHandles$Lookup.getDirectMethodCommon(Unknown Source) 2025-07-03T14:41:26.965006734+08:00 stderr F 2025-07-03T14:41:26.965057652+08:00 stderr P at java.base/java.lang.invoke.MethodHandles$Lookup.getDirectMethodNoSecurityManager(Unknown Source) 2025-07-03T14:41:26.965070428+08:00 stderr F 2025-07-03T14:41:26.965114278+08:00 stderr P at java.base/java.lang.invoke.MethodHandles$Lookup.unreflect(Unknown Source) 2025-07-03T14:41:26.965127911+08:00 stderr F 2025-07-03T14:41:26.965173685+08:00 stderr P at org.apache.cassandra.io.util.FileUtils.<clinit>(FileUtils.java:99) 2025-07-03T14:41:26.965187052+08:00 stderr F 2025-07-03T14:41:26.965223074+08:00 stderr P ... 8 more 2025-07-03T14:41:26.965258697+08:00 stderr F 2025-07-03T14:41:26.965954076+08:00 stdout F 06:41:26.965 [main] ERROR org.apache.cassandra.service.CassandraDaemon - Exception encountered during startup 2025-07-03T14:41:26.965954076+08:00 stdout F java.lang.ExceptionInInitializerError: null 2025-07-03T14:41:26.965954076+08:00 stdout F at org.apache.cassandra.config.DatabaseDescriptor.guessFileStore(DatabaseDescriptor.java:1208) 2025-07-03T14:41:26.965954076+08:00 stdout F at org.apache.cassandra.config.DatabaseDescriptor.applySimpleConfig(DatabaseDescriptor.java:550) 2025-07-03T14:41:26.965954076+08:00 stdout F at org.apache.cassandra.config.DatabaseDescriptor.applyAll(DatabaseDescriptor.java:350) 2025-07-03T14:41:26.965954076+08:00 stdout F at org.apache.cassandra.config.DatabaseDescriptor.daemonInitialization(DatabaseDescriptor.java:178) 2025-07-03T14:41:26.965954076+08:00 stdout F at org.apache.cassandra.config.DatabaseDescriptor.daemonInitialization(DatabaseDescriptor.java:162) 2025-07-03T14:41:26.965954076+08:00 stdout F at org.apache.cassandra.service.CassandraDaemon.applyConfig(CassandraDaemon.java:812) 2025-07-03T14:41:26.965954076+08:00 stdout F at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:748) 2025-07-03T14:41:26.965954076+08:00 stdout F at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:893) 2025-07-03T14:41:26.965954076+08:00 stdout F Caused by: java.lang.RuntimeException: java.lang.IllegalAccessException: access to public member failed: jdk.internal.ref.Cleaner.clean[Ljava.lang.Object;@71e693fa/invokeVirtual, from org.apache.cassandra.io.util.FileUtils (unnamed module @63d4e2ba) 2025-07-03T14:41:26.965954076+08:00 stdout F at org.apache.cassandra.io.util.FileUtils.<clinit>(FileUtils.java:110) 2025-07-03T14:41:26.965954076+08:00 stdout F ... 8 common frames omitted 2025-07-03T14:41:26.965954076+08:00 stdout F Caused by: java.lang.IllegalAccessException: access to public member failed: jdk.internal.ref.Cleaner.clean[Ljava.lang.Object;@71e693fa/invokeVirtual, from org.apache.cassandra.io.util.FileUtils (unnamed module @63d4e2ba) 2025-07-03T14:41:26.965954076+08:00 stdout F at java.base/java.lang.invoke.MemberName.makeAccessException(Unknown Source) 2025-07-03T14:41:26.965954076+08:00 stdout F at java.base/java.lang.invoke.MethodHandles$Lookup.checkAccess(Unknown Source) 2025-07-03T14:41:26.965954076+08:00 stdout F at java.base/java.lang.invoke.MethodHandles$Lookup.checkMethod(Unknown Source) 2025-07-03T14:41:26.965954076+08:00 stdout F at java.base/java.lang.invoke.MethodHandles$Lookup.getDirectMethodCommon(Unknown Source) 2025-07-03T14:41:26.965954076+08:00 stdout F at java.base/java.lang.invoke.MethodHandles$Lookup.getDirectMethodNoSecurityManager(Unknown Source) 2025-07-03T14:41:26.965954076+08:00 stdout F at java.base/java.lang.invoke.MethodHandles$Lookup.unreflect(Unknown Source) 2025-07-03T14:41:26.965954076+08:00 stdout F at org.apache.cassandra.io.util.FileUtils.<clinit>(FileUtils.java:99) 2025-07-03T14:41:26.965954076+08:00 stdout F ... 8 common frames omitted 这又是什么错
最新发布
07-04
评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值