批量入库Kerbos认证的HBase，入库越来越慢直到程序卡死不动，后台一直告警，求解决！！！

本文链接：https://blog.youkuaiyun.com/Luminitai/article/details/107655646

程序提交至spark运行，一开始运行正常随后越来越慢直到一直卡死，有时报错
java.io.IOException: Could not set up IO Streams to hostname/ip:60020
但总是后台一直告警：告警如下
INFO AsyncProcess: #16, table=xxxxxxxxxx, attempt=20/35 failed=24ops, last exception: org.apache.hadoop.hbase.exceptions.ConnectionClosingException: This connection is closing on hostname,60020,1588212757580, tracking started null, retrying after=20128ms, replay=24ops
  INFO AsyncProcess: #16, table=xxxxxxxxxx, attempt=20/35 failed=12ops, last exception: java.io.IOException: Couldn't setup connection for xxxxr@COM to hbase/hostname@COM on hostname,60020,1588212757580, tracking started null, retrying after=20129ms, replay=12ops
  INFO AsyncProcess: #16, table=xxxxxxxxxx, attempt=20/35 failed=27ops, last exception: org.apache.hadoop.hbase.exceptions.ConnectionClosingException: This connection is closing on hostname,60020,1588212757580, tracking started null, retrying after=20024ms, replay=27ops
  INFO AsyncProcess: #16, table=xxxxxxxxxx, attempt=20/35 failed=21ops, last exception: org.apache.hadoop.hbase.exceptions.ConnectionClosingException: This connection is closing on hostname,60020,1588212757580, tracking started null, retrying after=20152ms, replay=21ops
  INFO AsyncProcess: #16, table=xxxxxxxxxx, attempt=20/35 failed=26ops, last exception: org.apache.hadoop.hbase.exceptions.ConnectionClosingException: This connection is closing on hostname,60020,1588212757580, tracking started null, retrying after=20142ms, replay=26ops
  INFO AsyncProcess: #16, table=xxxxxxxxxx, attempt=20/35 failed=31ops, last exception: org.apache.hadoop.hbase.exceptions.ConnectionClosingException: This connection is closing on hostname,60020,1588212757580, tracking started null, retrying after=20161ms, replay=31ops
  INFO AsyncProcess: #16, table=xxxxxxxxxx, attempt=20/35 failed=27ops, last exception: org.apache.hadoop.hbase.exceptions.ConnectionClosingException: This connection is closing on hostname,60020,1588212757580, tracking started null, retrying after=20025ms, replay=27ops
  INFO AsyncProcess: #16, table=xxxxxxxxxx, attempt=20/35 failed=22ops, last exception: org.apache.hadoop.hbase.exceptions.ConnectionClosingException: This connection is closing on hostname,60020,1588212757580, tracking started null, retrying after=20171ms, replay=22ops
  INFO AsyncProcess: #16, table=xxxxxxxxxx, attempt=20/35 failed=18ops, last exception: org.apache.hadoop.hbase.exceptions.ConnectionClosingException: This connection is closing on hostname,60020,1588212757580, tracking started null, retrying after=20094ms, replay=18ops
  INFO AsyncProcess: #16, table=xxxxxxxxxx, attempt=20/35 failed=19ops, last exception: org.apache.hadoop.hbase.exceptions.ConnectionClosingException: This connection is closing on hostname,60020,1588212757580, tracking started null, retrying after=20009ms, replay=19ops
  INFO AsyncProcess: #16, table=xxxxxxxxxx, attempt=20/35 failed=25ops, last exception: org.apache.hadoop.hbase.exceptions.ConnectionClosingException: This connection is closing on hostname,60020,1588212757580, tracking started null, retrying after=20156ms, replay=25ops
  INFO AsyncProcess: #16, table=xxxxxxxxxx, attempt=20/35 failed=22ops, last exception: org.apache.hadoop.hbase.exceptions.ConnectionClosingException: This connection is closing on hostname,60020,1588212757580, tracking started null, retrying after=20019ms, replay=22ops
  INFO AsyncProcess: #16, table=xxxxxxxxxx, attempt=20/35 failed=18ops, last exception: org.apache.hadoop.hbase.exceptions.ConnectionClosingException: This connection is closing on hostname,60020,1588212757580, tracking started null, retrying after=20127ms, replay=18ops
  INFO AsyncProcess: #16, table=xxxxxxxxxx, attempt=20/35 failed=21ops, last exception: org.apache.hadoop.hbase.exceptions.ConnectionClosingException: This connection is closing on hostname,60020,1588212757580, tracking started null, retrying after=20188ms, replay=21ops
  INFO AsyncProcess: #16, table=xxxxxxxxxx, attempt=20/35 failed=32ops, last exception: org.apache.hadoop.hbase.exceptions.ConnectionClosingException: This connection is closing on hostname,60020,1588212757580, tracking started null, retrying after=20074ms, replay=32ops
  INFO AsyncProcess: #16, table=xxxxxxxxxx, attempt=20/35 failed=22ops, last exception: org.apache.hadoop.hbase.exceptions.ConnectionClosingException: This connection is closing on hostname,60020,1588212757580, tracking started null, retrying after=20118ms, replay=22ops
  INFO AsyncProcess: #16, table=xxxxxxxxxx, attempt=20/35 failed=26ops, last exception: org.apache.hadoop.hbase.exceptions.ConnectionClosingException: This connection is closing on hostname,60020,1588212757580, tracking started null, retrying after=20085ms, replay=26ops
  INFO AsyncProcess: #16, table=xxxxxxxxxx, attempt=20/35 failed=22ops, last exception: org.apache.hadoop.hbase.exceptions.ConnectionClosingException: This connection is closing on hostname,60020,1588212757580, tracking started null, retrying after=20003ms, replay=22ops
  INFO AsyncProcess: #16, table=xxxxxxxxxx, attempt=20/35 failed=21ops, last exception: org.apache.hadoop.hbase.exceptions.ConnectionClosingException: This connection is closing on hostname,60020,1588212757580, tracking started null, retrying after=20196ms, replay=21ops
  INFO AsyncProcess: #16, table=xxxxxxxxxx, attempt=20/35 failed=24ops, last exception: org.apache.hadoop.hbase.exceptions.ConnectionClosingException: This connection is closing on hostname,60020,1588212757580, tracking started null, retrying after=20044ms, replay=24ops
  INFO AsyncProcess: #16, table=xxxxxxxxxx, attempt=20/35 failed=28ops, last exception: org.apache.hadoop.hbase.exceptions.ConnectionClosingException: This connection is closing on hostname,60020,1588212757580, tracking started null, retrying after=20073ms, replay=28ops
  INFO AsyncProcess: #16, table=xxxxxxxxxx, attempt=20/35 failed=24ops, last exception: org.apache.hadoop.hbase.exceptions.ConnectionClosingException: This connection is closing on hostname,60020,1588212757580, tracking started null, retrying after=20097ms, replay=24ops
20/07/29 09:05:21 WARN UserGroupInformation: PriviledgedActionException as:xxxxr@COM(auth:KERBEROS) cause:org.apache.hadoop.ipc.RemoteException(javax.security.sasl.SaslException): GSS initiate failed
20/07/29 09:05:21 WARN UserGroupInformation: PriviledgedActionException as:xxxxr@COM(auth:KERBEROS) cause:javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Server not found in Kerberos database (7) - UNKNOWN_SERVER)]
频繁出现，最后直到报错