python redis连接数_redis_py_cluster读取redis遇到too many connections问题

最近项目中需要自己去往redis里面写数据，redis集群读写需要使用到redis_py_cluster这个python包，这中间遇到不少问题，其中就有数据序列化问题，有人使用spark 中的json序列化方式，与python还不兼容，最后是不序列化直接存字符串的方式解决了不同语言之间的兼容问题，但是自己在读写redis的时候遇到了too many connections的错误，字面上就是建立了太多的连接。

redis本来不就是频繁的读写操作？这个报连接太多嘛意思，一开始定位可能是redis集群版本与python包版本不匹配的问题，通过降级方式还是会出现上述现象。

下面是定义连接的方式

from rediscluster import StrictRedisCluster

redis_conn = StrictRedisCluster(startup_nodes=[{'host': '1333', 'port': 829}, {'host': '13334', 'port': 8269}],

skip_full_coverage_check=True)

注意上面只是定义连接，其实现在你还没有跟redis集群进行任何连接，当你在执行get或者set的时候才会进行连接。

StrictRedisCluster定义如下

def __init__(self, host=None, port=None, startup_nodes=None, max_connections=None, max_connections_per_node=False, init_slot_cache=True,

readonly_mode=False, reinitialize_steps=None, skip_full_coverage_check=False, nodemanager_follow_cluster=False,

connection_class=None, **kwargs):

"""

:startup_nodes:

List of nodes that initial bootstrapping can be done from

:host:

Can be used to point to a startup node

:port:

Can be used to point to a startup node

:max_connections:

Maximum number of connections that should be kept open at one time

:readonly_mode:

enable READONLY mode. You can read possibly stale data from slave.

:skip_full_coverage_check:

Skips the check of cluster-require-full-coverage config, useful for clusters

without the CONFIG command (like aws)

:nodemanager_follow_cluster:

The node manager will during initialization try the last set of nodes that

it was operating on. This will allow the client to drift along side the cluster

if the cluster nodes move around alot.

:**kwargs:

Extra arguments that will be sent into StrictRedis instance when created

(See Official redis-py doc for supported kwargs

[https://github.com/andymccurdy/redis-py/blob/master/redis/client.py])

Some kwargs is not supported and will raise RedisClusterException

- db (Redis do not support database SELECT in cluster mode)

"""

# Tweaks to StrictRedis client arguments when running in cluster mode

if "db" in kwargs:

raise RedisClusterException("Argument 'db' is not possible to use in cluster mode")

if kwargs.get('ssl', False):

connection_class = SSLClusterConnection

if "connection_pool" in kwargs:

pool = kwargs.pop('connection_pool')

else:

startup_nodes = [] if startup_nodes is None else startup_nodes

# Support host/port as argument

if host:

startup_nodes.append({"host": host, "port": port if port else 7000})

if readonly_mode:

connection_pool_cls = ClusterReadOnlyConnectionPool

else:

connection_pool_cls = ClusterConnectionPool

pool = connection_pool_cls(

startup_nodes=startup_nodes,

init_slot_cache=init_slot_cache,

max_connections=max_connections,

reinitialize_steps=reinitialize_steps,

max_connections_per_node=max_connections_per_node,

skip_full_coverage_check=skip_full_coverage_check,

nodemanager_follow_cluster=nodemanager_follow_cluster,

connection_class=connection_class,

**kwargs

)

super(StrictRedisCluster, self).__init__(connection_pool=pool, **kwargs)

其中有一个max_connections最大连接数的参数，这个如果你设置20，那么在你建立20个连接的时候会报错，这个就会出现文章一开始说的问题，这个也是自己折腾好久才发现的点。

你可以这么来定位这个问题，你是哟空上面的定义好之后执行一个get操作，在去查看这个变量

redis_conn.connection_pool._available_connections

这个可以打印当前已经建立的连接，你会发现有一条，当你继续读写数据的时候如果建立的连接超过20立马就会报错。

那么解决的方法是啥呢，我现在想的是超级简单：max_connnections参数不指定，因为你不指定，python会自己指定2**31次方

源码定义

self.max_connections = max_connections or 2 ** 31

这个为什么设置这么大也可以理解，基本上你一次get之后很快就会失去连接，redis不会保持很长的连接，所以自动就会断开，设置这么大其实对redis集群没有什么太大的影响，除非你的并发超级大，超过了redis默认最大的连接数，那么就会出现问题。