Jedis分片Sentinel连接池实验

最新推荐文章于 2024-08-27 10:03:03 发布

转载最新推荐文章于 2024-08-27 10:03:03 发布 · 605 阅读

本文介绍了一种结合Redis Sentinel与Jedis客户端的分片连接池方案，解决了同时进行Sharding与Sentinel监控的需求。文中详细描述了Sentinel配置、Jedis客户端扩展、连接泄露修复及Failover实验过程。

摘要生成于 C知道，由 DeepSeek-R1 满血版支持，前往体验 >

转：http://blog.youkuaiyun.com/dc_726/article/details/48084373

Jedis分片Sentinel连接池实验

1.起因

众所周知，Redis官方HA工具Sentinel已经问世很久了，但令人费解的是，Jedis官方却迟迟没有更新它的连接池。到目前Maven库中最新的2.7.3版本为止，都只能要么使用分片连接池，要么使用不分片的Sentinel连接池。如果既进行了Sharding，又对每组的主从实例配置Sentinel进行监控，怎么办？答案是只能自己开发了，晕！还好万能的GitHub上已经有人提供了一个简单可用的分片Sentinel连接池实现，于是就直接拿来用用。

2.Sentinel配置

我的Redis是这样配置的：一个主6379带一个从16379，名字叫mymaster-6379；另一个主6380带另一个从16380，名字叫mymaster-6380。Sentinel按照惯例，起三个实例26379,26380,26381。以其中一个为例，其他的Sentinel配置只是端口号不同：

<code class="hljs lasso has-numbering">port <span class="hljs-number">26379</span>
dir /tmp

sentinel monitor mymaster<span class="hljs-subst">-</span><span class="hljs-number">6379</span> <span class="hljs-number">192.168</span><span class="hljs-number">.241</span><span class="hljs-number">.220</span> <span class="hljs-number">6379</span> <span class="hljs-number">2</span>
sentinel down<span class="hljs-attribute">-after</span><span class="hljs-attribute">-milliseconds</span> mymaster<span class="hljs-subst">-</span><span class="hljs-number">6379</span> <span class="hljs-number">30000</span>
sentinel parallel<span class="hljs-attribute">-syncs</span> mymaster<span class="hljs-subst">-</span><span class="hljs-number">6379</span> <span class="hljs-number">1</span>
sentinel failover<span class="hljs-attribute">-timeout</span> mymaster<span class="hljs-subst">-</span><span class="hljs-number">6379</span> <span class="hljs-number">180000</span>

sentinel monitor mymaster<span class="hljs-subst">-</span><span class="hljs-number">6380</span> <span class="hljs-number">192.168</span><span class="hljs-number">.241</span><span class="hljs-number">.220</span> <span class="hljs-number">6380</span> <span class="hljs-number">2</span>
sentinel down<span class="hljs-attribute">-after</span><span class="hljs-attribute">-milliseconds</span> mymaster<span class="hljs-subst">-</span><span class="hljs-number">6380</span> <span class="hljs-number">30000</span>
sentinel parallel<span class="hljs-attribute">-syncs</span> mymaster<span class="hljs-subst">-</span><span class="hljs-number">6380</span> <span class="hljs-number">1</span>
sentinel failover<span class="hljs-attribute">-timeout</span> mymaster<span class="hljs-subst">-</span><span class="hljs-number">6380</span> <span class="hljs-number">180000</span></code>

如果想要Sentinel快速failover，就把sentinel down-after-milliseconds mymaster-6379 30000中的30秒改短点，这个时间是Sentinel Ping不到Master多久后开始failover。具体Redis主从复制和Sentinel如何配置就不赘述了，请参考我以前写的Redis主从和HA配置。

3.Jedis客户端扩展

3.1 测试代码

将GitHub那个开源的连接池引到工程里，使用起来很简单，只需通过构造方法传入所有Sentinel实例的连接地址以及master的名字列表。连接池配置可以使用默认的。

注意：一定要每次重新从pool中取一个连接，否则会一直访问老master。

<code class="language-java hljs  has-numbering">    <span class="hljs-keyword">public</span> <span class="hljs-keyword">static</span> <span class="hljs-keyword">void</span> <span class="hljs-title">main</span>(String[] args) <span class="hljs-keyword">throws</span> Exception {
        Set<String> sentinels = <span class="hljs-keyword">new</span> HashSet<String>();
        sentinels.add(<span class="hljs-string">"192.168.111.222:26379"</span>);
        sentinels.add(<span class="hljs-string">"192.168.111.222:26380"</span>);
        sentinels.add(<span class="hljs-string">"192.168.111.222:26381"</span>);

        ShardedJedisSentinelPool pool = <span class="hljs-keyword">new</span> ShardedJedisSentinelPool(
                Arrays.asList(<span class="hljs-string">"mymaster-6379"</span>, <span class="hljs-string">"mymaster-6380"</span>),
                sentinels
        );

        <span class="hljs-keyword">while</span> (<span class="hljs-keyword">true</span>) {
            String ts = <span class="hljs-keyword">new</span> SimpleDateFormat(<span class="hljs-string">"hh:mm:ss"</span>).format(<span class="hljs-keyword">new</span> Date());
            ShardedJedis jedis = pool.getResource();
            <span class="hljs-keyword">try</span> {
                System.out.println(ts + <span class="hljs-string">": hello="</span> + jedis.get(<span class="hljs-string">"hello"</span>));
            } <span class="hljs-keyword">catch</span> (Exception e) {
                System.out.println(ts + <span class="hljs-string">": Cannot connect to Redis"</span>);
            } <span class="hljs-keyword">finally</span> {
                jedis.close();
            }
            Thread.sleep(<span class="hljs-number">500</span>L);
        }
    }</code>

3.2 连接泄露Bug

循环时发现ShardedJedisSentinelPool连接池有时竟然满了！毕竟默认是8个连接，但为什么会这样？查了下代码，果然是这个原因！这个新扩展的pool没有为getResource()出来的ShardedJedis设置DataSource，所以关闭ShardedJedis时没法正常还回到连接池中。

补丁很简单，就是参照Jedis的ShardedJedisPool，覆写一些资源管理的方法。

<code class="language-java hljs  has-numbering">    <span class="hljs-annotation">@Override</span>
    <span class="hljs-keyword">public</span> ShardedJedis <span class="hljs-title">getResource</span>() {
        ShardedJedis jedis = <span class="hljs-keyword">super</span>.getResource();
        jedis.setDataSource(<span class="hljs-keyword">this</span>);
        <span class="hljs-keyword">return</span> jedis;
    }

    <span class="hljs-annotation">@Override</span>
    <span class="hljs-keyword">public</span> <span class="hljs-keyword">void</span> <span class="hljs-title">returnBrokenResource</span>(<span class="hljs-keyword">final</span> ShardedJedis resource) {
        <span class="hljs-keyword">if</span> (resource != <span class="hljs-keyword">null</span>) {
            returnBrokenResourceObject(resource);
        }
    }

    <span class="hljs-annotation">@Override</span>
    <span class="hljs-keyword">public</span> <span class="hljs-keyword">void</span> <span class="hljs-title">returnResource</span>(<span class="hljs-keyword">final</span> ShardedJedis resource) {
        <span class="hljs-keyword">if</span> (resource != <span class="hljs-keyword">null</span>) {
            resource.resetState();
            returnResourceObject(resource);
        }
    }</code>

3.3 KeyTag

此外，ShardedJedisSentinelPool的构造方法中没提供传入KeyTagPattern的地方，这也需要手动传入到ShardedJedisFactory中。需要的话，自己动手加一下吧！

4.Failover实验

以key=hello为例，事先在6380实例上准备好key的值。

4.1 实验结果

首先，启动Java程序不断访问。然后，在后台kill掉6380那个Redis实例。控制台输出会警告无法连接Redis。大概30秒左右，本地连接池注册的Sentinel监听器会收到failover消息。于是，从pool.getResource()拿到的连接自动切换到Slave实例了，又可以查询到hello对应的值了。

<code class="hljs livecodeserver has-numbering"><span class="hljs-number">10</span>:<span class="hljs-number">13</span>:<span class="hljs-number">42</span>: hello=nihaonihao111
<span class="hljs-number">10</span>:<span class="hljs-number">13</span>:<span class="hljs-number">42</span>: Cannot connect <span class="hljs-built_in">to</span> Redis
<span class="hljs-number">10</span>:<span class="hljs-number">13</span>:<span class="hljs-number">43</span>: Cannot connect <span class="hljs-built_in">to</span> Redis
<span class="hljs-number">10</span>:<span class="hljs-number">13</span>:<span class="hljs-number">45</span>: Cannot connect <span class="hljs-built_in">to</span> Redis
<span class="hljs-number">10</span>:<span class="hljs-number">13</span>:<span class="hljs-number">48</span>: Cannot connect <span class="hljs-built_in">to</span> Redis
<span class="hljs-number">10</span>:<span class="hljs-number">13</span>:<span class="hljs-number">50</span>: Cannot connect <span class="hljs-built_in">to</span> Redis
<span class="hljs-number">10</span>:<span class="hljs-number">13</span>:<span class="hljs-number">53</span>: Cannot connect <span class="hljs-built_in">to</span> Redis
<span class="hljs-number">10</span>:<span class="hljs-number">13</span>:<span class="hljs-number">56</span>: Cannot connect <span class="hljs-built_in">to</span> Redis
<span class="hljs-number">10</span>:<span class="hljs-number">13</span>:<span class="hljs-number">58</span>: Cannot connect <span class="hljs-built_in">to</span> Redis
<span class="hljs-number">10</span>:<span class="hljs-number">14</span>:<span class="hljs-number">01</span>: Cannot connect <span class="hljs-built_in">to</span> Redis
<span class="hljs-number">10</span>:<span class="hljs-number">14</span>:<span class="hljs-number">03</span>: Cannot connect <span class="hljs-built_in">to</span> Redis
<span class="hljs-number">10</span>:<span class="hljs-number">14</span>:<span class="hljs-number">06</span>: Cannot connect <span class="hljs-built_in">to</span> Redis
<span class="hljs-number">10</span>:<span class="hljs-number">14</span>:<span class="hljs-number">08</span>: Cannot connect <span class="hljs-built_in">to</span> Redis
<span class="hljs-number">10</span>:<span class="hljs-number">14</span>:<span class="hljs-number">11</span>: Cannot connect <span class="hljs-built_in">to</span> Redis
八月 <span class="hljs-number">24</span>, <span class="hljs-number">2015</span> <span class="hljs-number">10</span>:<span class="hljs-number">14</span>:<span class="hljs-number">13</span> 上午 redis.clients.jedis.ShardedJedisSentinelPool initPool
信息: Created ShardedJedisPool <span class="hljs-built_in">to</span> master <span class="hljs-keyword">at</span> [<span class="hljs-number">192.168</span><span class="hljs-number">.111</span><span class="hljs-number">.222</span>:<span class="hljs-number">6379</span> <span class="hljs-number">192.168</span><span class="hljs-number">.111</span><span class="hljs-number">.222</span>:<span class="hljs-number">16380</span> ]
<span class="hljs-number">10</span>:<span class="hljs-number">14</span>:<span class="hljs-number">13</span>: hello=nihaonihao111
<span class="hljs-number">10</span>:<span class="hljs-number">14</span>:<span class="hljs-number">14</span>: hello=nihaonihao111
<span class="hljs-number">10</span>:<span class="hljs-number">14</span>:<span class="hljs-number">14</span>: hello=nihaonihao111</code>

4.2 如何恢复

如果想反复试验怎么办？很简单，重新启动6380实例，但此时16380已经变成了Master，6380成了它的Slave。没关系！随便连接到一个Sentinel实例上，执行sentinel failover mymaster-6380就可以强制failover一次。这样6380和16380的主从关系就又恢复了！