Memcached的简介、原理和安装说明

最新推荐文章于 2022-02-08 10:11:08 发布

原创最新推荐文章于 2022-02-08 10:11:08 发布 · 935 阅读

0 ·

CC 4.0 BY-SA版权

文章标签：

#memcached #java #缓存

java 专栏收录该内容

35 篇文章

订阅专栏

本文深入解析 Memcached 缓存系统的工作原理、安装与配置方法，包括其核心特性和集群实现，同时提供了 Java 客户端集群的示例代码，以及与 Redis 的比较分析。

1、Memcached的简介

Memcached 是一个免费、开源、高性能的分布式内存对象缓存系统，用于动态Web应用以减轻数据库负载。它通过在内存中缓存数据和对象来减少读取数据库的次数，从而提高动态、数据库驱动网站的速度。Memcached基于一个存储键/值对的hashmap。其守护进程（daemon ）是用C写的，但是客户端可以用任何语言来编写，并通过memcached协议与守护进程通信。

1.1 优点

（1）基于内存的缓存，速度快。

（2）基于客户端的集群，算法简单，节点之间不需要相互通信。
1.2 缺点

（1）数据格式单一，只支持键/值对，不支持复杂数据类型，比如集合

（2）不支持模糊查询

（3）不支持排序；

（4）不支持数据持久化，数据只存在内存中，系统重启时，数据会丢失。

（5）单个键值最多只支持1M，单个节点的容量最多支持2G

（6）内存分配是基于固定的数据块，会浪费一定的内存空间。

（7）Memcached集群节点之间无法相互通信。
1.3 Memcached的用户

LiveJournal Wikipedia Flickr Bebo Twitter Typepad Yellowbot Youtube WordPress.com Craigslist Mixi

2、Memcached的原理

转自《Memcached 原理和使用详解》作者：heiyeluren（黑夜路人）

Memcached的主要特点

（1）基于C/S架构，协议简单

（2）基于libevent的事件处理

（3）自主内存存储处理

数据存储方式：Slab Allocation

Slab Allocator的基本原理是按照预先规定的大小，将分配的内存分割成特定长度的块(chunk)，并把尺寸相同的块分成组(chunk的集合)。

Page：分配给Slab的内存空间，默认是1MB。分配给Slab之后根据slab的大小切分成chunk。

Chunk:用于缓存记录的内存空间。

Slab Class：特定大小的chunk的组

数据过期方式：Lazy Expiration + LRU

1. Lazy Expiration

内部不会监视记录是否过期，而是在get时查看记录的时间戳，检查记录是否过期。

2．Memcached会优先使用已超时的记录的内存空间，当Memcached的内存空间不足时，就从最近最少使用的记录中查找，并将其空间分配给新的记录。

（1）基于客户端的Memcached分布式

Libevent是一套跨平台的事件处理接口的封装，能够兼容包括这些操作系统，Windows/Linux/BSD/Solaris等操作系统的事件处理。

包装的接口包括：

poll、select(Windows)、epoll(Linux)、kqueue(BSD)、/dev/pool(Solaris)

Memcached使用libevent来进行网络并发连接的处理，能够保持在很大并发情况下，仍旧能够保持快速的响应能力。

3、Memcached的安装说明

3.1 安装libevent

3.1.1 下载libevent

https://github.com/downloads/libevent/libevent/libevent-1.4.14b-stable.tar.gz

3.1.2 安装libevent

[root@bogon softwares]# tar -zxvf libevent-1.4.14b-stable.tar.gz
[root@bogon softwares]# cd libevent-1.4.14b-stable
[root@bogon libevent-1.4.14b-stable]# ./configure --prefix=/usr
[root@bogon libevent-1.4.14b-stable]# make
[root@bogon libevent-1.4.14b-stable]# make install

3.2 下载Memcached

3.2.1 下载Memcached

http://www.memcached.org/files/memcached-1.4.17.tar.gz

3.2.2 安装Memcached

[root@bogon softwares]# tar -zxvf memcached-1.4.17.tar.gz
[root@bogon memcached-1.4.17]# cd ..
[root@bogon softwares]# cd memcached-1.4.17
[root@bogon memcached-1.4.17]# ./configure --prefix=/usr/local
[root@bogon memcached-1.4.17]# make
[root@bogon memcached-1.4.17]# make install
[root@bogon bin]# cd /usr/local/bin
[root@bogon bin]# ls
memcached

3.3 查看帮助信息

[root@bogon bin]# ./memcached -h
memcached 1.4.17
-p <num>      TCP port number to listen on (default: 11211)
-U <num>      UDP port number to listen on (default: 11211, 0 is off)
-s <file>     UNIX socket path to listen on (disables network support)
-A            enable ascii "shutdown" command
-a <mask>     access mask for UNIX socket, in octal (default: 0700)
-l <addr>     interface to listen on (default: INADDR_ANY, all addresses)
              <addr> may be specified as host:port. If you don't specify
              a port number, the value you specified with -p or -U is
              used. You may specify multiple addresses separated by comma
              or by using -l multiple times
-d            run as a daemon
-r            maximize core file limit
-u <username> assume identity of <username> (only when run as root)
-m <num>      max memory to use for items in megabytes (default: 64 MB)
-M            return error on memory exhausted (rather than removing items)
-c <num>      max simultaneous connections (default: 1024)
-k            lock down all paged memory.  Note that there is a
              limit on how much memory you may lock.  Trying to
              allocate more than that would fail, so be sure you
              set the limit correctly for the user you started
              the daemon with (not for -u <username> user;
              under sh this is done with 'ulimit -S -l NUM_KB').
-v            verbose (print errors/warnings while in event loop)
-vv           very verbose (also print client commands/reponses)
-vvv          extremely verbose (also print internal state transitions)
-h            print this help and exit
-i            print memcached and libevent license
-P <file>     save PID in <file>, only used with -d option
-f <factor>   chunk size growth factor (default: 1.25)
-n <bytes>    minimum space allocated for key+value+flags (default: 48)
-L            Try to use large memory pages (if available). Increasing
              the memory page size could reduce the number of TLB misses
              and improve the performance. In order to get large pages
              from the OS, memcached will allocate the total item-cache
              in one large chunk.
-D <char>     Use <char> as the delimiter between key prefixes and IDs.
              This is used for per-prefix stats reporting. The default is
              ":" (colon). If this option is specified, stats collection
              is turned on automatically; if not, then it may be turned on
              by sending the "stats detail on" command to the server.
-t <num>      number of threads to use (default: 4)
-R            Maximum number of requests per event, limits the number of
              requests process for a given connection to prevent 
              starvation (default: 20)
-C            Disable use of CAS
-b            Set the backlog queue limit (default: 1024)
-B            Binding protocol - one of ascii, binary, or auto (default)
-I            Override the size of each slab page. Adjusts max item size
              (default: 1mb, min: 1k, max: 128m)
-F            Disable flush_all command
-o            Comma separated list of extended or experimental options
              - (EXPERIMENTAL) maxconns_fast: immediately close new
                connections if over maxconns limit
              - hashpower: An integer multiplier for how large the hash
                table should be. Can be grown at runtime if not big enough.
                Set this based on "STAT hash_power_level" before a 
                restart.
              - tail_repair_time: Time in seconds that indicates how long to wait before
                forcefully taking over the LRU tail item whose refcount has leaked.
                The default is 3 hours.

3.4 启动memcached

使用11211端口、root用户、最大占用512M内存、1024软连接，输出客户端请求，以守护进程方式运行。

[root@bogon bin]# ./memcached -p 11211 -d -u root -m 512 -c 1024 -vvv
slab class   1: chunk size        80 perslab   13107
slab class   2: chunk size       104 perslab   10082
slab class   3: chunk size       136 perslab    7710
slab class   4: chunk size       176 perslab    5957
slab class   5: chunk size       224 perslab    4681
slab class   6: chunk size       280 perslab    3744
slab class   7: chunk size       352 perslab    2978
slab class   8: chunk size       440 perslab    2383
slab class   9: chunk size       552 perslab    1899
slab class  10: chunk size       696 perslab    1506
slab class  11: chunk size       872 perslab    1202
slab class  12: chunk size      1096 perslab     956
slab class  13: chunk size      1376 perslab     762
slab class  14: chunk size      1720 perslab     609
slab class  15: chunk size      2152 perslab     487
slab class  16: chunk size      2696 perslab     388
slab class  17: chunk size      3376 perslab     310
slab class  18: chunk size      4224 perslab     248
slab class  19: chunk size      5280 perslab     198
slab class  20: chunk size      6600 perslab     158
slab class  21: chunk size      8256 perslab     127
slab class  22: chunk size     10320 perslab     101
slab class  23: chunk size     12904 perslab      81
slab class  24: chunk size     16136 perslab      64
slab class  25: chunk size     20176 perslab      51
slab class  26: chunk size     25224 perslab      41
slab class  27: chunk size     31536 perslab      33
slab class  28: chunk size     39424 perslab      26
slab class  29: chunk size     49280 perslab      21
slab class  30: chunk size     61600 perslab      17
slab class  31: chunk size     77000 perslab      13
slab class  32: chunk size     96256 perslab      10
slab class  33: chunk size    120320 perslab       8
slab class  34: chunk size    150400 perslab       6
slab class  35: chunk size    188000 perslab       5
slab class  36: chunk size    235000 perslab       4
slab class  37: chunk size    293752 perslab       3
slab class  38: chunk size    367192 perslab       2
slab class  39: chunk size    458992 perslab       2
slab class  40: chunk size    573744 perslab       1
slab class  41: chunk size    717184 perslab       1
slab class  42: chunk size   1048576 perslab       1
<26 server listening (auto-negotiate)
<27 server listening (auto-negotiate)
<28 send buffer was 112640, now 268435456
<28 server listening (udp)
<28 server listening (udp)
<28 server listening (udp)
<28 server listening (udp)
<29 send buffer was 112640, now 268435456
<29 server listening (udp)
<29 server listening (udp)
<29 server listening (udp)
<29 server listening (udp)

3.5 查看memcached运行状态

telnet 127.0.0.1 11211
stats
STAT pid 676
STAT uptime 11815
STAT time 1422369026
STAT version 1.4.13
STAT libevent 2.0.21-stable
STAT pointer_size 32
STAT curr_connections 5
STAT total_connections 6
STAT connection_structures 6
STAT reserved_fds 20
STAT cmd_get 0
STAT cmd_set 0
STAT cmd_flush 0
STAT cmd_touch 0
STAT get_hits 0
STAT get_misses 0
STAT delete_misses 0
STAT delete_hits 0
STAT incr_misses 0
STAT incr_hits 0
STAT decr_misses 0
STAT decr_hits 0
STAT cas_misses 0
STAT cas_hits 0
STAT cas_badval 0
STAT touch_hits 0
STAT touch_misses 0
STAT auth_cmds 0
STAT auth_errors 0
STAT bytes_read 7
STAT bytes_written 0
STAT limit_maxbytes 67108864
STAT accepting_conns 1
STAT listen_disabled_num 0
STAT threads 4
STAT conn_yields 0
STAT hash_power_level 16
STAT hash_bytes 262144
STAT hash_is_expanding 0
STAT expired_unfetched 0
STAT evicted_unfetched 0
STAT bytes 0
STAT curr_items 0
STAT total_items 0
STAT evictions 0
STAT reclaimed 0
END

6、Memcached的集群

6.2 安装两个Memcached

(1)11211端口

[root@zookeeper_slave_01 bin]# pwd
/usr/local/bin
[root@zookeeper_slave_01 bin]# ./memcached -d -p 11211 -u root -m 128 -c 1024 -vvv

(2)11212端口

[root@zookeeper_slave_01 bin]# pwd
/soft_install/memcached-1.4.17/bin
[root@zookeeper_slave_01 bin]# ./memcached -d -p 11212 -u root -m 128 -c 1024 -vvv

6.3 JAVA客户端实现集群

package com.memecached;

import com.danga.MemCached.MemCachedClient;
import com.danga.MemCached.SockIOPool;

public class MemcachedClusters {
	private  static MemCachedClient memCachedClient = new MemCachedClient();
	
	// 设置与缓存服务器的连接池
	static {
		// 服务器列表和其权重
		String[] servers = { "192.168.74.129:11211", "192.168.74.129:11212" };
		Integer[] weights = { 1,2 };

		// 获取socke连接池的实例对象
		SockIOPool pool = SockIOPool.getInstance();

		// 设置服务器信息
		pool.setServers(servers);
		pool.setWeights(weights);

		// 设置初始连接数、最小和最大连接数以及最大处理时间
		pool.setInitConn(5);
		pool.setMinConn(5);
		pool.setMaxConn(250);
		pool.setMaxIdle(1000 * 60 * 60 * 6);

		// 设置主线程的睡眠时间
		pool.setMaintSleep(30);

		// 设置TCP的参数，连接超时等
		pool.setNagle(false);
		pool.setSocketTO(3000);
		pool.setSocketConnectTO(0);

		// 初始化连接池
		pool.initialize();
	}
	
	public static void main(String[] args) {
		int totalCount = 100;
		for (int i = 0; i < totalCount; i++) {
			memCachedClient.set("username"+i, "张三");
		}
		
		System.out.println(memCachedClient.stats());

	}

}

查看程序运行结果

{192.168.74.129:11212={total_items=67
, touch_misses=0
, cmd_touch=0
, auth_errors=0
, evictions=0
, time=1423814085
, pointer_size=64
, threads=4
, bytes_read=2479
, hash_is_expanding=0
, curr_connections=10
, reclaimed=0
, bytes_written=536
, hash_power_level=16
, cas_hits=0
, cmd_flush=0
, reserved_fds=20
, cas_badval=0
, get_hits=0
, curr_items=67
, evicted_unfetched=0
, hash_bytes=524288
, decr_misses=0
, delete_hits=0
, bytes=5487
, rusage_system=0.024996
, listen_disabled_num=0
, version=1.4.17
, incr_hits=0
, expired_unfetched=0
, limit_maxbytes=134217728
, get_misses=0
, connection_structures=11
, delete_misses=0
, total_connections=11
, malloc_fails=0
, rusage_user=0.000000
, uptime=463
, libevent=1.4.13-stable
, pid=2765
, touch_hits=0
, cas_misses=0
, accepting_conns=1
, cmd_get=0
, cmd_set=67
, incr_misses=0
, auth_cmds=0
, decr_hits=0
, conn_yields=0
}, 192.168.74.129:11211={total_items=33
, touch_misses=0
, cmd_touch=0
, auth_errors=0
, evictions=0
, time=1423814085
, pointer_size=64
, threads=4
, bytes_read=1225
, hash_is_expanding=0
, curr_connections=10
, reclaimed=0
, bytes_written=264
, hash_power_level=16
, cas_hits=0
, cmd_flush=0
, reserved_fds=20
, cas_badval=0
, get_hits=0
, curr_items=33
, evicted_unfetched=0
, hash_bytes=524288
, decr_misses=0
, delete_hits=0
, bytes=2703
, rusage_system=0.047992
, listen_disabled_num=0
, version=1.4.17
, incr_hits=0
, expired_unfetched=0
, limit_maxbytes=134217728
, get_misses=0
, connection_structures=11
, delete_misses=0
, total_connections=11
, malloc_fails=0
, rusage_user=0.002999
, uptime=473
, libevent=1.4.13-stable
, pid=2758
, touch_hits=0
, cas_misses=0
, accepting_conns=1
, cmd_get=0
, cmd_set=33
, incr_misses=0
, auth_cmds=0
, decr_hits=0
, conn_yields=0
}}

4、Memcached与Redis的比较(待续...)