Redis持久化策略之Rdb和Aof

什么是Redis的持久化       

Redis本质上是一个在内存上的数据库，内存的数据变化是很快的，也容易发生丢失。比如突然的宕机会导致数据的不完整，甚至数据的丢失，当我们再次重启机器的时候，就需要获取原来的数据，否则会造成数据的丢失。Redis为我们提供了持久化的机制，最开始的时候，Redis服务提供四种持久化存储方案：RDB、AOF、虚拟内存（VM）和　DISKSTORE。虚拟内存（VM）方式，从Redis Version 2.4开始就被官方明确表示不再建议使用，Version 3.2版本中更找不到关于虚拟内存（VM）的任何配置范例。DISKSTORE方式，是从Redis Version 2.8版本开始提出的一个存储设想，到目前为止Redis官方也没有在任何stable版本中明确建议使用这用方式。目前使用的分别是RDB(Redis DataBase)和AOF(Append Only File)。持久化的时候会把数据保存在硬盘上，RDB持久化的默认文件为dump.rdb，aof持久化的默认文件后缀名为.aof。虽说两者同时都为持久化，但是本质上还是不同的，并且，在机器重启的时候，首先加载的是aof文件，但是这并不影响RDB和AOF同时共存。

RDB持久化

关于RDB的持久化方式，可以参考配置文件进行相关配置，RDB持久化的配置方式在SNAPSHOTTING （快照）目录下，具体如下：

################################ SNAPSHOTTING  ################################
#
# Save the DB on disk:
#   save <seconds> <changes>
#   Will save the DB if both the given number of seconds and the given
#   number of write operations against the DB occurred.
#   In the example below the behaviour will be to save:
#   after 900 sec (15 min) if at least 1 key changed
#   after 300 sec (5 min) if at least 10 keys changed
#   after 60 sec if at least 10000 keys changed
#   Note: you can disable saving completely by commenting out all "save" lines.
#   It is also possible to remove all the previously configured save
#   points by adding a save directive with a single empty string argument
#   like in the following example:
#   save ""  //如果想禁用RDB持久化的策略，只要不设置任何save指令，或者给save传入一个空字符串参 
               数也可以
//RDB是整个内存的压缩过的Snapshot，RDB的数据结构，可以配置复合的快照触发条件，这里的修改不包括 
  get
save 900 1  //15分钟内改了1次
save 300 10  //5分钟内改了10次
save 60 10000 //1分钟内改了1万次

# By default Redis will stop accepting writes if RDB snapshots are enabled
# (at least one save point) and the latest background save failed.
# This will make the user aware (in a hard way) that data is not persisting
# on disk properly, otherwise chances are that no one will notice and some
# disaster will happen.
# If the background saving process will start working again Redis will
# automatically allow writes again.
# However if you have setup your proper monitoring of the Redis server
# and persistence, you may want to disable this feature so that Redis will
# continue to work as usual even if there are problems with disk,
# permissions, and so forth.

stop-writes-on-bgsave-error yes  //当出现错误时停止持久化，如果配置成no，表示你不在乎数据不一 
                                   致或者有其他的手段发现和控制

# Compress string objects using LZF when dump .rdb databases?
# For default that's set to 'yes' as it's almost always a win.
# If you want to save some CPU in the saving child set it to 'no' but
# the dataset will likely be bigger if you have compressible values or keys.
rdbcompression yes //对于存储到磁盘中的快照，可以设置是否进行压缩存储。如果是的话，redis会采用
                    LZF算法进行压缩。如果你不想消耗CPU来进行压缩的话，可以设置为关闭此功能

# Since version 5 of RDB a CRC64 checksum is placed at the end of the file.
# This makes the format more resistant to corruption but there is a performance
# hit to pay (around 10%) when saving and loading RDB files, so you can disable it
# for maximum performances.
# RDB files created with checksum disabled have a checksum of zero that will
# tell the loading code to skip the check.
rdbchecksum yes  //在存储快照后，还可以让redis使用CRC64算法来进行数据校验，但是这样做会增加大约
                   10%的性能消耗，如果希望获取到最大的性能提升，可以关闭此功能
# The filename where to dump the DB
dbfilename dump.rdb 

# The working directory.
# The DB will be written inside this directory, with the filename specified
# above using the 'dbfilename' configuration directive.
# The Append Only File will also be created inside this directory.
# Note that you must specify a directory here, not a file name.
dir ./

RDB持久化会在指定的时间间隔内将内存中的数据集快照写入磁盘，也就是行话讲的Snapshot快照，它恢复时是将快照文件直接读到内存里，Redis会单独创建（fork）一个子进程来进行持久化，会先将数据写入到一个临时文件中，待持久化过程都结束了，再用这个临时文件替换上次持久化好的文件。整个过程中，主进程是不进行任何IO操作的，这就确保了极高的性能如果需要进行大规模数据的恢复，且对于数据恢复的完整性不是非常敏感，那RDB方式要比AOF方式更加的高效。RDB的缺点是最后一次持久化后的数据可能丢失。

Fork的作用是复制一个与当前进程一样的进程。新进程的所有数据（变量、环境变量、程序计数器等）数值都和原进程一致，但是是一个全新的进程，并作为原进程的子进程，如果原进程所占内存相当庞大，那么就需要fork之后的进程所占内存问题，大致2倍的膨胀性需要考虑。

在持久化时可以通过配置文件来配置相应的持久化，如果需要马上进行持久化，则需要使用save命令或者BGSAVE命令，save时只管保存，其它不管，全部阻塞，BGSAVE：Redis会在后台异步进行快照操作，快照同时还可以响应客户端请求。可以通过lastsave命令获取最后一次成功执行快照的时间，需要注意，执行flushall命令，也会产生dump.rdb文件，但里面是空的，无意义。

RDB持久化的优点：

适合大规模的数据恢复，对数据完整性和一致性要求不高，非常适用于备份，全量复制等场景，Redis加载RDB恢复数据远远快于AOF的方式

RDB持久化的缺点：

在一定间隔时间做一次备份，所以如果redis意外down掉的话，就会丢失最后一次快照后的所有修改，Fork的时候，内存中的数据被克隆了一份，大致2倍的膨胀性需要考虑，RDB方式数据没办法做到实时持久化/秒级持久化。因为bgsave每次运 行都要执行fork操作创建子进程，属于重量级操作，频繁执行成本过高。并且RDB不适合实时持久化

RDB总结

AOF持久化机制

AOF以日志的形式来记录每个写操作，将Redis执行过的所有写指令记录下来(读操作不记录)，只许追加文件但不可以改写文件，redis启动之初会读取该文件重新构建数据，换言之，redis重启的话就根据日志文件的内容将写指令从前到后执行一次以完成数据的恢复工作，AOF的主要作用 是解决了数据持久化的实时性，AOF全称是Append Only File，从字面上理解就是“只进行增加的文件”。AOF方式的核心设计思量可以总结为一句话：忠实记录Redis服务启动成功后的每一次影响数据状态的操作命令，以便在Redis服务异常崩溃的情况出现时，可以按照这些操作命令恢复数据状态。既然要记录每次影响数据状态的操作命令，就意味着AOF文件会越来越大！这是必然的。还好Redis为AOF提供了一种重写AOF文件的功能，保证了AOF文件中可以存储尽可能少的操作命令就能保证数据恢复到最新状态，这个功能被称为日志重写功能

############################## APPEND ONLY MODE ###############################
# By default Redis asynchronously dumps the dataset on disk. This mode is
# good enough in many applications, but an issue with the Redis process or
# a power outage may result into a few minutes of writes lost (depending on
# the configured save points).
# The Append Only File is an alternative persistence mode that provides
# much better durability. For instance using the default data fsync policy
# (see later in the config file) Redis can lose just one second of writes in a
# dramatic event like a server power outage, or a single write if something
# wrong with the Redis process itself happens, but the operating system is
# still running correctly.
# AOF and RDB persistence can be enabled at the same time without problems.
# If the AOF is enabled on startup Redis will load the AOF, that is the file
# with the better durability guarantees.
# Please check http://redis.io/topics/persistence for more information.

appendonly yes  //启动：设置Yes,修改默认的appendonly no，改为yes

# The name of the append only file (default: "appendonly.aof")
appendfilename "appendonly.aof"  //Aof保存的是appendonly.aof文件

# The fsync() call tells the Operating System to actually write data on disk
# instead of waiting for more data in the output buffer. Some OS will really flush
# data on disk, some other OS will just try to do it ASAP.
# Redis supports three different modes:
# no: don't fsync, just let the OS flush the data when it wants. Faster.
# always: fsync after every write to the append only log. Slow, Safest.
# everysec: fsync only one time every second. Compromise.
# The default is "everysec", as that's usually the right compromise between
# speed and data safety. It's up to you to understand if you can relax this to
# "no" that will let the operating system flush the output buffer when
# it wants, for better performances (but if you can live with the idea of
# some data loss consider the default persistence mode that's snapshotting),
# or on the contrary, use "always" that's very slow but a bit safer than
# everysec. More details please check the following article:
# http://antirez.com/post/redis-persistence-demystified.html
# If unsure, use "everysec".appendfsync always appendfsync everysec 

appendfsync no  //Always：同步持久化 每次发生数据变更会被立即记录到磁盘  性能较差但数据完整性比 
                 较好
                //Everysec：出厂默认推荐，异步操作，每秒记录   如果一秒内宕机，有数据丢失
                //no: 从不同步
# When the AOF fsync policy is set to always or everysec, and a background
# saving process (a background save or AOF log background rewriting) is
# performing a lot of I/O against the disk, in some Linux configurations
# Redis may block too long on the fsync() call. Note that there is no fix for
# this currently, as even performing fsync in a different thread will block
# our synchronous write(2) call.
# In order to mitigate this problem it's possible to use the following option
# that will prevent fsync() from being called in the main process while a
# BGSAVE or BGREWRITEAOF is in progress.
# This means that while another child is saving, the durability of Redis is
# the same as "appendfsync none". In practical terms, this means that it is
# possible to lose up to 30 seconds of log in the worst scenario (with the
# default Linux settings).
# If you have latency problems turn this to "yes". Otherwise leave it as
# "no" that is the safest pick from the point of view of durability.

no-appendfsync-on-rewrite no  //重写时是否可以运用Appendfsync，用默认no即可，保证数据安全性。

# Automatic rewrite of the append only file.
# Redis is able to automatically rewrite the log file implicitly calling
# BGREWRITEAOF when the AOF log size grows by the specified percentage.
# This is how it works: Redis remembers the size of the AOF file after the
# latest rewrite (if no rewrite has happened since the restart, the size of
# the AOF at startup is used).
# This base size is compared to the current size. If the current size is
# bigger than the specified percentage, the rewrite is triggered. Also
# you need to specify a minimal size for the AOF file to be rewritten, this
# is useful to avoid rewriting the AOF file even if the percentage increase
# is reached but it is still pretty small.
# Specify a percentage of zero in order to disable the automatic AOF
# rewrite feature.

auto-aof-rewrite-percentage 100  //设置重写的基准值，代表当前AOF文件空间 
                     （aof_current_size）和上一次重写后AOF文件空间（aof_base_size）的比值。
auto-aof-rewrite-min-size 64mb  //设置重写的基准值，表示运行AOF重写时文件最小体积，默认 为 
                                  64MB。

# An AOF file may be found to be truncated at the end during the Redis
# startup process, when the AOF data gets loaded back into memory.
# This may happen when the system where Redis is running
# crashes, especially when an ext4 filesystem is mounted without the
# data=ordered option (however this can't happen when Redis itself
# crashes or aborts but the operating system still works correctly).
# Redis can either exit with an error when this happens, or load as much
# data as possible (the default now) and start if the AOF file is found
# to be truncated at the end. The following option controls this behavior.
# If aof-load-truncated is set to yes, a truncated AOF file is loaded and
# the Redis server starts emitting a log to inform the user of the event.
# Otherwise if the option is set to no, the server aborts with an error
# and refuses to start. When the option is set to no, the user requires
# to fix the AOF file using the "redis-check-aof" utility before to restart
# the server.
# Note that if the AOF file will be found to be corrupted in the middle
# the server will still exit with an error. This option only applies when
# Redis will try to read more data from the AOF file but not enough bytes
# will be found.

aof-load-truncated yes

AOF持久化的Rewrite机制

AOF采用文件追加方式，文件会越来越大为避免出现此种情况，新增了重写机制,当AOF文件的大小超过所设定的阈值时，Redis就会启动AOF文件的内容压缩，只保留可以恢复数据的最小指令集.可以使用命令bgrewriteaof，原理就是：AOF文件持续增长而过大时，会fork出一条新进程来将文件重写(也是先写临时文件最后再rename)，遍历新进程的内存中数据，每条记录有一条的Set语句。重写aof文件的操作，并没有读取旧的aof文件，而是将整个内存中的数据库内容用命令的方式重写了一个新的aof文件，这点和快照有点类似。重写后的AOF文件可以变小是因为：进程内已经超时的数据不再写入文件，旧的AOF文件含有无效命令，重写使用进程内数据直接生成，这样新的AOF文件只保留最终数据的写入命令。多条写命令可以合并为一个，为了防止单条命令过大造成客户端缓冲区溢出，对于list、set、hash、zset等类型操作，以64个元素为界拆分为多条，AOF重写降低了文件占用空间，除此之外，另一个目的是：更小的AOF 文件可以更快地被Redis加载

触发AOF重写

手动使用命令：bgrewriteaof

根据配置文件中配置的参数：auto-aof-rewrite-min-size和auto-aof-rewrite-percentage判断是否进行重写

AOF持久化的优点

可以根据实际设置不同的同步策略，更加的灵活，可以设置：每修改同步：appendfsync always   同步持久化 每次发生数据变更会被立即记录到磁盘  性能较差但数据完整性比较好，每秒同步：appendfsync everysec    异步操作，每秒记录   如果一秒内宕机，有数据丢失，解决了数据持久化的实时性

缺点

相同数据集的数据而言aof文件要远大于rdb文件，恢复速度慢于rdb，Aof运行效率要慢于rdb,每秒同步策略效率较好，不同步效率和rdb相同

总结

无论是使用RDB或者AOF方式进行持久化，如果恢复过程中出现错误，可以使用Redis-check-aof /Redis-check-rdb对文件进行修复，RDB持久化方式能够在指定的时间间隔能对你的数据进行快照存储，AOF持久化方式记录每次对服务器写的操作,当服务器重启的时候会重新执行这些命令来恢复原始的数据,AOF命令以redis协议追加保存每次写的操作到文件末尾.Redis还能对AOF文件进行后台重写,使得AOF文件的体积不至于过大，如果你只希望你的数据在服务器运行的时候存在,你也可以不使用任何持久化方式.，同时开启两种持久化方式时，当redis重启的时候会优先载入AOF文件来恢复原始的数据，因为在通常情况下AOF文件保存的数据集要比RDB文件保存的数据集要完整，RDB的数据不实时，同时使用两者时服务器重启也只会找AOF文件。那要不要只使用AOF呢？作者建议不要，因为RDB更适合用于备份数据库(AOF在不断变化不好备份)，快速重启，而且不会有AOF可能潜在的bug，留着作为一个万一的手段。

因为RDB文件只用作后备用途，建议只在Slave上持久化RDB文件，而且只要15分钟备份一次就够了，只保留save 900 1这条规则。如果Enalbe AOF，好处是在最恶劣情况下也只会丢失不超过两秒数据，启动脚本较简单只load自己的AOF文件就可以了。代价一是带来了持续的IO，二是AOF rewrite的最后将rewrite过程中产生的新数据写到新文件造成的阻塞几乎是不可避免的。只要硬盘许可，应该尽量减少AOF rewrite的频率，AOF重写的基础大小默认值64M太小了，可以设到5G以上。默认超过原大小100%大小时重写可以改到适当的数值。如果不Enable AOF ，仅靠Master-Slave Replication 实现高可用性也可以。能省掉一大笔IO也减少了rewrite时带来的系统波动。代价是如果Master/Slave同时倒掉，会丢失十几分钟的数据，启动脚本也要比较两个Master/Slave中的RDB文件，载入较新的那个。新浪微博就选用了这种架构