【Redis-6.0.8】事件循环器AE(上)

目录

 

0.阅读

1.引言

1.1 redis中的事件分类

1.2 为兼容各平台写的代码

1.3 linux上巧妙借用系统函数epoll_wait设置阻塞时间以达到触发条件的代码

1.4 两种事件的调度

2.源码阅读

2.1 ae.h中提供的内容

2.1.1 宏

2.1.2  重定义函数原型

2.1.3 aeFileEvent(IO事件)结构体的定义

2.1.4 aeTimeEvent(定时器事件)结构体的定义

2.1.5 aeFiredEvent(管理就绪事件)结构体的定义

2.1.6 aeEventLoop(事件循环)结构体的定义

2.1.7 ae.h会提供的函数声明

2.2  与AE流程相关的代码逻辑

2.2.1 initServer中与AE有关的初始化

2.2.2 aeMain中的事件循环

2.2.3 全文搜索设置的wfileProc和rfileProc

2.2.4 读事件，写事件与时间事件的处理逻辑

---

0.阅读

My-基础回顾

图文并茂-说明了flag的用处-初始为AE_NOE

大佬的redis源码注释

redis源码分析—AE事件处理机制

redis事件模型源码分析2：aeEvent模型

Redis源码剖析和注释（十九）--- Redis 事件处理实现

Redis源码分析（二十）--- ae事件驱动

为什么redis取出来是null_Redis事件循环器(AE)实现剖析

Linux网络编程---I/O复用模型之epoll-复习

回调函数与钩子函数

函数指针与回调函数详解

evport-Event ports-Solaris 10

参考-较新版本的代码分析-有AE_BARRIE

aeSetDontWait的理解

复习C语言中的运算符的优先级

侦听fd与客户端fd是如何挂载到EPFD上去的-有系列文章

网络套接字和本地套接字

当心！TCP本机客户端连接本机服务器-dog250

1.引言

1.1 redis中的事件分类

redis中有两类事件:

1. IO事件，统一封装成aeFileEvent，底层调用系统支持的多路复用层（evport、epoll、kqueue、select）；
2. 定时器事件，以aeTimeEvent描述，巧妙借用系统函数epoll_wait等阻塞函数设置阻塞时间以达到触发条件. 

1.2 为兼容各平台写的代码

// 根据宏定义包含系统支持的多路复用模型，从上到下按性能高低排序
#ifdef HAVE_EVPORT
#include "ae_evport.c"  // solaris
#else
    #ifdef HAVE_EPOLL
    #include "ae_epoll.c"      // linux 
    #else
        #ifdef HAVE_KQUEUE
        #include "ae_kqueue.c" // freeBSD & macosx
        #else
        #include "ae_select.c" // windows
        #endif
    #endif
#endif

1.3 linux上巧妙借用系统函数epoll_wait设置阻塞时间以达到触发条件的代码

retval = epoll_wait(state->epfd,state->events,eventLoop->setsize,
            tvp ? (tvp->tv_sec*1000 + tvp->tv_usec/1000) : -1);

1.4 两种事件的调度

如上，Redis中有两种事件，分别是文件事件和时间事件，它们通过epoll_wait(linux下)巧妙地产生了联系.

调度指的是什么时候执行文件事件，什么时候执行时间事件. 两种事件都是在aeProcessEvents中进行处理.
大致流程是这样的，首先遍历时间事件的链表找出离现在最近的时间事件的时间间隔,根据这个最小时间间隔来
设置io多路复用接口的等待时间(即linux上的epoll_wait中设置的等待时间).此时并没有执行时间事件,要先
处理文件事件再处理时间事件.
比如说：serverCron函数每100ms执行一次，现在是50ms, 假设多路复用是用的epoll，那么就设置epoll_wait
的等待时间为50，这样假如没有网络事件到达50m后就返回去处理时间事件.另一种情况，如果检测到时间事件已
经到达了，那么就设置epoll_wait的等待时间为0，这样不管有没有网络事件发生epoll_wait也会立即返回，因
为有时间事件等着去处理.

 

2.源码阅读

2.1 ae.h中提供的内容

2.1.1 宏

/* aeResizeSetSize,aeCreateFileEvent,aeDeleteTimeEvent函数的可能返回值*/
#define AE_OK 0  
/* 作为众多ae系列函数的返回值使用 */
#define AE_ERR -1
/* 没有事件注册上 */
#define AE_NONE 0       /* No events registered. */
/* 有读事件注册上 */
#define AE_READABLE 1   /* Fire when descriptor is readable. */
/* 有写事件注册上 */
#define AE_WRITABLE 2   /* Fire when descriptor is writable. */
/* 
   一般情况先执行readable事件再执行writable事件，
   这里的AE_BARRIER表示事件屏障，设置了AE_BARRIER之后,如果已经有了readable事件之后就不会触发 
   writable事件,在如下场景下适用:
   例如：你想要的以批量的方式先将file同步到磁盘，然后再回复给客户端
   这种情况下先执行writable事件再执行readable事件.
   感觉
    
   这里还需要再多多理解,不是很明白.(本段解释可能不对，感觉这里的英文解释写的不是很好，据下面
   的说，应该就是如果设置了AE_BARRIER，就优先处理写事件）
*/
#define AE_BARRIER 4    /* With WRITABLE, never fire the event if the
                           READABLE event already fired in the same event
                           loop iteration. Useful when you want to persist
                           things to disk before sending replies, and want
                           to do that in a group fashion. */
/* 0001-表示文件事件 */
#define AE_FILE_EVENTS (1<<0)
/* 0010-表示时间事件 */
#define AE_TIME_EVENTS (1<<1)
/* 0011-表示文件事件和时间事件 */
#define AE_ALL_EVENTS (AE_FILE_EVENTS|AE_TIME_EVENTS)
/* 0100-表示函数处理完事件后直接返回，不阻塞等待 */
#define AE_DONT_WAIT (1<<2)
/* 
1000-用于aeProcessEvents(aeEventLoop *eventLoop, int flags)等处,如果
flags设置了AE_CALL_BEFORE_SLEEP,eventLoop->beforesleep回调函数会被调用
*/
#define AE_CALL_BEFORE_SLEEP (1<<3)
/* 
10000-用于aeProcessEvents(aeEventLoop *eventLoop, int flags)等处,如果
flags设置了AE_CALL_BEFORE_SLEEP,eventLoop->aftersleep回调函数会被调用
*/
#define AE_CALL_AFTER_SLEEP (1<<4)
/* 表示没有事件了,在processTimeEvents中被用到 */
#define AE_NOMORE -1
/* 表示删除事件ID，在processTimeEvents，aeDeleteTimeEvent等函数中被用到 */
#define AE_DELETED_EVENT_ID -1
/* 仅仅用作编译器处理，防止因为没有到相关变量而被当做错误 */
#define AE_NOTUSED(V) ((void) V)

2.1.2  重定义函数原型

/* Types and data structures */
typedef void aeFileProc(struct aeEventLoop *eventLoop, int fd, void *clientData, int mask);
typedef int aeTimeProc(struct aeEventLoop *eventLoop, long long id, void *clientData);
typedef void aeEventFinalizerProc(struct aeEventLoop *eventLoop, void *clientData);
typedef void aeBeforeSleepProc(struct aeEventLoop *eventLoop);

aeFileProc          -有IO事件时处理IO事件的函数原型
aeTimeProc          -有时间事件时处理时间事件的函数原型
aeEventFinalizerProc-一个对eventLoop和clientData处理的函数原型
aeBeforeSleepProc   -一个对eventLoop处理的函数原型，后面此函数类型具体的对象有beforeSleep和afterSleep

2.1.3 aeFileEvent(IO事件)结构体的定义

/* IO事件结构体 */
/* File event structure */
typedef struct aeFileEvent {
    /* 文件事件类型：是AE_READABLE,AE_WRITABLE和AE_BARRIER中的一个 */
    int mask; /* one of AE_(READABLE|WRITABLE|BARRIER) */
    /* 有可读IO事件时的处理函数 */
    aeFileProc *rfileProc;
    /* 有可写IO事件时的处理函数 */
    aeFileProc *wfileProc;
    /* 客户端传入的数据 */
    void *clientData;
} aeFileEvent;

2.1.4 aeTimeEvent(定时器事件)结构体的定义

/* Time event structure */
typedef struct aeTimeEvent {
    /* 时间事件的id */
    long long id; /* time event identifier. */
    /* 时间事件到达的指定时间的秒数 */
    long when_sec; /* seconds */
    /* 时间事件到达的指定时间的毫秒数*/
    long when_ms; /* milliseconds */
    /* 时间事件处理函数 */
    aeTimeProc *timeProc;
    /* 时间事件终结函数 */
    aeEventFinalizerProc *finalizerProc;
    /* 客户端传入的数据 */
    void *clientData;
    /* 指向上一个时间事件的指针 */
    struct aeTimeEvent *prev;
    /* 指向下一个时间事件的指针 */
    struct aeTimeEvent *next;
    /* 引用次数，防止时间事件在多次被调用后被释放*/
    int refcount; /* refcount to prevent timer events from being
  		           * freed in recursive time event calls. */
} aeTimeEvent;

2.1.5 aeFiredEvent(管理就绪事件)结构体的定义

/* 就绪事件 */
/* A fired event */
typedef struct aeFiredEvent {
    /* 就绪事件的文件描述符 */
    int fd;
    /* 就绪事件类型，如AE_NONE，AE_READABLE，AE_WRITABLE等 */
    int mask;
} aeFiredEvent;

2.1.6 aeEventLoop(事件循环)结构体的定义

/* State of an event based program */
typedef struct aeEventLoop {
    /* 当前已注册的最大的文件描述符 */
    int maxfd;   /* highest file descriptor currently registered */
    /* 文件描述符监听集合的大小 */
    int setsize; /* max number of file descriptors tracked */
    /* 下一个时间事件的ID */
    long long timeEventNextId;
    /* 最后一次执行事件的时间 */
    time_t lastTime;     /* Used to detect system clock skew */
    /* 已注册的事件 */
    aeFileEvent *events; /* Registered events */
    /* 已触发的事件 */
    aeFiredEvent *fired; /* Fired events */
    /* 时间事件的头节点指针 */
    aeTimeEvent *timeEventHead;
    /* 事件处理开关 */
    int stop;
    /* 多路复用库的事件状态数据 */
    void *apidata; /* This is used for polling API specific data */
    /* 执行处理事件之前的函数 */
    aeBeforeSleepProc *beforesleep;
    /* 执行处理事件之后的函数 */
    aeBeforeSleepProc *aftersleep;
    /* 事件循环的标志，初始值赋值为NONE(0)，在aeCreateEventLoop中搜索eventLoop->flags可看到*/
    int flags;
} aeEventLoop;

2.1.7 ae.h会提供的函数声明

/* 创建aeEventLoop */
aeEventLoop *aeCreateEventLoop(int setsize);
/* 删除EventLoop，释放相应的事件所占的空间 */
void aeDeleteEventLoop(aeEventLoop *eventLoop);
/* 设置eventLoop中的停止属性为1，服务器中似乎没有用到，压测和客户端中有用到这个函数 */
void aeStop(aeEventLoop *eventLoop); 
/* 在eventLoop中创建文件事件 */
int aeCreateFileEvent(aeEventLoop *eventLoop, int fd, int mask,
        aeFileProc *proc, void *clientData);
/* 删除文件事件 */
void aeDeleteFileEvent(aeEventLoop *eventLoop, int fd, int mask);
/* 根据文件描述符id，找出文件的属性，是读事件还是写事件 */
int aeGetFileEvents(aeEventLoop *eventLoop, int fd);
/* 在eventLoop中添加时间事件，创建的时间为当前时间加上自己传入的时间 */
long long aeCreateTimeEvent(aeEventLoop *eventLoop, long long milliseconds,
        aeTimeProc *proc, void *clientData,
        aeEventFinalizerProc *finalizerProc);
/* 根据时间id，删除时间事件，涉及链表的操作 */
int aeDeleteTimeEvent(aeEventLoop *eventLoop, long long id);
/* 处理eventLoop中的所有类型事件 */
int aeProcessEvents(aeEventLoop *eventLoop, int flags);
/* 让某事件等待 */
int aeWait(int fd, int mask, long long milliseconds);
/* ae事件执行主程序 */
void aeMain(aeEventLoop *eventLoop);
/* 获取接口名 */
char *aeGetApiName(void);
/* 设置eventLoop->beforesleep回调函数*/
void aeSetBeforeSleepProc(aeEventLoop *eventLoop, aeBeforeSleepProc *beforesleep);
/* 设置eventLoop->aftersleep回调函数 */
void aeSetAfterSleepProc(aeEventLoop *eventLoop, aeBeforeSleepProc *aftersleep);
/* 获取eventLoop的长度*/
int aeGetSetSize(aeEventLoop *eventLoop);
/* 设置eventLoop的长度*/
int aeResizeSetSize(aeEventLoop *eventLoop, int setsize);
/* 通知事件的下一个迭代器将超时设置为零，即不等待 */
void aeSetDontWait(aeEventLoop *eventLoop, int noWait);

2.2  与AE流程相关的代码逻辑

int main(){
    ...
    initServer();
    ...
    aeMain(server.el);
    aeDeleteEventLoop(server.el);
    ...
}

2.2.1 initServer中与AE有关的初始化

一个server.c的注释

2.2.1.1 initServer中的重点函数

int main(){
    ...
    initServer();
    ...
}

void initServer(void) {

/* 创建侦听 fd */
/* Open the TCP listening socket for the user commands. */
    if (server.port != 0 &&
        listenToPort(server.port,server.ipfd,&server.ipfd_count) == C_ERR)
        exit(1);
    if (server.tls_port != 0 &&
        listenToPort(server.tls_port,server.tlsfd,&server.tlsfd_count) == C_ERR)
        exit(1);
/* 将侦听fd设置为非阻塞的 */
// anetNonBlock  // 在listenToPort调用到了,initServer中也可能调用到
...
/* 初始化server.el ,注意在这里的aeCreateEventLoop内部调用了epoll_create */
server.el = aeCreateEventLoop(server.maxclients+CONFIG_FDSET_INCR);
    if (server.el == NULL) {
        serverLog(LL_WARNING,
            "Failed creating the event loop. Error message: '%s'",
            strerror(errno));
        exit(1);
    }
...
    /* 创建Redis的定时器，设置定时器的回调函数为serverCron */
    /* Create the timer callback, this is our way to process many background
     * operations incrementally, like clients timeout, eviction of unaccessed
     * expired keys and so forth. */
    if (aeCreateTimeEvent(server.el, 1, serverCron, NULL, NULL) == AE_ERR) {
        serverPanic("Can't create event loop timers.");
        exit(1);
    }
    /* 设置TCP和Unix本地套接字接受新链接时的事件响应函数 */
    /* Create an event handler for accepting new connections in TCP and Unix
     * domain sockets. */
    for (j = 0; j < server.ipfd_count; j++) {
        if (aeCreateFileEvent(server.el, server.ipfd[j], AE_READABLE,
            acceptTcpHandler,NULL) == AE_ERR)
            {
                serverPanic(
                    "Unrecoverable error creating server.ipfd file event.");
            }
    }
    for (j = 0; j < server.tlsfd_count; j++) {
        if (aeCreateFileEvent(server.el, server.tlsfd[j], AE_READABLE,
            acceptTLSHandler,NULL) == AE_ERR)
            {
                serverPanic(
                    "Unrecoverable error creating server.tlsfd file event.");
            }
    }
    if (server.sofd > 0 && aeCreateFileEvent(server.el,server.sofd,AE_READABLE,
        acceptUnixHandler,NULL) == AE_ERR) serverPanic("Unrecoverable error creating server.sofd file event.");

    /*为管道注册一个用于唤醒事件循环的可读事件，需要注意模块中被阻塞的客户端 */
    /* Register a readable event for the pipe used to awake the event loop
     * when a blocked client in a module needs attention. */
    if (aeCreateFileEvent(server.el, server.module_blocked_pipe[0], AE_READABLE,
        moduleBlockedClientPipeReadable,NULL) == AE_ERR) {
            serverPanic(
                "Error registering the readable event for the module "
                "blocked clients subsystem.");
    }
    /*  注册before和after睡眠函数(注意要在加载持久化的数据之前进行，因为它会被 
        processEventsWhileBlocked函数用到 */
    /* Register before and after sleep handlers (note this needs to be done
     * before loading persistence since it is used by processEventsWhileBlocked. */
    aeSetBeforeSleepProc(server.el,beforeSleep);
    aeSetAfterSleepProc(server.el,afterSleep);
...
}


网络编程关键步骤：
listenToPort      -- 监听端口
anetNonBlock      -- 将侦听fd设置为非阻塞的
aeCreateEventLoop -- 调用epoll_create
acceptTcpHandler  -- 此处仅设置回调函数，回调函数中实现将侦听fd绑定到epfd上去，触发的时候调用



问题：
aeCreateFileEvent的原型是：
int aeCreateFileEvent(aeEventLoop *eventLoop, int fd, int mask, aeFileProc *proc, void *clientData)
{
   ...
   if (mask & AE_READABLE) fe->rfileProc = proc;
   if (mask & AE_WRITABLE) fe->wfileProc = proc;
   ...
}

相关调用：
aeCreateFileEvent(server.el, server.ipfd[j], AE_READABLE,acceptTcpHandler,NULL)；
aeCreateFileEvent(server.el, server.tlsfd[j], AE_READABLE,acceptTLSHandler,NULL)；
aeCreateFileEvent(server.el,server.module_blocked_pipe[0],AE_READABLE,moduleBlockedClientPipeReadable,NULL)；
aeCreateFileEvent(server.el,server.sofd,AE_READABLE,acceptUnixHandler,NULL);

问：
调用aeCreateFileEvent的时候第二个参数分别为：
server中的 
          ipfd，
          tlsfd，
          module_blocked_pipe，
          sofd，
这四个参数分别都代表什么意思？

注册的回调函数为：
acceptTcpHandler -- 用于accept client的connect
acceptTLSHandler -- 用于acceptclient的本地connect
moduleBlockedClientPipeReadable
acceptUnixHandler
这四个回调函数有什么不同的用途？




补充：不仅仅在初始化的时候有AE相关操作，在全局静态区也有，如
aeCreateFileEvent(loop,e->fd,AE_WRITABLE,redisAeWriteEvent,e);
aeCreateFileEvent(loop,e->fd,AE_READABLE,redisAeReadEvent,e);

2.2.1.2 稍微看下aeCreateEventLoop-内部执行了epoll_create

aeEventLoop *aeCreateEventLoop(int setsize) {
    aeEventLoop *eventLoop;
    int i;
    if ((eventLoop = zmalloc(sizeof(*eventLoop))) == NULL) goto err;
    eventLoop->events = zmalloc(sizeof(aeFileEvent)*setsize);
    eventLoop->fired = zmalloc(sizeof(aeFiredEvent)*setsize);
    if (eventLoop->events == NULL || eventLoop->fired == NULL) goto err;
    eventLoop->setsize = setsize;
    eventLoop->lastTime = time(NULL);
    eventLoop->timeEventHead = NULL;
    eventLoop->timeEventNextId = 0;
    eventLoop->stop = 0;
    eventLoop->maxfd = -1;
    eventLoop->beforesleep = NULL;
    eventLoop->aftersleep = NULL;
    eventLoop->flags = 0;
    if (aeApiCreate(eventLoop) == -1) goto err;
    /* Events with mask == AE_NONE are not set. So let's initialize the
     * vector with it. */
    for (i = 0; i < setsize; i++)
        eventLoop->events[i].mask = AE_NONE;
    return eventLoop;

err:
    if (eventLoop) {
        zfree(eventLoop->events);
        zfree(eventLoop->fired);
        zfree(eventLoop);
    }
    return NULL;
}

2.2.2 aeMain中的事件循环

2.2.2.1 aeMain的实现

void aeMain(aeEventLoop *eventLoop) {
    eventLoop->stop = 0;
    while (!eventLoop->stop) {
        aeProcessEvents(eventLoop, AE_ALL_EVENTS|
                                   AE_CALL_BEFORE_SLEEP|
                                   AE_CALL_AFTER_SLEEP);
    }
}

 2.2.2.2 aeProcessEvents的实现

int aeProcessEvents(aeEventLoop *eventLoop, int flags)
{
    /*
     1.定义临时变量processed(已经处理好的事件数)并初始化；
     2.定义临时变量numevents(事件数)
    */
    int processed = 0, numevents;

    /* 如果不处理定时器事件,也不处理文件事件,就立即直接返回 */    
    /* Nothing to do? return ASAP */
    if (!(flags & AE_TIME_EVENTS) && !(flags & AE_FILE_EVENTS)) return 0;
     
    /* 请注意，既然我们要处理时间事件，即使没有要处理的文件事件，我们仍要调用select()，以便在下
       一次事件准备启动之前进行休眠 */
    /* Note that we want call select() even if there are no
     * file events to process as long as we want to process time
     * events, in order to sleep until the next time event is ready
     * to fire. */
    /* 如果【有监控文件事件】或者【有要处理定时器事件并且没有设置不阻塞标志】则进入逻辑*/
    if (eventLoop->maxfd != -1 ||
        ((flags & AE_TIME_EVENTS) && !(flags & AE_DONT_WAIT))) {
        int j;
        aeTimeEvent *shortest = NULL;
        struct timeval tv, *tvp;
        
        /* 如果处理定时器事件且没有设置不阻塞标志，取最快到达的时间事件 */
        if (flags & AE_TIME_EVENTS && !(flags & AE_DONT_WAIT))
            shortest = aeSearchNearestTimer(eventLoop);
        /* 如果获取的最快到达的时间事件shortest非空 */
        if (shortest) {
            long now_sec, now_ms;

            aeGetTime(&now_sec, &now_ms); /* 获取当前时间 */
            tvp = &tv;
            
            /* 计算还有多少毫秒能够触发下一次时间事件 */
            /* How many milliseconds we need to wait for the next
             * time event to fire? */
            long long ms =
                (shortest->when_sec - now_sec)*1000 +
                shortest->when_ms - now_ms;
            
            /* 对计算出来的ms做出一些处理,如果ms大于零则将其赋值给临时变量tvp，否则将tvp两部分 
               都置为0*/
            if (ms > 0) {
                tvp->tv_sec = ms/1000;
                tvp->tv_usec = (ms % 1000)*1000;
            } else {
                tvp->tv_sec = 0;
                tvp->tv_usec = 0;
            }
        } 
        else  /* 如果获取的shortest为空，即不存在时间事件 */
        {
            /* 如果设置了不阻塞标志，则将阻塞时间为0，表示不阻塞 */
            /* If we have to check for events but need to return
             * ASAP because of AE_DONT_WAIT we need to set the timeout
             * to zero */
            if (flags & AE_DONT_WAIT) {
                tv.tv_sec = tv.tv_usec = 0;
                tvp = &tv;
            } else { /* 阻塞直到第一个时间事件的到来 */
                /* Otherwise we can block */
                tvp = NULL; /* wait forever */
            }
        }
        
        /* 如果事件循环的标志位不阻塞等待，那么将tv的成员变量值置为0并且将tvp指向tv的地址*/
        if (eventLoop->flags & AE_DONT_WAIT) {
            tv.tv_sec = tv.tv_usec = 0;
            tvp = &tv;
        }
        /* 执行阻塞前的处理函数 */
        if (eventLoop->beforesleep != NULL && flags & AE_CALL_BEFORE_SLEEP)
            eventLoop->beforesleep(eventLoop);
       
        /* 调用aeApiPoll,这个函数仅仅会返回超时或者当有事件触发的时候回返回就绪文件事件个数
           linux下的epoll_wait的返回值:若成功，返回就绪的文件描述符个数;
                              若出错, 返回-1，
                              若超时, 返回0
           这里只会返回0或者就绪的文件描述符个数.
         */
        /* 阻塞等待文件事件,这个方法在超时或者有事件触发时才会返回 */
        /* Call the multiplexing API, will return only on timeout or when
         * some event fires. */
        numevents = aeApiPoll(eventLoop, tvp);
       
        /* 执行阻塞后的处理函数 */
        /* After sleep callback. */
        if (eventLoop->aftersleep != NULL && flags & AE_CALL_AFTER_SLEEP)
            eventLoop->aftersleep(eventLoop);
        

        /* 循环处理触发的文件事件 */
        for (j = 0; j < numevents; j++) {
            /* 获取触发的文件事件 */
            aeFileEvent *fe = &eventLoop->events[eventLoop->fired[j].fd];
            int mask = eventLoop->fired[j].mask;
            int fd = eventLoop->fired[j].fd;
            int fired = 0; /* Number of events fired for current fd. */

            /* Normally we execute the readable event first, and the writable
             * event laster. This is useful as sometimes we may be able
             * to serve the reply of a query immediately after processing the
             * query.
             *
             * However if AE_BARRIER is set in the mask, our application is
             * asking us to do the reverse: never fire the writable event
             * after the readable. In such a case, we invert the calls.
             * This is useful when, for instance, we want to do things
             * in the beforeSleep() hook, like fsynching a file to disk,
             * before replying to a client. */

            /* 查看事件是否设置AE_BARRIER标志,如果设置了AE_BARRIER标志,优先处理写事件 */
            int invert = fe->mask & AE_BARRIER;

            /* Note the "fe->mask & mask & ..." code: maybe an already
             * processed event removed an element that fired and we still
             * didn't processed, so we check if the event is still valid.
             *
             * Fire the readable event if the call sequence is not
             * inverted. */
             /*

             */
             /* 如果没有设置AE_BARRIER标志，优先处理读事件 */
            if (!invert && fe->mask & mask & AE_READABLE) {
                fe->rfileProc(eventLoop,fd,fe->clientData,mask);
                fired++;
                fe = &eventLoop->events[fd]; /* Refresh in case of resize. */
            }
            
            /*  
                 如果是写事件，且在【没有翻转读写顺序】或【fe->wfileProc与fe->rfileProc】
                 的情况下，执行写事件处理函数
                 疑问：为什么要做【fe->wfileProc与fe->rfileProc】的判断？
                       可能是默写版本中的错误，下面有refresh in case of resize，应该
                       是为了解决一些存在的瞬时状态的bug而产生的更严谨的写法.
            */
            /* Fire the writable event. */
            if (fe->mask & mask & AE_WRITABLE) {
                if (!fired || fe->wfileProc != fe->rfileProc) {
                    fe->wfileProc(eventLoop,fd,fe->clientData,mask);
                    fired++;
                }
            }

            /* If we have to invert the call, fire the readable event now
             * after the writable one. */
            /* 如果我们一定要颠倒读事件与写事件的处理顺序 */
            if (invert) {
                /* 重新获取一下文件事件防止被更新 */
                fe = &eventLoop->events[fd]; /* Refresh in case of resize. */
                /* 
                  如果是读事件且没有被置成触发状态且【fe->wfileProc != fe->rfileProc】
                  执行读文件事件处理函数.
                */
                if ((fe->mask & mask & AE_READABLE) &&
                    (!fired || fe->wfileProc != fe->rfileProc))
                {
                    fe->rfileProc(eventLoop,fd,fe->clientData,mask);
                    fired++; // 触发统计值加一
                }
            }
            /* 处理值加一 */
            processed++;
        }
    }
    /* Check time events */
    /* 如果是时间事件 */
    if (flags & AE_TIME_EVENTS)
        processed += processTimeEvents(eventLoop);/* 调用时间事件处理函数 */

    return processed; /* return the number of processed file/time events */
}

2.2.2.3 linux下的aeApiPoll-调用了epoll_wait

static int aeApiPoll(aeEventLoop *eventLoop, struct timeval *tvp) {
    aeApiState *state = eventLoop->apidata;
    int retval, numevents = 0;
    retval = epoll_wait(state->epfd,state->events,eventLoop->setsize,
            tvp ? (tvp->tv_sec*1000 + tvp->tv_usec/1000) : -1);
    if (retval > 0) {
        int j;
        numevents = retval;
        for (j = 0; j < numevents; j++) {
            int mask = 0;
            struct epoll_event *e = state->events+j;

            if (e->events & EPOLLIN) mask |= AE_READABLE; 
            if (e->events & EPOLLOUT) mask |= AE_WRITABLE;
            if (e->events & EPOLLERR) mask |= AE_WRITABLE|AE_READABLE;
            if (e->events & EPOLLHUP) mask |= AE_WRITABLE|AE_READABLE;
            eventLoop->fired[j].fd = e->data.fd;
            eventLoop->fired[j].mask = mask;
        }
    }
    return numevents;
}

/*
复习：

struct epoll_event {
    uint32_t     events;      /* Epoll events */
    epoll_data_t data;        /* User data variable */
};
events成员描述事件类型，将以下宏定义通过位或方式组合:
EPOLLIN  :表示对应的文件描述符可以读(包括对端SOCKET正常关闭)
POLLOUT  :表示对应的文件描述符可以写
EPOLLPRI :表示对应的文件描述符有紧急的数据可读(这里应该表示有带外数据到来)
EPOLLERR :表示对应的文件描述符发生错误
EPOLLHUP :表示对应的文件描述符被挂断;
EPOLLET  :将EPOLL设为边缘触发(Edge Triggered)模式,这是相对于水平触发(Level Triggered)来说的
EPOLLONESHOT:只监听一次事件,当监听完这次事件之后,如果还需要继续监听这个socket的话,需要再次把这个socket加入到EPOLL队列里

*/

2.2.3 全文搜索设置的wfileProc和rfileProc

int aeCreateFileEvent(aeEventLoop *eventLoop, int fd, int mask,
        aeFileProc *proc, void *clientData)
{
    if (fd >= eventLoop->setsize) {
        errno = ERANGE;
        return AE_ERR;
    }
    aeFileEvent *fe = &eventLoop->events[fd];

    if (aeApiAddEvent(eventLoop, fd, mask) == -1)
        return AE_ERR;
    fe->mask |= mask;
    if (mask & AE_READABLE) fe->rfileProc = proc;
    if (mask & AE_WRITABLE) fe->wfileProc = proc;
    fe->clientData = clientData;
    if (fd > eventLoop->maxfd)
        eventLoop->maxfd = fd;
    return AE_OK;
}


/* aof.c */
aeCreateFileEvent(server.el, server.aof_pipe_write_data_to_child,AE_WRITABLE, aofChildWriteDiffData, NULL);

/* cluster.c */
aeCreateFileEvent(server.el, server.cfd[j], AE_READABLE,clusterAcceptHandler, NULL);

/* connections */
aeCreateFileEvent(server.el, conn->fd, AE_WRITABLE,conn->type->ae_handler, conn); 
aeCreateFileEvent(server.el,conn->fd,AE_READABLE,conn->type->ae_handler,conn);

/* rdb.c */
aeCreateFileEvent(server.el, server.rdb_pipe_read, AE_READABLE, rdbPipeReadHandler,NULL);
aeCreateFileEvent(server.el, server.rdb_pipe_read, AE_READABLE, rdbPipeReadHandler,NULL);


/* redis-benchmark.c */
aeCreateFileEvent(el,c->context->fd,AE_WRITABLE,writeHandler,c);
aeCreateFileEvent(el,c->context->fd,AE_READABLE,readHandler,c);
aeCreateFileEvent(el,c->context->fd,AE_WRITABLE,writeHandler,c);


/* replication.c */
(aeCreateFileEvent(server.el, server.rdb_pipe_read, AE_READABLE,rdbPipeReadHandler,NULL); 

/* sentinel.c */
aeCreateFileEvent(loop,e->fd,AE_READABLE,redisAeReadEvent,e); 
aeCreateFileEvent(loop,e->fd,AE_WRITABLE,redisAeWriteEvent,e);


/* server.c */
aeCreateFileEvent(server.el, server.ipfd[j], AE_READABLE,acceptTcpHandler,NULL);
aeCreateFileEvent(server.el, server.tlsfd[j], AE_READABLE,acceptTLSHandler,NULL);
aeCreateFileEvent(server.el,server.sofd,AE_READABLE,acceptUnixHandler,NULL);
aeCreateFileEvent(server.el,server.module_blocked_pipe[0],AE_READABLE,moduleBlockedClientPipeReadable,NULL);

2.2.4 读事件，写事件与时间事件的处理逻辑

epoll编程回顾

2.2.3.1 读事件处理-问题

1.关注acceptTcpHandler
void acceptTcpHandler(aeEventLoop *el, int fd, void *privdata, int mask) {
    int cport, cfd, max = MAX_ACCEPTS_PER_CALL;
    char cip[NET_IP_STR_LEN];
    UNUSED(el);
    UNUSED(mask);
    UNUSED(privdata);

    while(max--) {
        /* 返回客户端连接的套接字 */
        cfd = anetTcpAccept(server.neterr, fd, cip, sizeof(cip), &cport); 
        if (cfd == ANET_ERR) {
            if (errno != EWOULDBLOCK)
                serverLog(LL_WARNING,
                    "Accepting client connection: %s", server.neterr);
            return;
        }
        serverLog(LL_VERBOSE,"Accepted %s:%d", cip, cport);
        /*
          首先关注connCreateAcceptedSocket(cfd)；
          再关注acceptCommonHandler.
        */
        acceptCommonHandler(connCreateAcceptedSocket(cfd),0,cip);
    }
}


2.关注anetTcpAccept，返回连接上的客户端的套接字

2.1 关注anetTcpAccept的实现：
int anetTcpAccept(char *err, int s, char *ip, size_t ip_len, int *port) {
    int fd;
    struct sockaddr_storage sa;
    socklen_t salen = sizeof(sa);
    if ((fd = anetGenericAccept(err,s,(struct sockaddr*)&sa,&salen)) == -1)
        return ANET_ERR;

    if (sa.ss_family == AF_INET) {
        struct sockaddr_in *s = (struct sockaddr_in *)&sa;
        if (ip) inet_ntop(AF_INET,(void*)&(s->sin_addr),ip,ip_len);
        if (port) *port = ntohs(s->sin_port);
    } else {
        struct sockaddr_in6 *s = (struct sockaddr_in6 *)&sa;
        if (ip) inet_ntop(AF_INET6,(void*)&(s->sin6_addr),ip,ip_len);
        if (port) *port = ntohs(s->sin6_port);
    }
    return fd;
}

2.2 关注子步骤anetGenericAccept的实现(这里实现了socket的accept)
static int anetGenericAccept(char *err, int s, struct sockaddr *sa, socklen_t *len) {
    int fd;
    while(1) {
        fd = accept(s,sa,len);
        if (fd == -1) {
            if (errno == EINTR)
                continue;
            else {
                anetSetError(err, "accept: %s", strerror(errno));
                return ANET_ERR;
            }
        }
        break;
    }
    return fd;
}




3.关注connCreateAcceptedSocket(cfd),函数得到一个connection类型的指针,其工作为：
  (1)执行connCreateSocket(),将conn->type指向CT_Socket这个变量，从而得到CT_Socket所拥有的的 
     一切特性,其实CT_Socket所拥有的就是一系列的操作函数;
  (2)将fd赋值给链接的fd；
  (3)将链接的state设置为CONN_STATE_ACCEPTING；
  (4)返回conn.

connection *connCreateAcceptedSocket(int fd) {
    connection *conn = connCreateSocket();
    conn->fd = fd;
    conn->state = CONN_STATE_ACCEPTING;
    return conn;
}


connection *connCreateSocket() {
    connection *conn = zcalloc(sizeof(connection)); // 申请内存
    conn->type = &CT_Socket; // 初始化type的值
    conn->fd = -1;           // 初始化fd的值
    return conn;             // 返回conn
}

// struct connection 的定义
typedef struct connection connection;
struct connection {
    ConnectionType *type; // type指向一系列操作struct connection实例对象的函数
    ConnectionState state;
    short int flags;
    short int refs;
    int last_errno;
    void *private_data;
    ConnectionCallbackFunc conn_handler;
    ConnectionCallbackFunc write_handler;
    ConnectionCallbackFunc read_handler;
    int fd;
};


// ConnectionType 的定义
typedef struct ConnectionType {
    void (*ae_handler)(struct aeEventLoop *el, int fd, void *clientData, int mask);
    int (*connect)(struct connection *conn, const char *addr, int port, const char *source_addr, ConnectionCallbackFunc connect_handler);
    int (*write)(struct connection *conn, const void *data, size_t data_len);
    int (*read)(struct connection *conn, void *buf, size_t buf_len);
    void (*close)(struct connection *conn);
    int (*accept)(struct connection *conn, ConnectionCallbackFunc accept_handler);
    int (*set_write_handler)(struct connection *conn, ConnectionCallbackFunc handler, int barrier);
    int (*set_read_handler)(struct connection *conn, ConnectionCallbackFunc handler);
    const char *(*get_last_error)(struct connection *conn);
    int (*blocking_connect)(struct connection *conn, const char *addr, int port, long long timeout);
    ssize_t (*sync_write)(struct connection *conn, char *ptr, ssize_t size, long long timeout);
    ssize_t (*sync_read)(struct connection *conn, char *ptr, ssize_t size, long long timeout);
    ssize_t (*sync_readline)(struct connection *conn, char *ptr, ssize_t size, long long timeout);
    int (*get_type)(struct connection *conn);
} ConnectionType;

// CT_Socket 变量的初始化
ConnectionType CT_Socket = {
    .ae_handler = connSocketEventHandler,          // 被下面三处注册为回调函数，下面有实现
    .close = connSocketClose,
    .write = connSocketWrite,
    .read = connSocketRead,
    .accept = connSocketAccept,  
    .connect = connSocketConnect,                   // 注册了回调函数
    .set_write_handler = connSocketSetWriteHandler, // 注册了回调函数
    .set_read_handler = connSocketSetReadHandler,   // 注册了回调函数
    .get_last_error = connSocketGetLastError,
    .blocking_connect = connSocketBlockingConnect,
    .sync_write = connSocketSyncWrite,
    .sync_read = connSocketSyncRead,
    .sync_readline = connSocketSyncReadLine,
    .get_type = connSocketGetType
};

/* 
注意：
(1)connSocketConnect，connSocketSetWriteHandler
调用了aeCreateFileEvent(server.el, conn->fd, AE_WRITABLE,conn->type->ae_handler, conn); 

(2)connSocketSetReadHandler
调用了aeCreateFileEvent(server.el,conn->fd,AE_READABLE,conn->type->ae_handler,conn)

*/

如果有一个struct connection类型的指针变量为conn，那么它就会有如下的操作函数：
conn->type->ae_handler
conn->type->close 
conn->type->write 
conn->type->read 
conn->type->accept 
conn->type->connect 
conn->type->set_write_handler 
conn->type->set_read_handler 
conn->type->get_last_error 
conn->type->blocking_connect 
conn->type->sync_write 
conn->type->sync_read 
conn->type->sync_readline 
conn->type->get_type 




/* 函数connSocketEventHandler的实现 */
static void connSocketEventHandler(struct aeEventLoop *el, int fd, void *clientData, int mask)
{
    UNUSED(el);
    UNUSED(fd);
    connection *conn = clientData;

    if (conn->state == CONN_STATE_CONNECTING &&
            (mask & AE_WRITABLE) && conn->conn_handler) {

        if (connGetSocketError(conn)) {
            conn->last_errno = errno;
            conn->state = CONN_STATE_ERROR;
        } else {
            conn->state = CONN_STATE_CONNECTED;
        }

        if (!conn->write_handler) aeDeleteFileEvent(server.el,conn->fd,AE_WRITABLE);

        if (!callHandler(conn, conn->conn_handler)) return;
        conn->conn_handler = NULL;
    }

    /* Normally we execute the readable event first, and the writable
     * event later. This is useful as sometimes we may be able
     * to serve the reply of a query immediately after processing the
     * query.
     *
     * However if WRITE_BARRIER is set in the mask, our application is
     * asking us to do the reverse: never fire the writable event
     * after the readable. In such a case, we invert the calls.
     * This is useful when, for instance, we want to do things
     * in the beforeSleep() hook, like fsync'ing a file to disk,
     * before replying to a client. */
    int invert = conn->flags & CONN_FLAG_WRITE_BARRIER;

    int call_write = (mask & AE_WRITABLE) && conn->write_handler;
    int call_read = (mask & AE_READABLE) && conn->read_handler;

    /* Handle normal I/O flows */
    if (!invert && call_read) {
        if (!callHandler(conn, conn->read_handler)) return;
    }
    /* Fire the writable event. */
    if (call_write) {
        if (!callHandler(conn, conn->write_handler)) return;
    }
    /* If we have to invert the call, fire the readable event now
     * after the writable one. */
    if (invert && call_read) {
        if (!callHandler(conn, conn->read_handler)) return;
    }
}



/* Register a write handler, to be called when the connection is writable.
 * If NULL, the existing handler is removed.
 *
 * The barrier flag indicates a write barrier is requested, resulting with
 * CONN_FLAG_WRITE_BARRIER set. This will ensure that the write handler is
 * always called before and not after the read handler in a single event
 * loop.
 */
static int connSocketSetWriteHandler(connection *conn, ConnectionCallbackFunc func, int barrier) {
    if (func == conn->write_handler) return C_OK;

    conn->write_handler = func;
    if (barrier)
        conn->flags |= CONN_FLAG_WRITE_BARRIER;
    else
        conn->flags &= ~CONN_FLAG_WRITE_BARRIER;
    if (!conn->write_handler)
        aeDeleteFileEvent(server.el,conn->fd,AE_WRITABLE);
    else
        if (aeCreateFileEvent(server.el,conn->fd,AE_WRITABLE,
                    conn->type->ae_handler,conn) == AE_ERR) return C_ERR;
    return C_OK;
}


/* Register a read handler, to be called when the connection is readable.
 * If NULL, the existing handler is removed.
 */
static int connSocketSetReadHandler(connection *conn, ConnectionCallbackFunc func) {
    if (func == conn->read_handler) return C_OK;

    conn->read_handler = func;
    if (!conn->read_handler)
        aeDeleteFileEvent(server.el,conn->fd,AE_READABLE);
    else
        if (aeCreateFileEvent(server.el,conn->fd,
                    AE_READABLE,conn->type->ae_handler,conn) == AE_ERR) return C_ERR;
    return C_OK;
}


4. 关注acceptCommonHandler(connCreateAcceptedSocket(cfd),0,cip);(在大步骤1中最后一步)
acceptCommonHandler的实现：
static void acceptCommonHandler(connection *conn, int flags, char *ip) {
    client *c;
    char conninfo[100];
    UNUSED(ip);

    if (connGetState(conn) != CONN_STATE_ACCEPTING) {
        serverLog(LL_VERBOSE,
            "Accepted client connection in error state: %s (conn: %s)",
            connGetLastError(conn),
            connGetInfo(conn, conninfo, sizeof(conninfo)));
        connClose(conn);
        return;
    }

    /* Limit the number of connections we take at the same time.
     *
     * Admission control will happen before a client is created and connAccept()
     * called, because we don't want to even start transport-level negotiation
     * if rejected. */
    if (listLength(server.clients) + getClusterConnectionsCount()
        >= server.maxclients)
    {
        char *err;
        if (server.cluster_enabled)
            err = "-ERR max number of clients + cluster "
                  "connections reached\r\n";
        else
            err = "-ERR max number of clients reached\r\n";

        /* That's a best effort error message, don't check write errors.
         * Note that for TLS connections, no handshake was done yet so nothing
         * is written and the connection will just drop. */
        if (connWrite(conn,err,strlen(err)) == -1) {
            /* Nothing to do, Just to avoid the warning... */
        }
        server.stat_rejected_conn++;
        connClose(conn);
        return;
    }

    /* Create connection and client */
    if ((c = createClient(conn)) == NULL) {
        serverLog(LL_WARNING,
            "Error registering fd event for the new client: %s (conn: %s)",
            connGetLastError(conn),
            connGetInfo(conn, conninfo, sizeof(conninfo)));
        connClose(conn); /* May be already closed, just ignore errors */
        return;
    }

    /* Last chance to keep flags */
    c->flags |= flags;

    /* Initiate accept.
     *
     * Note that connAccept() is free to do two things here:
     * 1. Call clientAcceptHandler() immediately;
     * 2. Schedule a future call to clientAcceptHandler().
     *
     * Because of that, we must do nothing else afterwards.
     */
    if (connAccept(conn, clientAcceptHandler) == C_ERR) {
        char conninfo[100];
        if (connGetState(conn) == CONN_STATE_ERROR)
            serverLog(LL_WARNING,
                    "Error accepting a client connection: %s (conn: %s)",
                    connGetLastError(conn), connGetInfo(conn, conninfo, sizeof(conninfo)));
        freeClient(connGetPrivateData(conn));
        return;
    }
}

2.2.3.2 写事件处理-问题

2.2.3.3 时间事件处理

/* Process time events */
static int processTimeEvents(aeEventLoop *eventLoop) {
    int processed = 0;
    aeTimeEvent *te;
    long long maxId;
    time_t now = time(NULL);

    /* If the system clock is moved to the future, and then set back to the
     * right value, time events may be delayed in a random way. Often this
     * means that scheduled operations will not be performed soon enough.
     *
     * Here we try to detect system clock skews, and force all the time
     * events to be processed ASAP when this happens: the idea is that
     * processing events earlier is less dangerous than delaying them
     * indefinitely, and practice suggests it is. */
    if (now < eventLoop->lastTime) {
        te = eventLoop->timeEventHead;
        while(te) {
            te->when_sec = 0;
            te = te->next;
        }
    }
    eventLoop->lastTime = now;

    te = eventLoop->timeEventHead;
    maxId = eventLoop->timeEventNextId-1;
    while(te) {
        long now_sec, now_ms;
        long long id;

        /* Remove events scheduled for deletion. */
        if (te->id == AE_DELETED_EVENT_ID) {
            aeTimeEvent *next = te->next;
            /* If a reference exists for this timer event,
             * don't free it. This is currently incremented
             * for recursive timerProc calls */
            if (te->refcount) {
                te = next;
                continue;
            }
            if (te->prev)
                te->prev->next = te->next;
            else
                eventLoop->timeEventHead = te->next;
            if (te->next)
                te->next->prev = te->prev;
            if (te->finalizerProc)
                te->finalizerProc(eventLoop, te->clientData);
            zfree(te);
            te = next;
            continue;
        }

        /* Make sure we don't process time events created by time events in
         * this iteration. Note that this check is currently useless: we always
         * add new timers on the head, however if we change the implementation
         * detail, this check may be useful again: we keep it here for future
         * defense. */
        if (te->id > maxId) {
            te = te->next;
            continue;
        }
        aeGetTime(&now_sec, &now_ms);
        if (now_sec > te->when_sec ||
            (now_sec == te->when_sec && now_ms >= te->when_ms))
        {
            int retval;

            id = te->id;
            te->refcount++;
            retval = te->timeProc(eventLoop, id, te->clientData);
            te->refcount--;
            processed++;
            if (retval != AE_NOMORE) {
                aeAddMillisecondsToNow(retval,&te->when_sec,&te->when_ms);
            } else {
                te->id = AE_DELETED_EVENT_ID;
            }
        }
        te = te->next;
    }
    return processed;
}