libevent源码剖析之bufferevent

1 简介

• bufferevent 是 libevent 中提供的一种高级 I/O 抽象，建立在 evbuffer 之上，用于管理 带缓冲的事件驱动 I/O。它简化了异步 I/O 的管理，使得开发者在处理网络数据时可以更加方便地进行读取、写入和处理错误。bufferevent 背后通过结合eventloop事件循环、evbuffer 以及底层 I/O 操作，形成一个结构化的 I/O 流管理方式。
• 有了evbuffer和bufferevent，开发者便可以专注于基于libevent的业务逻辑编写，不必再费心于网络细节的处理；
• 如果没有evbuffer和bufferevent的加持，libevent则退化为仅实现了reactor的事件驱动库，就如同另外一个开源事件驱动库--libev；

    bufferevent在libevent中所处层次： 

 bufferevent层次图

2 bufferevent 的功能

bufferevent 提供了以下几大功能：

1. 异步 I/O 操作：支持异步读写操作，不会阻塞主线程。
2. 双缓冲机制：分别维护读缓冲区和写缓冲区，读写操作使用独立的缓冲区 (evbuffer) 来管理数据。
3. 自动管理事件：自动处理 I/O 事件，如数据可读、可写、错误处理等，无需手动管理底层事件。
4. 读写超时：支持超时机制，用户可以设置读取或写入超时。
5. 高级控制：提供读写暂停、调整缓冲区大小等高级功能。

3 bufferevent 的核心数据结构

bufferevent 的内部数据结构主要由以下部分组成：

3.1 读写缓冲区

bufferevent 内部有两个 evbuffer，分别用于存储读取和写入的数据：

• input evbuffer：用于存放从底层读取的数据，供用户消费。
• output evbuffer：用于存放待写入的数据，供底层发送。

3.2 事件回调

每个 bufferevent 都有一组事件回调函数，用于处理不同的事件：

• 读事件回调：当有新数据可供读取时触发。
• 写事件回调：当数据已被成功写入到底层 I/O 时触发。
• 事件回调：用于处理如连接关闭、错误等事件。

3.3 底层 I/O 操作

底层的 I/O 操作可以是普通的文件描述符（如套接字），也可以是更高级的 I/O 模型（如 SSL/TLS），bufferevent 使用事件驱动方式来管理这些 I/O 操作。

34 超时管理

bufferevent 提供了读写超时功能，通过设置不同的超时参数，当读取或写入超时后，自动触发相应的超时处理逻辑。

4 bufferevent 的实现原理

bufferevent 的实现基于 事件驱动模型，并通过 evbuffer 进行缓冲管理。它将复杂的异步 I/O 逻辑封装在高级接口中，便于开发者高效地进行异步网络编程。

4.1 数据流动机制

• 读取数据：底层 I/O 事件触发时，bufferevent 会从文件描述符（如套接字）中读取数据并将其存入 input evbuffer，然后触发用户定义的 读回调。
• 写入数据：用户可以将数据添加到 output evbuffer 中，当 output evbuffer 有数据时，bufferevent 会自动监听底层 I/O 是否可写，并将缓冲区的数据写入文件描述符。

整个数据流动可以通过以下步骤总结：

1. 底层事件触发：当底层 I/O 事件（如 readable 或 writable）发生时，libevent 的事件循环会通知 bufferevent。
2. 数据传递：

   • 读取事件时，bufferevent 将数据从文件描述符读取到 input evbuffer。

   • 写入事件时，bufferevent 会将 output evbuffer 中的数据写入文件描述符。

3. 触发回调：当读取或写入完成后，bufferevent 会调用用户提供的回调函数，通知用户有新数据可处理或写入已完成。

4. 处理超时：如果在设定的时间内没有完成读取或写入，bufferevent 会触发相应的超时事件。

4.2 双缓冲机制

bufferevent 使用了两个独立的缓冲区（input evbuffer 和 output evbuffer）来管理读写数据：

• input evbuffer 负责存储从 I/O 读取到的数据，开发者可以通过 bufferevent_read 或者直接操作 evbuffer 来获取数据。
• output evbuffer 负责缓存需要发送的数据，开发者可以通过 bufferevent_write 将数据放入 output evbuffer，然后 libevent 会负责将这些数据发送到底层的文件描述符（如套接字）。

这种设计确保了读写操作互不影响，并且通过 evbuffer 的高效管理，避免了频繁的数据拷贝和内存操作。

4.3 事件与回调机制

bufferevent 支持三类事件回调：

• 读回调：当有新数据进入 input evbuffer 时，触发用户定义的读回调函数。
• 写回调：当 output evbuffer 中的数据被写入到底层 I/O 中时，触发写回调函数。
• 事件回调：用于处理连接关闭、错误或超时等事件。通过这个回调，用户可以处理诸如网络断开、连接成功等事件。

4.4 底层 I/O 模型

bufferevent 底层支持多种 I/O 模型，例如：

• 普通套接字：使用系统的 read 和 write 操作，也支持readv / write /writev / WSASend / sendfile。
• SSL/TLS 加密传输：libevent 提供了 bufferevent_openssl，专门处理加密的传输数据。
• bufferevent_pair：用于创建一对 bufferevent，彼此互通消息，类似于 UNIX 域套接字或管道的双向通信机制，可用于线程间通信&模拟网络套接字&进程内部通信。
• 其他类型：libevent 也允许用户自定义底层的 I/O 操作，可以通过继承 bufferevent 来实现。

5 源码剖析

5.1 bufferevent

    首先，让我们来认识下bufferevent结构体：

// 读写事件水位
struct event_watermark {
    // 低水位
	size_t low;
    // 高水位
	size_t high;
};

/**
  Shared implementation of a bufferevent.

  This type is exposed only because it was exposed in previous versions,
  and some people's code may rely on manipulating it.  Otherwise, you
  should really not rely on the layout, size, or contents of this structure:
  it is fairly volatile, and WILL change in future versions of the code.
**/
struct bufferevent {
	/** Event base for which this bufferevent was created. */
    // 此bufferevent所属event_base实例
	struct event_base *ev_base;
	/** Pointer to a table of function pointers to set up how this
	    bufferevent behaves. */
	const struct bufferevent_ops *be_ops;

	/** A read event that triggers when a timeout has happened or a socket
	    is ready to read data.  Only used by some subtypes of
	    bufferevent. */
    // 读事件
	struct event ev_read;
	/** A write event that triggers when a timeout has happened or a socket
	    is ready to write data.  Only used by some subtypes of
	    bufferevent. */
    // 写事件
	struct event ev_write;

	/** An input buffer. Only the bufferevent is allowed to add data to
	    this buffer, though the user is allowed to drain it. */
    // 读evbuffer
	struct evbuffer *input;

	/** An input buffer. Only the bufferevent is allowed to drain data
	    from this buffer, though the user is allowed to add it. */
    // 写evbuffer
	struct evbuffer *output;

    // 读写水位控制
	struct event_watermark wm_read;
	struct event_watermark wm_write;

    // 读事件就绪后回调
	bufferevent_data_cb readcb;
    // 写事件就绪后回调
	bufferevent_data_cb writecb;
	/* This should be called 'eventcb', but renaming it would break
	 * backward compatibility */
    // 关闭或错误事件回调
	bufferevent_event_cb errorcb;
    // 读写事件回调的回调参数
	void *cbarg;

    // 读超时时间
	struct timeval timeout_read;
    // 写超时时间
	struct timeval timeout_write;

	/** Events that are currently enabled: currently EV_READ and EV_WRITE
	    are supported. */
    // EV_READ和EV_WRITE属性
	short enabled;
};

    bufferevent_data_cb回调函数定义： 

/**
   A read or write callback for a bufferevent.

   The read callback is triggered when new data arrives in the input
   buffer and the amount of readable data exceed the low watermark
   which is 0 by default.

   The write callback is triggered if the write buffer has been
   exhausted or fell below its low watermark.

   @param bev the bufferevent that triggered the callback
   @param ctx the user-specified context for this bufferevent
 */
typedef void (*bufferevent_data_cb)(struct bufferevent *bev, void *ctx);

     bufferevent_event_cb回调定义：

/**
   An event/error callback for a bufferevent.

   The event callback is triggered if either an EOF condition or another
   unrecoverable error was encountered.

   For bufferevents with deferred callbacks, this is a bitwise OR of all errors
   that have happened on the bufferevent since the last callback invocation.

   @param bev the bufferevent for which the error condition was reached
   @param what a conjunction of flags: BEV_EVENT_READING or BEV_EVENT_WRITING
	  to indicate if the error was encountered on the read or write path,
	  and one of the following flags: BEV_EVENT_EOF, BEV_EVENT_ERROR,
	  BEV_EVENT_TIMEOUT, BEV_EVENT_CONNECTED.

   @param ctx the user-specified context for this bufferevent
*/
typedef void (*bufferevent_event_cb)(struct bufferevent *bev, short what, void *ctx);

    其中，bufferevent_ops是对bufferevent之bufferevent_socket / bufferevent_pair / bufferevent_asyn / buffevent_openssl / buffevent_ratelim / bufferevent_filter操作的抽象，再来看看bufferevent_ops结构体定义： 

/**
   Implementation table for a bufferevent: holds function pointers and other
   information to make the various bufferevent types work.
*/
struct bufferevent_ops {
	/** The name of the bufferevent's type. */
	const char *type;
	/** At what offset into the implementation type will we find a
	    bufferevent structure?

	    Example: if the type is implemented as
	    struct bufferevent_x {
	       int extra_data;
	       struct bufferevent bev;
	    }
	    then mem_offset should be offsetof(struct bufferevent_x, bev)
	*/
	off_t mem_offset;

	/** Enables one or more of EV_READ|EV_WRITE on a bufferevent.  Does
	    not need to adjust the 'enabled' field.  Returns 0 on success, -1
	    on failure.
	 */
    // 关注EV_READ|EV_WRITE事件
	int (*enable)(struct bufferevent *, short);

	/** Disables one or more of EV_READ|EV_WRITE on a bufferevent.  Does
	    not need to adjust the 'enabled' field.  Returns 0 on success, -1
	    on failure.
	 */
    // 取消关注EV_READ|EV_WRITE事件
	int (*disable)(struct bufferevent *, short);

	/** Detatches the bufferevent from related data structures. Called as
	 * soon as its reference count reaches 0. */
	void (*unlink)(struct bufferevent *);

	/** Free any storage and deallocate any extra data or structures used
	    in this implementation. Called when the bufferevent is
	    finalized.
	 */
    // 析构操作
	void (*destruct)(struct bufferevent *);

	/** Called when the timeouts on the bufferevent have changed.*/
    // 当bufferevent上的timeout发生变化时调用
	int (*adj_timeouts)(struct bufferevent *);

	/** Called to flush data. */
    // flush掉
	int (*flush)(struct bufferevent *, short, enum bufferevent_flush_mode);

	/** Called to access miscellaneous fields. */
    // 用来访问bufferevent的某些字段
	int (*ctrl)(struct bufferevent *, enum bufferevent_ctrl_op, union bufferevent_ctrl_data *);
};

     水位相关api接口：

// 挂起read读：暂时不读了
void
bufferevent_suspend_read_(struct bufferevent *bufev, bufferevent_suspend_flags what)
{
	struct bufferevent_private *bufev_private =
	    EVUTIL_UPCAST(bufev, struct bufferevent_private, bev);
	BEV_LOCK(bufev);
	if (!bufev_private->read_suspended)
		bufev->be_ops->disable(bufev, EV_READ);
	bufev_private->read_suspended |= what;
	BEV_UNLOCK(bufev);
}

// 恢复read读
void
bufferevent_unsuspend_read_(struct bufferevent *bufev, bufferevent_suspend_flags what)
{
	struct bufferevent_private *bufev_private =
	    EVUTIL_UPCAST(bufev, struct bufferevent_private, bev);
	BEV_LOCK(bufev);
	bufev_private->read_suspended &= ~what;
	if (!bufev_private->read_suspended && (bufev->enabled & EV_READ))
		bufev->be_ops->enable(bufev, EV_READ);
	BEV_UNLOCK(bufev);
}

// 挂起write写：暂时不写了
void
bufferevent_suspend_write_(struct bufferevent *bufev, bufferevent_suspend_flags what)
{
	struct bufferevent_private *bufev_private =
	    EVUTIL_UPCAST(bufev, struct bufferevent_private, bev);
	BEV_LOCK(bufev);
	if (!bufev_private->write_suspended)
		bufev->be_ops->disable(bufev, EV_WRITE);
	bufev_private->write_suspended |= what;
	BEV_UNLOCK(bufev);
}

// 恢复write写
void
bufferevent_unsuspend_write_(struct bufferevent *bufev, bufferevent_suspend_flags what)
{
	struct bufferevent_private *bufev_private =
	    EVUTIL_UPCAST(bufev, struct bufferevent_private, bev);
	BEV_LOCK(bufev);
	bufev_private->write_suspended &= ~what;
	if (!bufev_private->write_suspended && (bufev->enabled & EV_WRITE))
		bufev->be_ops->enable(bufev, EV_WRITE);
	BEV_UNLOCK(bufev);
}


/* Callback to implement watermarks on the input buffer.  Only enabled
 * if the watermark is set. */
// 设置读高水位回调，低于读高水位则恢复读
static void
bufferevent_inbuf_wm_cb(struct evbuffer *buf,
    const struct evbuffer_cb_info *cbinfo,
    void *arg)
{
	struct bufferevent *bufev = arg;
	size_t size;

	size = evbuffer_get_length(buf);

	if (size >= bufev->wm_read.high)
		bufferevent_wm_suspend_read(bufev);
	else
		bufferevent_wm_unsuspend_read(bufev);
}

/*
 * Sets the water marks
 */

void
bufferevent_setwatermark(struct bufferevent *bufev, short events,
    size_t lowmark, size_t highmark)
{
	struct bufferevent_private *bufev_private =
	    EVUTIL_UPCAST(bufev, struct bufferevent_private, bev);

	BEV_LOCK(bufev);
	if (events & EV_WRITE) {
		bufev->wm_write.low = lowmark;
		bufev->wm_write.high = highmark;
	}

	if (events & EV_READ) {
		bufev->wm_read.low = lowmark;
		bufev->wm_read.high = highmark;

		if (highmark) {
			/* There is now a new high-water mark for read.
			   enable the callback if needed, and see if we should
			   suspend/bufferevent_wm_unsuspend. */

			if (bufev_private->read_watermarks_cb == NULL) {
				bufev_private->read_watermarks_cb =
				    evbuffer_add_cb(bufev->input,
						    bufferevent_inbuf_wm_cb,
						    bufev);
			}
			evbuffer_cb_set_flags(bufev->input,
				      bufev_private->read_watermarks_cb,
				      EVBUFFER_CB_ENABLED|EVBUFFER_CB_NODEFER);

			if (evbuffer_get_length(bufev->input) >= highmark)
				bufferevent_wm_suspend_read(bufev);
			else if (evbuffer_get_length(bufev->input) < highmark)
				bufferevent_wm_unsuspend_read(bufev);
		} else {
			/* There is now no high-water mark for read. */
			if (bufev_private->read_watermarks_cb)
				evbuffer_cb_clear_flags(bufev->input,
				    bufev_private->read_watermarks_cb,
				    EVBUFFER_CB_ENABLED);
			bufferevent_wm_unsuspend_read(bufev);
		}
	}
	BEV_UNLOCK(bufev);
}

int
bufferevent_getwatermark(struct bufferevent *bufev, short events,
    size_t *lowmark, size_t *highmark)
{
	if (events == EV_WRITE) {
		BEV_LOCK(bufev);
		if (lowmark)
			*lowmark = bufev->wm_write.low;
		if (highmark)
			*highmark = bufev->wm_write.high;
		BEV_UNLOCK(bufev);
		return 0;
	}

	if (events == EV_READ) {
		BEV_LOCK(bufev);
		if (lowmark)
			*lowmark = bufev->wm_read.low;
		if (highmark)
			*highmark = bufev->wm_read.high;
		BEV_UNLOCK(bufev);
		return 0;
	}
	return -1;
}

 事件相关api：

// 线程安全的执行defer回调，包括read / write / event回调
static void
bufferevent_run_deferred_callbacks_locked(struct event_callback *cb, void *arg)
{
	struct bufferevent_private *bufev_private = arg;
	struct bufferevent *bufev = &bufev_private->bev;

	BEV_LOCK(bufev);
	if ((bufev_private->eventcb_pending & BEV_EVENT_CONNECTED) &&
	    bufev->errorcb) {
		/* The "connected" happened before any reads or writes, so
		   send it first. */
		bufev_private->eventcb_pending &= ~BEV_EVENT_CONNECTED;
		bufev->errorcb(bufev, BEV_EVENT_CONNECTED, bufev->cbarg);
	}
	if (bufev_private->readcb_pending && bufev->readcb) {
		bufev_private->readcb_pending = 0;
		bufev->readcb(bufev, bufev->cbarg);
	}
	if (bufev_private->writecb_pending && bufev->writecb) {
		bufev_private->writecb_pending = 0;
		bufev->writecb(bufev, bufev->cbarg);
	}
	if (bufev_private->eventcb_pending && bufev->errorcb) {
		short what = bufev_private->eventcb_pending;
		int err = bufev_private->errno_pending;
		bufev_private->eventcb_pending = 0;
		bufev_private->errno_pending = 0;
		EVUTIL_SET_SOCKET_ERROR(err);
		bufev->errorcb(bufev, what, bufev->cbarg);
	}
	bufferevent_decref_and_unlock_(bufev);
}

// 单线程版本：执行defer回调，包括read / write / event回调
static void
bufferevent_run_deferred_callbacks_unlocked(struct event_callback *cb, void *arg)
{
	struct bufferevent_private *bufev_private = arg;
	struct bufferevent *bufev = &bufev_private->bev;

	BEV_LOCK(bufev);
#define UNLOCKED(stmt) \
	do { BEV_UNLOCK(bufev); stmt; BEV_LOCK(bufev); } while(0)

	if ((bufev_private->eventcb_pending & BEV_EVENT_CONNECTED) &&
	    bufev->errorcb) {
		/* The "connected" happened before any reads or writes, so
		   send it first. */
		bufferevent_event_cb errorcb = bufev->errorcb;
		void *cbarg = bufev->cbarg;
		bufev_private->eventcb_pending &= ~BEV_EVENT_CONNECTED;
		UNLOCKED(errorcb(bufev, BEV_EVENT_CONNECTED, cbarg));
	}
	if (bufev_private->readcb_pending && bufev->readcb) {
		bufferevent_data_cb readcb = bufev->readcb;
		void *cbarg = bufev->cbarg;
		bufev_private->readcb_pending = 0;
		UNLOCKED(readcb(bufev, cbarg));
	}
	if (bufev_private->writecb_pending && bufev->writecb) {
		bufferevent_data_cb writecb = bufev->writecb;
		void *cbarg = bufev->cbarg;
		bufev_private->writecb_pending = 0;
		UNLOCKED(writecb(bufev, cbarg));
	}
	if (bufev_private->eventcb_pending && bufev->errorcb) {
		bufferevent_event_cb errorcb = bufev->errorcb;
		void *cbarg = bufev->cbarg;
		short what = bufev_private->eventcb_pending;
		int err = bufev_private->errno_pending;
		bufev_private->eventcb_pending = 0;
		bufev_private->errno_pending = 0;
		EVUTIL_SET_SOCKET_ERROR(err);
		UNLOCKED(errorcb(bufev,what,cbarg));
	}
	bufferevent_decref_and_unlock_(bufev);
#undef UNLOCKED
}

#define SCHEDULE_DEFERRED(bevp)						\
	do {								\
		if (event_deferred_cb_schedule_(			\
			    (bevp)->bev.ev_base,			\
			&(bevp)->deferred))				\
			bufferevent_incref_(&(bevp)->bev);		\
	} while (0)


// 运行readcb回调
void
bufferevent_run_readcb_(struct bufferevent *bufev, int options)
{
	/* Requires that we hold the lock and a reference */
	struct bufferevent_private *p =
	    EVUTIL_UPCAST(bufev, struct bufferevent_private, bev);
	if (bufev->readcb == NULL)
		return;
	if ((p->options|options) & BEV_OPT_DEFER_CALLBACKS) {
		p->readcb_pending = 1;
		SCHEDULE_DEFERRED(p);
	} else {
		bufev->readcb(bufev, bufev->cbarg);
	}
}

// 运行writecb回调
void
bufferevent_run_writecb_(struct bufferevent *bufev, int options)
{
	/* Requires that we hold the lock and a reference */
	struct bufferevent_private *p =
	    EVUTIL_UPCAST(bufev, struct bufferevent_private, bev);
	if (bufev->writecb == NULL)
		return;
	if ((p->options|options) & BEV_OPT_DEFER_CALLBACKS) {
		p->writecb_pending = 1;
		SCHEDULE_DEFERRED(p);
	} else {
		bufev->writecb(bufev, bufev->cbarg);
	}
}

#define BEV_TRIG_ALL_OPTS (			\
		BEV_TRIG_IGNORE_WATERMARKS|	\
		BEV_TRIG_DEFER_CALLBACKS	\
	)

// 此为对bufferevent_run_readcb_和bufferevent_run_writecb_的封装
void
bufferevent_trigger(struct bufferevent *bufev, short iotype, int options)
{
	bufferevent_incref_and_lock_(bufev);
	bufferevent_trigger_nolock_(bufev, iotype, options&BEV_TRIG_ALL_OPTS);
	bufferevent_decref_and_unlock_(bufev);
}

// 运行eventcb回调
void
bufferevent_run_eventcb_(struct bufferevent *bufev, short what, int options)
{
	/* Requires that we hold the lock and a reference */
	struct bufferevent_private *p =
	    EVUTIL_UPCAST(bufev, struct bufferevent_private, bev);
	if (bufev->errorcb == NULL)
		return;
	if ((p->options|options) & BEV_OPT_DEFER_CALLBACKS) {
		p->eventcb_pending |= what;
		p->errno_pending = EVUTIL_SOCKET_ERROR();
		SCHEDULE_DEFERRED(p);
	} else {
		bufev->errorcb(bufev, what, bufev->cbarg);
	}
}

// 此为对bufferevent_run_eventcb_的封装
void
bufferevent_trigger_event(struct bufferevent *bufev, short what, int options)
{
	bufferevent_incref_and_lock_(bufev);
	bufferevent_run_eventcb_(bufev, what, options&BEV_TRIG_ALL_OPTS);
	bufferevent_decref_and_unlock_(bufev);
}

// 对bufferevent的初始化
int
bufferevent_init_common_(struct bufferevent_private *bufev_private,
    struct event_base *base,
    consbufferevent_run_readcb_t struct bufferevent_ops *ops,
    enum bufferevent_options options)
{
	struct bufferevent *bufev = &bufev_private->bev;

    // 分配input evbuffer
	if (!bufev->input) {
		if ((bufev->input = evbuffer_new()) == NULL)
			return -1;
	}

    // 分配output evbuffer
	if (!bufev->output) {
		if ((bufev->output = evbuffer_new()) == NULL) {
			evbuffer_free(bufev->input);
			return -1;
		}
	}

	bufev_private->refcnt = 1;
	bufev->ev_base = base;

	/* Disable timeouts. */
	evutil_timerclear(&bufev->timeout_read);
	evutil_timerclear(&bufev->timeout_write);

	bufev->be_ops = ops;

	bufferevent_ratelim_init_(bufev_private);

	/*
	 * Set to EV_WRITE so that using bufferevent_write is going to
	 * trigger a callback.  Reading needs to be explicitly enabled
	 * because otherwise no data will be available.
	 */
	bufev->enabled = EV_WRITE;

#ifndef EVENT__DISABLE_THREAD_SUPPORT
	if (options & BEV_OPT_THREADSAFE) {
		if (bufferevent_enable_locking_(bufev, NULL) < 0) {
			/* cleanup */
			evbuffer_free(bufev->input);
			evbuffer_free(bufev->output);
			bufev->input = NULL;
			bufev->output = NULL;
			return -1;
		}
	}
#endif
	if ((options & (BEV_OPT_DEFER_CALLBACKS|BEV_OPT_UNLOCK_CALLBACKS))
	    == BEV_OPT_UNLOCK_CALLBACKS) {
		event_warnx("UNLOCK_CALLBACKS requires DEFER_CALLBACKS");
		return -1;
	}
    // 延迟回调初始化
	if (options & BEV_OPT_UNLOCK_CALLBACKS)
		event_deferred_cb_init_(
		    &bufev_private->deferred,
		    event_base_get_npriorities(base) / 2,
		    bufferevent_run_deferred_callbacks_unlocked,
		    bufev_private);
	else
		event_deferred_cb_init_(
		    &bufev_private->deferred,
		    event_base_get_npriorities(base) / 2,
		    bufferevent_run_deferred_callbacks_locked,
		    bufev_private);

	bufev_private->options = options;

	evbuffer_set_parent_(bufev->input, bufev);
	evbuffer_set_parent_(bufev->output, bufev);

	return 0;
}

// 给bufferevent设置读写事件回调
void
bufferevent_setcb(struct bufferevent *bufev,
    bufferevent_data_cb readcb, bufferevent_data_cb writecb,
    bufferevent_event_cb eventcb, void *cbarg)
{
	BEV_LOCK(bufev);

	bufev->readcb = readcb;
	bufev->writecb = writecb;
	bufev->errorcb = eventcb;

	bufev->cbarg = cbarg;
	BEV_UNLOCK(bufev);
}

// 获取bufferevent的读写事件回调函数指针
void
bufferevent_getcb(struct bufferevent *bufev,
    bufferevent_data_cb *readcb_ptr,
    bufferevent_data_cb *writecb_ptr,
    bufferevent_event_cb *eventcb_ptr,
    void **cbarg_ptr)
{
	BEV_LOCK(bufev);
	if (readcb_ptr)
		*readcb_ptr = bufev->readcb;
	if (writecb_ptr)
		*writecb_ptr = bufev->writecb;
	if (eventcb_ptr)
		*eventcb_ptr = bufev->errorcb;
	if (cbarg_ptr)
		*cbarg_ptr = bufev->cbarg;

	BEV_UNLOCK(bufev);
}

读写操作api:

// 写const void*数据接口
int
bufferevent_write(struct bufferevent *bufev, const void *data, size_t size)
{
	if (evbuffer_add(bufev->output, data, size) == -1)
		return (-1);

	return 0;
}

// 写evbuffer*数据接口
int
bufferevent_write_buffer(struct bufferevent *bufev, struct evbuffer *buf)
{
	if (evbuffer_add_buffer(bufev->output, buf) == -1)
		return (-1);

	return 0;
}

// 从evbuffer读size字节到data中
size_t
bufferevent_read(struct bufferevent *bufev, void *data, size_t size)
{
	return (evbuffer_remove(bufev->input, data, size));
}

// 将bufferevet中的evbuffer挪到buf中
int
bufferevent_read_buffer(struct bufferevent *bufev, struct evbuffer *buf)
{
	return (evbuffer_add_buffer(buf, bufev->input));
}

// 关注读
int
bufferevent_enable(struct bufferevent *bufev, short event)
{
	struct bufferevent_private *bufev_private =
	    EVUTIL_UPCAST(bufev, struct bufferevent_private, bev);
	short impl_events = event;
	int r = 0;

	bufferevent_incref_and_lock_(bufev);
	if (bufev_private->read_suspended)
		impl_events &= ~EV_READ;
	if (bufev_private->write_suspended)
		impl_events &= ~EV_WRITE;

	bufev->enabled |= event;

	if (impl_events && bufev->be_ops->enable(bufev, impl_events) < 0)
		r = -1;

	bufferevent_decref_and_unlock_(bufev);
	return r;
}

// 给读写操作设置超时时间
int
bufferevent_set_timeouts(struct bufferevent *bufev,
			 const struct timeval *tv_read,
			 const struct timeval *tv_write)
{
	int r = 0;
	BEV_LOCK(bufev);
	if (tv_read) {
		bufev->timeout_read = *tv_read;
	} else {
		evutil_timerclear(&bufev->timeout_read);
	}
	if (tv_write) {
		bufev->timeout_write = *tv_write;
	} else {
		evutil_timerclear(&bufev->timeout_write);
	}

	if (bufev->be_ops->adj_timeouts)
		r = bufev->be_ops->adj_timeouts(bufev);
	BEV_UNLOCK(bufev);

	return r;
}


/* Obsolete; use bufferevent_set_timeouts */
void
bufferevent_settimeout(struct bufferevent *bufev,
		       int timeout_read, int timeout_write)
{
	struct timeval tv_read, tv_write;
	struct timeval *ptv_read = NULL, *ptv_write = NULL;

	memset(&tv_read, 0, sizeof(tv_read));
	memset(&tv_write, 0, sizeof(tv_write));

	if (timeout_read) {
		tv_read.tv_sec = timeout_read;
		ptv_read = &tv_read;
	}
	if (timeout_write) {
		tv_write.tv_sec = timeout_write;
		ptv_write = &tv_write;
	}

	bufferevent_set_timeouts(bufev, ptv_read, ptv_write);
}


// 不再关注读了
int
bufferevent_disable_hard_(struct bufferevent *bufev, short event)
{
	int r = 0;
	struct bufferevent_private *bufev_private =
	    EVUTIL_UPCAST(bufev, struct bufferevent_private, bev);

	BEV_LOCK(bufev);
	bufev->enabled &= ~event;

	bufev_private->connecting = 0;
	if (bufev->be_ops->disable(bufev, event) < 0)
		r = -1;

	BEV_UNLOCK(bufev);
	return r;
}

// 不再关注读了
int
bufferevent_disable(struct bufferevent *bufev, short event)
{
	int r = 0;

	BEV_LOCK(bufev);
	bufev->enabled &= ~event;

	if (bufev->be_ops->disable(bufev, event) < 0)
		r = -1;

	BEV_UNLOCK(bufev);
	return r;
}

     关于bufferevent的几个主要api就暂时介绍至此。

5.2 buffevent_socket

// bufferevent_socket相关操作定义
const struct bufferevent_ops bufferevent_ops_socket = {
	"socket",
	evutil_offsetof(struct bufferevent_private, bev),
	be_socket_enable,
	be_socket_disable,
	NULL, /* unlink */
	be_socket_destruct,
	bufferevent_generic_adj_existing_timeouts_,
	be_socket_flush,
	be_socket_ctrl,
};

    socket地址相关接口实现： 

const struct sockaddr*
bufferevent_socket_get_conn_address_(struct bufferevent *bev)
{
	struct bufferevent_private *bev_p =
	    EVUTIL_UPCAST(bev, struct bufferevent_private, bev);

	return (struct sockaddr *)&bev_p->conn_address;
}
static void
bufferevent_socket_set_conn_address_fd(struct bufferevent_private *bev_p, int fd)
{
	socklen_t len = sizeof(bev_p->conn_address);

	struct sockaddr *addr = (struct sockaddr *)&bev_p->conn_address;
	if (addr->sa_family != AF_UNSPEC)
		getpeername(fd, addr, &len);
}
static void
bufferevent_socket_set_conn_address(struct bufferevent_private *bev_p,
	struct sockaddr *addr, size_t addrlen)
{
	EVUTIL_ASSERT(addrlen <= sizeof(bev_p->conn_address));
	memcpy(&bev_p->conn_address, addr, addrlen);
}

    读写操作接口：

// 此为bufferevent的write evbuffer添加的回调函数
// 作用：当开发者准备好了write数据内容，需要关注write事件以写入这波数据内容
static void
bufferevent_socket_outbuf_cb(struct evbuffer *buf,
    const struct evbuffer_cb_info *cbinfo,
    void *arg)
{
	struct bufferevent *bufev = arg;
	struct bufferevent_private *bufev_p =
	    EVUTIL_UPCAST(bufev, struct bufferevent_private, bev);

	if (cbinfo->n_added &&
	    (bufev->enabled & EV_WRITE) &&
	    !event_pending(&bufev->ev_write, EV_WRITE, NULL) &&
	    !bufev_p->write_suspended) {
		/* Somebody added data to the buffer, and we would like to
		 * write, and we were not writing.  So, start writing. */
		if (bufferevent_add_event_(&bufev->ev_write, &bufev->timeout_write) == -1) {
		    /* Should we log this? */
		}
	}
}

// 读事件就绪了，执行此回调从socket中读取数据
static void
bufferevent_readcb(evutil_socket_t fd, short event, void *arg)
{
	struct bufferevent *bufev = arg;
	struct bufferevent_private *bufev_p =
	    EVUTIL_UPCAST(bufev, struct bufferevent_private, bev);
	struct evbuffer *input;
	int res = 0;
	short what = BEV_EVENT_READING;
	ev_ssize_t howmuch = -1, readmax=-1;

	bufferevent_incref_and_lock_(bufev);

    // 读超时
	if (event == EV_TIMEOUT) {
		/* Note that we only check for event==EV_TIMEOUT. If
		 * event==EV_TIMEOUT|EV_READ, we can safely ignore the
		 * timeout, since a read has occurred */
		what |= BEV_EVENT_TIMEOUT;
		goto error;
	}

	input = bufev->input;

	/*
	 * If we have a high watermark configured then we don't want to
	 * read more data than would make us reach the watermark.
	 */
	if (bufev->wm_read.high != 0) {
		howmuch = bufev->wm_read.high - evbuffer_get_length(input);
		/* we somehow lowered the watermark, stop reading */
        // bufferevent的读evbuffer达到了高水位，暂停读
		if (howmuch <= 0) {
			bufferevent_wm_suspend_read(bufev);
			goto done;
		}
	}
	readmax = bufferevent_get_read_max_(bufev_p);
	if (howmuch < 0 || howmuch > readmax) /* The use of -1 for "unlimited"
					       * uglifies this code. XXXX */
		howmuch = readmax;
	if (bufev_p->read_suspended)
		goto done;

    // 先冻结读evbuffer
	evbuffer_unfreeze(input, 0);
    // 再从fd上读howmuch字节数据
	res = evbuffer_read(input, fd, (int)howmuch); /* XXXX evbuffer_read would do better to take and return ev_ssize_t */
    // 读完解冻
	evbuffer_freeze(input, 0);

	if (res == -1) {
		int err = evutil_socket_geterror(fd);
		if (EVUTIL_ERR_RW_RETRIABLE(err))
			goto reschedule;
		if (EVUTIL_ERR_CONNECT_REFUSED(err)) {
			bufev_p->connection_refused = 1;
			goto done;
		}
		/* error case */
		what |= BEV_EVENT_ERROR;
	} else if (res == 0) {
        // 对端close了write
		/* eof case */
		what |= BEV_EVENT_EOF;
	}

	if (res <= 0)
		goto error;

	bufferevent_decrement_read_buckets_(bufev_p, res);

	/* Invoke the user callback - must always be called last */
    // 执行读回调
	bufferevent_trigger_nolock_(bufev, EV_READ, 0);

	goto done;

 reschedule:
	goto done;

 error:
	bufferevent_disable(bufev, EV_READ);
	bufferevent_run_eventcb_(bufev, what, 0);

 done:
	bufferevent_decref_and_unlock_(bufev);
}

// 写事件就绪了，回调此函数具体执行对fd的write操作
static void
bufferevent_writecb(evutil_socket_t fd, short event, void *arg)
{
	struct bufferevent *bufev = arg;
	struct bufferevent_private *bufev_p =
	    EVUTIL_UPCAST(bufev, struct bufferevent_private, bev);
	int res = 0;
	short what = BEV_EVENT_WRITING;
	int connected = 0;
	ev_ssize_t atmost = -1;

	bufferevent_incref_and_lock_(bufev);

	if (event == EV_TIMEOUT) {
		/* Note that we only check for event==EV_TIMEOUT. If
		 * event==EV_TIMEOUT|EV_WRITE, we can safely ignore the
		 * timeout, since a read has occurred */
		what |= BEV_EVENT_TIMEOUT;
		goto error;
	}
    // connect事件其实是关注write事件，而不是其他类型事件
	if (bufev_p->connecting) {
		int c = evutil_socket_finished_connecting_(fd);
		/* we need to fake the error if the connection was refused
		 * immediately - usually connection to localhost on BSD */
		if (bufev_p->connection_refused) {
			bufev_p->connection_refused = 0;
			c = -1;
		}

		if (c == 0)
			goto done;

		bufev_p->connecting = 0;
		if (c < 0) {
			event_del(&bufev->ev_write);
			event_del(&bufev->ev_read);
			bufferevent_run_eventcb_(bufev, BEV_EVENT_ERROR, 0);
			goto done;
		} else {
			connected = 1;
			bufferevent_socket_set_conn_address_fd(bufev_p, fd);
#ifdef _WIN32
			if (BEV_IS_ASYNC(bufev)) {
				event_del(&bufev->ev_write);
				bufferevent_async_set_connected_(bufev);
				bufferevent_run_eventcb_(bufev,
						BEV_EVENT_CONNECTED, 0);
				goto done;
			}
#endif
            // 表示connect建立成功，执行connect成功回调
			bufferevent_run_eventcb_(bufev,
					BEV_EVENT_CONNECTED, 0);
			if (!(bufev->enabled & EV_WRITE) ||
			    bufev_p->write_suspended) {
				event_del(&bufev->ev_write);
				goto done;
			}
		}
	}

    // 计算bufferevent的写evbuffer中当前最大数据长度
	atmost = bufferevent_get_write_max_(bufev_p);

	if (bufev_p->write_suspended)
		goto done;

	if (evbuffer_get_length(bufev->output)) {
        // 具体执行向fd中写数据
		evbuffer_unfreeze(bufev->output, 1);
		res = evbuffer_write_atmost(bufev->output, fd, atmost);
		evbuffer_freeze(bufev->output, 1);
		if (res == -1) {
			int err = evutil_socket_geterror(fd);
			if (EVUTIL_ERR_RW_RETRIABLE(err))
				goto reschedule;
			what |= BEV_EVENT_ERROR;
		} else if (res == 0) {
			/* eof case
			   XXXX Actually, a 0 on write doesn't indicate
			   an EOF. An ECONNRESET might be more typical.
			 */
			what |= BEV_EVENT_EOF;
		}
		if (res <= 0)
			goto error;

		bufferevent_decrement_write_buckets_(bufev_p, res);
	}

    // 一旦bufferevent的写evbuffer中没有可写数据了，则暂停关注write事件
	if (evbuffer_get_length(bufev->output) == 0) {
		event_del(&bufev->ev_write);
	}

	/*
	 * Invoke the user callback if our buffer is drained or below the
	 * low watermark.
	 */
    // 写完毕后执行写回调，通知开发者
	if (res || !connected) {
		bufferevent_trigger_nolock_(bufev, EV_WRITE, 0);
	}

	goto done;

 reschedule:
	if (evbuffer_get_length(bufev->output) == 0) {
		event_del(&bufev->ev_write);
	}
	goto done;

 error:
	bufferevent_disable(bufev, EV_WRITE);
	bufferevent_run_eventcb_(bufev, what, 0);

 done:
	bufferevent_decref_and_unlock_(bufev);
}

 创建tcp客户端通信：

// 创建bufferevent实例，用以tcp客户端通信
struct bufferevent *
bufferevent_socket_new(struct event_base *base, evutil_socket_t fd,
    int options)
{
	struct bufferevent_private *bufev_p;
	struct bufferevent *bufev;

#ifdef _WIN32
	if (base && event_base_get_iocp_(base))
		return bufferevent_async_new_(base, fd, options);
#endif

    // 先创建bufferevent_private实例，再通过它拿到bufferevent实例
	if ((bufev_p = mm_calloc(1, sizeof(struct bufferevent_private)))== NULL)
		return NULL;
    // 对通用部分进行初始化操作
	if (bufferevent_init_common_(bufev_p, base, &bufferevent_ops_socket,
				    options) < 0) {
		mm_free(bufev_p);
		return NULL;
	}
    // 通过bufferevent_private拿到bufferevent实例
	bufev = &bufev_p->bev;
    // 此标志表示链式存储，必要时会用到socketpair
	evbuffer_set_flags(bufev->output, EVBUFFER_FLAG_DRAINS_TO_FD);

    // 关注读写事件，并设置对应回调
	event_assign(&bufev->ev_read, bufev->ev_base, fd,
	    EV_READ|EV_PERSIST|EV_FINALIZE, bufferevent_readcb, bufev);
	event_assign(&bufev->ev_write, bufev->ev_base, fd,
	    EV_WRITE|EV_PERSIST|EV_FINALIZE, bufferevent_writecb, bufev);

    // 关注bufferevent的写evbuffer水位，如果到写的低水位了，便暂停关注写
	evbuffer_add_cb(bufev->output, bufferevent_socket_outbuf_cb, bufev);

	evbuffer_freeze(bufev->input, 0);
	evbuffer_freeze(bufev->output, 1);

	return bufev;
}

// 发起tcp connect请求
int
bufferevent_socket_connect(struct bufferevent *bev,
    const struct sockaddr *sa, int socklen)
{
	struct bufferevent_private *bufev_p =
	    EVUTIL_UPCAST(bev, struct bufferevent_private, bev);

	evutil_socket_t fd;
	int r = 0;
	int result=-1;
	int ownfd = 0;

	bufferevent_incref_and_lock_(bev);

	if (!bufev_p)
		goto done;

	fd = bufferevent_getfd(bev);
	if (fd < 0) {
		if (!sa)
			goto done;
		fd = evutil_socket_(sa->sa_family,
		    SOCK_STREAM|EVUTIL_SOCK_NONBLOCK, 0);
		if (fd < 0)
			goto done;
		ownfd = 1;
	}
	if (sa) {
#ifdef _WIN32
		if (bufferevent_async_can_connect_(bev)) {
			bufferevent_setfd(bev, fd);
			r = bufferevent_async_connect_(bev, fd, sa, socklen);
			if (r < 0)
				goto freesock;
			bufev_p->connecting = 1;
			result = 0;
			goto done;
		} else
#endif
		r = evutil_socket_connect_(&fd, sa, socklen);
		if (r < 0)
			goto freesock;
	}
#ifdef _WIN32
	/* ConnectEx() isn't always around, even when IOCP is enabled.
	 * Here, we borrow the socket object's write handler to fall back
	 * on a non-blocking connect() when ConnectEx() is unavailable. */
	if (BEV_IS_ASYNC(bev)) {
		event_assign(&bev->ev_write, bev->ev_base, fd,
		    EV_WRITE|EV_PERSIST|EV_FINALIZE, bufferevent_writecb, bev);
	}
#endif
	bufferevent_setfd(bev, fd);
	if (r == 0) {
		if (! be_socket_enable(bev, EV_WRITE)) {
			bufev_p->connecting = 1;
			result = 0;
			goto done;
		}
	} else if (r == 1) {
		/* The connect succeeded already. How very BSD of it. */
		result = 0;
		bufev_p->connecting = 1;
		bufferevent_trigger_nolock_(bev, EV_WRITE, BEV_OPT_DEFER_CALLBACKS);
	} else {
		/* The connect failed already.  How very BSD of it. */
		result = 0;
		bufferevent_run_eventcb_(bev, BEV_EVENT_ERROR, BEV_OPT_DEFER_CALLBACKS);
		bufferevent_disable(bev, EV_WRITE|EV_READ);
	}

	goto done;

freesock:
	bufferevent_run_eventcb_(bev, BEV_EVENT_ERROR, 0);
	if (ownfd)
		evutil_closesocket(fd);
	/* do something about the error? */
done:
	bufferevent_decref_and_unlock_(bev);
	return result;
}

// 获取connect建连成功的地址信息
static void
bufferevent_connect_getaddrinfo_cb(int result, struct evutil_addrinfo *ai,
    void *arg)
{
	struct bufferevent *bev = arg;
	struct bufferevent_private *bev_p =
	    EVUTIL_UPCAST(bev, struct bufferevent_private, bev);
	int r;
	BEV_LOCK(bev);

	bufferevent_unsuspend_write_(bev, BEV_SUSPEND_LOOKUP);
	bufferevent_unsuspend_read_(bev, BEV_SUSPEND_LOOKUP);

	bev_p->dns_request = NULL;

	if (result == EVUTIL_EAI_CANCEL) {
		bev_p->dns_error = result;
		bufferevent_decref_and_unlock_(bev);
		return;
	}
	if (result != 0) {
		bev_p->dns_error = result;
		bufferevent_run_eventcb_(bev, BEV_EVENT_ERROR, 0);
		bufferevent_decref_and_unlock_(bev);
		if (ai)
			evutil_freeaddrinfo(ai);
		return;
	}

	/* XXX use the other addrinfos? */
	/* XXX use this return value */
	bufferevent_socket_set_conn_address(bev_p, ai->ai_addr, (int)ai->ai_addrlen);
	r = bufferevent_socket_connect(bev, ai->ai_addr, (int)ai->ai_addrlen);
	(void)r;
	bufferevent_decref_and_unlock_(bev);
	evutil_freeaddrinfo(ai);
}

// 这是根据host或ip来发起tcp的connect建连操作
int
bufferevent_socket_connect_hostname(struct bufferevent *bev,
    struct evdns_base *evdns_base, int family, const char *hostname, int port)
{
	char portbuf[10];
	struct evutil_addrinfo hint;
	struct bufferevent_private *bev_p =
	    EVUTIL_UPCAST(bev, struct bufferevent_private, bev);

	if (family != AF_INET && family != AF_INET6 && family != AF_UNSPEC)
		return -1;
	if (port < 1 || port > 65535)
		return -1;

	memset(&hint, 0, sizeof(hint));
	hint.ai_family = family;
	hint.ai_protocol = IPPROTO_TCP;
	hint.ai_socktype = SOCK_STREAM;

	evutil_snprintf(portbuf, sizeof(portbuf), "%d", port);

	BEV_LOCK(bev);
	bev_p->dns_error = 0;

	bufferevent_suspend_write_(bev, BEV_SUSPEND_LOOKUP);
	bufferevent_suspend_read_(bev, BEV_SUSPEND_LOOKUP);

	bufferevent_incref_(bev);
	bev_p->dns_request = evutil_getaddrinfo_async_(evdns_base, hostname,
	    portbuf, &hint, bufferevent_connect_getaddrinfo_cb, bev);
	BEV_UNLOCK(bev);

	return 0;
}

// 获取dns错误
int
bufferevent_socket_get_dns_error(struct bufferevent *bev)
{
	int rv;
	struct bufferevent_private *bev_p =
	    EVUTIL_UPCAST(bev, struct bufferevent_private, bev);

	BEV_LOCK(bev);
	rv = bev_p->dns_error;
	BEV_UNLOCK(bev);

	return rv;
}

    关于bufferevent_socket的几个重要api就介绍到此。 

5.3 bufferevent_async

    这个bufferevent的IO模型主要用于win32下IOCP多路复用，主要是用于实现IOCP客户端的。

// 对IOCP客户端实例进行初始化
struct bufferevent *
bufferevent_async_new_(struct event_base *base,
    evutil_socket_t fd, int options)
{
	struct bufferevent_async *bev_a;
	struct bufferevent *bev;
	struct event_iocp_port *iocp;

	options |= BEV_OPT_THREADSAFE;

    // 首先拿到IOCP完成端口
	if (!(iocp = event_base_get_iocp_(base)))
		return NULL;

    // 将此客户端fd与IOCP完成端口关联起来，才能用IOCP功能
	if (fd >= 0 && event_iocp_port_associate_(iocp, fd, 1)<0) {
		int err = GetLastError();
		/* We may have alrady associated this fd with a port.
		 * Let's hope it's this port, and that the error code
		 * for doing this neer changes. */
		if (err != ERROR_INVALID_PARAMETER)
			return NULL;
	}

    // 先分配一个bufferevent_async实例
	if (!(bev_a = mm_calloc(1, sizeof(struct bufferevent_async))))
		return NULL;
    // 拿到bufferevent实例 
	bev = &bev_a->bev.bev;
    // 创建IOCP客户端读evbuffer
	if (!(bev->input = evbuffer_overlapped_new_(fd))) {
		mm_free(bev_a);
		return NULL;
	}
    // 创建IOCP客户端写evbuffer
	if (!(bev->output = evbuffer_overlapped_new_(fd))) {
		evbuffer_free(bev->input);
		mm_free(bev_a);
		return NULL;
	}

    // 对evbuffer通用自动进行初始化
	if (bufferevent_init_common_(&bev_a->bev, base, &bufferevent_ops_async,
		options)<0)
		goto err;

    // 关注读写evbuffer的变化
	evbuffer_add_cb(bev->input, be_async_inbuf_callback, bev);
	evbuffer_add_cb(bev->output, be_async_outbuf_callback, bev);

    // 分别关注IOCP的connect/read/write完成事件
	event_overlapped_init_(&bev_a->connect_overlapped, connect_complete);
	event_overlapped_init_(&bev_a->read_overlapped, read_complete);
	event_overlapped_init_(&bev_a->write_overlapped, write_complete);

	bufferevent_init_generic_timeout_cbs_(bev);

	bev_a->ok = fd >= 0;

	return bev;
err:
	bufferevent_free(&bev_a->bev.bev);
	return NULL;
}

    发起IOCP客户端connect请求后，在完成了connect操作后，会回调到connect_complete，在此关注异步IO读写，并反馈connect成功给user侧： 

// connect完成回调
static void
connect_complete(struct event_overlapped *eo, ev_uintptr_t key,
    ev_ssize_t nbytes, int ok)
{
	struct bufferevent_async *bev_a = upcast_connect(eo);
	struct bufferevent *bev = &bev_a->bev.bev;
	evutil_socket_t sock;

	BEV_LOCK(bev);

	EVUTIL_ASSERT(bev_a->bev.connecting);
	bev_a->bev.connecting = 0;
	sock = evbuffer_overlapped_get_fd_(bev_a->bev.bev.input);
	/* XXXX Handle error? */
	setsockopt(sock, SOL_SOCKET, SO_UPDATE_CONNECT_CONTEXT, NULL, 0);

    // 连接成功，若已关注读写完成事件，则关注读写完成
	if (ok)
		bufferevent_async_set_connected_(bev);
	else
		bev_async_set_wsa_error(bev, eo);

    // connect成功回调：若设置了延迟回调选项，此通知将在eventloop线程中执行
	bufferevent_run_eventcb_(bev,
			ok? BEV_EVENT_CONNECTED : BEV_EVENT_ERROR, 0);

	event_base_del_virtual_(bev->ev_base);

	bufferevent_decref_and_unlock_(bev);
}

// IOCP读完成后回调
static void
read_complete(struct event_overlapped *eo, ev_uintptr_t key,
    ev_ssize_t nbytes, int ok)
{
	struct bufferevent_async *bev_a = upcast_read(eo);
	struct bufferevent *bev = &bev_a->bev.bev;
	short what = BEV_EVENT_READING;
	ev_ssize_t amount_unread;
	BEV_LOCK(bev);
	EVUTIL_ASSERT(bev_a->read_in_progress);

	amount_unread = bev_a->read_in_progress - nbytes;
    // 更新读evbuffer元数据
	evbuffer_commit_read_(bev->input, nbytes);
	bev_a->read_in_progress = 0;
	if (amount_unread)
		bufferevent_decrement_read_buckets_(&bev_a->bev, -amount_unread);

	if (!ok)
		bev_async_set_wsa_error(bev, eo);

	if (bev_a->ok) {
		if (ok && nbytes) {
			BEV_RESET_GENERIC_READ_TIMEOUT(bev);
            // 通知user侧，并继续关注读
			bufferevent_trigger_nolock_(bev, EV_READ, 0);
			bev_async_consider_reading(bev_a);
		} else if (!ok) {
			what |= BEV_EVENT_ERROR;
			bev_a->ok = 0;
            // 读的时候发生错误了
			bufferevent_run_eventcb_(bev, what, 0);
		} else if (!nbytes) {
            // 读完毕
			what |= BEV_EVENT_EOF;
			bev_a->ok = 0;
			bufferevent_run_eventcb_(bev, what, 0);
		}
	}

	bufferevent_decref_and_unlock_(bev);
}

// 写完成回调
static void
write_complete(struct event_overlapped *eo, ev_uintptr_t key,
    ev_ssize_t nbytes, int ok)
{
	struct bufferevent_async *bev_a = upcast_write(eo);
	struct bufferevent *bev = &bev_a->bev.bev;
	short what = BEV_EVENT_WRITING;
	ev_ssize_t amount_unwritten;

	BEV_LOCK(bev);
	EVUTIL_ASSERT(bev_a->write_in_progress);

	amount_unwritten = bev_a->write_in_progress - nbytes;
    // 写完成的数据可丢弃之
	evbuffer_commit_write_(bev->output, nbytes);
	bev_a->write_in_progress = 0;

	if (amount_unwritten)
		bufferevent_decrement_write_buckets_(&bev_a->bev,
		                                     -amount_unwritten);


	if (!ok)
		bev_async_set_wsa_error(bev, eo);

	if (bev_a->ok) {
		if (ok && nbytes) {
			BEV_RESET_GENERIC_WRITE_TIMEOUT(bev);
            // 写完成了，通知user侧
			bufferevent_trigger_nolock_(bev, EV_WRITE, 0);
            // 继续关注写完成
			bev_async_consider_writing(bev_a);
		} else if (!ok) {
            // 写的时候出错了
			what |= BEV_EVENT_ERROR;
			bev_a->ok = 0;
			bufferevent_run_eventcb_(bev, what, 0);
		} else if (!nbytes) {
            // 写完了
			what |= BEV_EVENT_EOF;
			bev_a->ok = 0;
			bufferevent_run_eventcb_(bev, what, 0);
		}
	}

	bufferevent_decref_and_unlock_(bev);
}

     关于bufferevent_async的实现，主要就是以上功能，介绍至此。

5.4 bufferevent_pair

    这个是模拟网络socket双信道通信，可在单进程内多线程之间通信之用。让咱们先来看一个例子：

#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <event2/event.h>
#include <event2/bufferevent.h>
#include <event2/buffer.h>
#include <unistd.h>

void read_cb(struct bufferevent *bev, void *ctx) {
    char message[256];
    struct evbuffer *input = bufferevent_get_input(bev);
    size_t length = evbuffer_get_length(input);
    
    if (length > 0) {
        evbuffer_remove(input, message, length);
        message[length] = '\0';
        printf("Received message: %s\n", message);
    }
}

void write_cb(struct bufferevent *bev, void *ctx) {
    printf("Data sent successfully.\n");
}

void event_cb(struct bufferevent *bev, short events, void *ctx) {
    if (events & BEV_EVENT_ERROR) {
        perror("Error on bufferevent");
    }
}

int main() {
    struct event_base *base;
    struct bufferevent *bev_pair[2];  // Two bufferevent for communication
    
    base = event_base_new();
    
    // Create a bufferevent pair
    if (bufferevent_pair_new(base, BEV_OPT_CLOSE_ON_FREE, bev_pair) == -1) {
        perror("Error creating bufferevent pair");
        return 1;
    }

    // Set callbacks for both bufferevents
    bufferevent_setcb(bev_pair[0], read_cb, write_cb, event_cb, NULL);
    bufferevent_setcb(bev_pair[1], read_cb, write_cb, event_cb, NULL);
    
    // Enable read and write events
    bufferevent_enable(bev_pair[0], EV_READ | EV_WRITE);
    bufferevent_enable(bev_pair[1], EV_READ | EV_WRITE);

    // Send data from bev_pair[0] to bev_pair[1]
    const char *message = "Hello from bufferevent_pair[0]!";
    bufferevent_write(bev_pair[0], message, strlen(message));

    // Run the event loop
    event_base_dispatch(base);
    
    // Free resources
    bufferevent_free(bev_pair[0]);
    bufferevent_free(bev_pair[1]);
    event_base_free(base);
    
    return 0;
}

    这个例子创建1对bufferevent，并分别注册读写event事件、同时关注读写事件，然后向bev_pai[0]写数据，bev_pair[1]便可读取所写的数据了。 

    接下来看下libevent是如何实现的。首先，创建1对bufferevent：

// 1对bufferevent结构定义
struct bufferevent_pair {
    // 必须拥有bufferevent_private实例
	struct bufferevent_private bev;
	struct bufferevent_pair *partner;
	/* For ->destruct() lock checking */
	struct bufferevent_pair *unlinked_partner;
};

static inline int
be_pair_wants_to_talk(struct bufferevent_pair *src,
    struct bufferevent_pair *dst)
{
	return (downcast(src)->enabled & EV_WRITE) &&
	    (downcast(dst)->enabled & EV_READ) &&
	    !dst->bev.read_suspended &&
	    evbuffer_get_length(downcast(src)->output);
}

// 此接口即是向一个bufferevent写数据
static void
be_pair_transfer(struct bufferevent *src, struct bufferevent *dst,
    int ignore_wm)
{
	size_t dst_size;
	size_t n;

	evbuffer_unfreeze(src->output, 1);
	evbuffer_unfreeze(dst->input, 0);

	if (dst->wm_read.high) {
		dst_size = evbuffer_get_length(dst->input);
		if (dst_size < dst->wm_read.high) {
			n = dst->wm_read.high - dst_size;
			evbuffer_remove_buffer(src->output, dst->input, n);
		} else {
			if (!ignore_wm)
				goto done;
			n = evbuffer_get_length(src->output);
			evbuffer_add_buffer(dst->input, src->output);
		}
	} else {
		n = evbuffer_get_length(src->output);
		evbuffer_add_buffer(dst->input, src->output);
	}

	if (n) {
		BEV_RESET_GENERIC_READ_TIMEOUT(dst);

		if (evbuffer_get_length(dst->output))
			BEV_RESET_GENERIC_WRITE_TIMEOUT(dst);
		else
			BEV_DEL_GENERIC_WRITE_TIMEOUT(dst);
	}

	bufferevent_trigger_nolock_(dst, EV_READ, 0);
	bufferevent_trigger_nolock_(src, EV_WRITE, 0);
done:
	evbuffer_freeze(src->output, 1);
	evbuffer_freeze(dst->input, 0);
}

static void
be_pair_outbuf_cb(struct evbuffer *outbuf,
    const struct evbuffer_cb_info *info, void *arg)
{
	struct bufferevent_pair *bev_pair = arg;
	struct bufferevent_pair *partner = bev_pair->partner;

	incref_and_lock(downcast(bev_pair));

	if (info->n_added > info->n_deleted && partner) {
		/* We got more data.  If the other side's reading, then
		   hand it over. */
		if (be_pair_wants_to_talk(bev_pair, partner)) {
			be_pair_transfer(downcast(bev_pair), downcast(partner), 0);
		}
	}

	decref_and_unlock(downcast(bev_pair));
}

static struct bufferevent_pair *
bufferevent_pair_elt_new(struct event_base *base,
    int options)
{
	struct bufferevent_pair *bufev;
	if (! (bufev = mm_calloc(1, sizeof(struct bufferevent_pair))))
		return NULL;
	if (bufferevent_init_common_(&bufev->bev, base, &bufferevent_ops_pair,
		options)) {
		mm_free(bufev);
		return NULL;
	}
    // 关注写evbuffer的变化
	if (!evbuffer_add_cb(bufev->bev.bev.output, be_pair_outbuf_cb, bufev)) {
		bufferevent_free(downcast(bufev));
		return NULL;
	}

	bufferevent_init_generic_timeout_cbs_(&bufev->bev.bev);

	return bufev;
}

int
bufferevent_pair_new(struct event_base *base, int options,
    struct bufferevent *pair[2])
{
	struct bufferevent_pair *bufev1 = NULL, *bufev2 = NULL;
	int tmp_options;

	options |= BEV_OPT_DEFER_CALLBACKS;
	tmp_options = options & ~BEV_OPT_THREADSAFE;

    // 创建1个bufferevent元素
	bufev1 = bufferevent_pair_elt_new(base, options);
	if (!bufev1)
		return -1;
	bufev2 = bufferevent_pair_elt_new(base, tmp_options);
	if (!bufev2) {
		bufferevent_free(downcast(bufev1));
		return -1;
	}

	if (options & BEV_OPT_THREADSAFE) {
		/*XXXX check return */
		bufferevent_enable_locking_(downcast(bufev2), bufev1->bev.lock);
	}

    // 你中有我，我中有你
	bufev1->partner = bufev2;
	bufev2->partner = bufev1;

	evbuffer_freeze(downcast(bufev1)->input, 0);
	evbuffer_freeze(downcast(bufev1)->output, 1);
	evbuffer_freeze(downcast(bufev2)->input, 0);
	evbuffer_freeze(downcast(bufev2)->output, 1);

    // 拿到bufferevent对返回
	pair[0] = downcast(bufev1);
	pair[1] = downcast(bufev2);

	return 0;
}

bufferevent_pair_get_partner 实现：

struct bufferevent *
bufferevent_pair_get_partner(struct bufferevent *bev)
{
	struct bufferevent_pair *bev_p;
	struct bufferevent *partner = NULL;
	bev_p = upcast(bev);
	if (! bev_p)
		return NULL;

	incref_and_lock(bev);
	if (bev_p->partner)
		partner = downcast(bev_p->partner);
	decref_and_unlock(bev);
	return partner;
}

 bufferevent_pair只有此2对外接口，用来模拟socket双信道，相对简单，不多赘述。

6 示例

6.1 简易例子

以下是一个简单的 bufferevent 使用示例，展示如何使用 bufferevent 来进行异步 I/O：

#include <event2/event.h>
#include <event2/bufferevent.h>
#include <stdio.h>
#include <string.h>
#include <arpa/inet.h>

// 读回调函数
void read_cb(struct bufferevent *bev, void *ctx) {
    char buf[256];
    int n = bufferevent_read(bev, buf, sizeof(buf));
    buf[n] = '\0';
    printf("Received: %s\n", buf);
}

// 事件回调函数
void event_cb(struct bufferevent *bev, short events, void *ctx) {
    if (events & BEV_EVENT_EOF) {
        printf("Connection closed.\n");
    } else if (events & BEV_EVENT_ERROR) {
        printf("Some error occurred.\n");
    }
    bufferevent_free(bev); // 释放 bufferevent
}

int main() {
    struct event_base *base = event_base_new();

    // 创建套接字并连接到服务器
    int fd = socket(AF_INET, SOCK_STREAM, 0);
    struct sockaddr_in sin;
    memset(&sin, 0, sizeof(sin));
    sin.sin_family = AF_INET;
    sin.sin_addr.s_addr = inet_addr("127.0.0.1");
    sin.sin_port = htons(12345);
    connect(fd, (struct sockaddr *)&sin, sizeof(sin));

    // 创建 bufferevent 并设置回调
    struct bufferevent *bev = bufferevent_socket_new(base, fd, BEV_OPT_CLOSE_ON_FREE);
    bufferevent_setcb(bev, read_cb, NULL, event_cb, NULL);
    bufferevent_enable(bev, EV_READ | EV_WRITE);

    // 启动事件循环
    event_base_dispatch(base);
    event_base_free(base);
    return 0;
}

6.2 客户端

    用bufferevent高级接口写一个tcp的客户端例子：

#include <event2/event.h>
#include <event2/bufferevent.h>
#include <event2/buffer.h>
#include <stdio.h>
#include <string.h>
#include <arpa/inet.h>

// 读取回调函数
void read_cb(struct bufferevent *bev, void *ctx) {
    char buf[512];
    int n = bufferevent_read(bev, buf, sizeof(buf) - 1);
    buf[n] = '\0';  // 保证字符串以 '\0' 结尾
    printf("Received: %s\n", buf);
}

// 写回调函数（当数据成功写入 socket 时触发）
void write_cb(struct bufferevent *bev, void *ctx) {
    printf("Data sent successfully.\n");
}

// 事件回调函数（处理连接关闭、错误等事件）
void event_cb(struct bufferevent *bev, short events, void *ctx) {
    if (events & BEV_EVENT_CONNECTED) {
        printf("Connected to the server.\n");

        // 连接成功后向服务器发送一条消息
        const char *msg = "Hello from client!";
        bufferevent_write(bev, msg, strlen(msg));
    } else if (events & BEV_EVENT_ERROR) {
        printf("Error encountered!\n");
        bufferevent_free(bev);  // 释放 bufferevent 资源
    } else if (events & BEV_EVENT_EOF) {
        printf("Connection closed by the server.\n");
        bufferevent_free(bev);
    }
}

// 主函数
int main(int argc, char **argv) {
    if (argc < 3) {
        printf("Usage: %s <server_ip> <server_port>\n", argv[0]);
        return -1;
    }

    const char *server_ip = argv[1];
    int server_port = atoi(argv[2]);

    // 创建一个 libevent 的事件基础设施
    struct event_base *base = event_base_new();
    if (!base) {
        fprintf(stderr, "Could not initialize libevent!\n");
        return -1;
    }

    // 创建套接字并初始化服务器地址
    struct sockaddr_in server_addr;
    memset(&server_addr, 0, sizeof(server_addr));
    server_addr.sin_family = AF_INET;
    server_addr.sin_addr.s_addr = inet_addr(server_ip);
    server_addr.sin_port = htons(server_port);

    // 创建 bufferevent，并设置为非阻塞模式
    struct bufferevent *bev = bufferevent_socket_new(base, -1, BEV_OPT_CLOSE_ON_FREE);
    if (!bev) {
        fprintf(stderr, "Error creating bufferevent!\n");
        event_base_free(base);
        return -1;
    }

    // 设置回调函数
    bufferevent_setcb(bev, read_cb, write_cb, event_cb, NULL);
    bufferevent_enable(bev, EV_READ | EV_WRITE);  // 启用读写事件

    // 发起非阻塞的连接
    if (bufferevent_socket_connect(bev, (struct sockaddr *)&server_addr, sizeof(server_addr)) < 0) {
        fprintf(stderr, "Error starting connection.\n");
        bufferevent_free(bev);
        event_base_free(base);
        return -1;
    }

    // 启动事件循环，处理所有事件
    event_base_dispatch(base);

    // 清理资源
    event_base_free(base);
    return 0;
}

 6.3 服务端

    再用bufferevent写一个简易的tcp服务端程序：

#include <event2/event.h>
#include <event2/bufferevent.h>
#include <event2/listener.h>
#include <event2/buffer.h>
#include <stdio.h>
#include <string.h>
#include <arpa/inet.h>

// 读取回调函数
void read_cb(struct bufferevent *bev, void *ctx) {
    char buf[512];
    int n = bufferevent_read(bev, buf, sizeof(buf) - 1); // 读取数据
    buf[n] = '\0';  // 确保字符串以 '\0' 结尾
    printf("Received message: %s\n", buf);

    // 回送收到的消息给客户端
    bufferevent_write(bev, buf, n);
}

// 事件回调函数（处理错误、关闭等事件）
void event_cb(struct bufferevent *bev, short events, void *ctx) {
    if (events & BEV_EVENT_ERROR) {
        perror("Error from bufferevent");
    }
    if (events & (BEV_EVENT_EOF | BEV_EVENT_ERROR)) {
        bufferevent_free(bev);  // 释放 bufferevent
        printf("Connection closed.\n");
    }
}

// 监听连接回调函数，当客户端连接时调用
void accept_conn_cb(struct evconnlistener *listener, evutil_socket_t fd, struct sockaddr *addr, int socklen, void *ctx) {
    printf("New connection accepted.\n");

    struct event_base *base = evconnlistener_get_base(listener);

    // 为新连接创建 bufferevent
    struct bufferevent *bev = bufferevent_socket_new(base, fd, BEV_OPT_CLOSE_ON_FREE);

    // 设置回调函数
    bufferevent_setcb(bev, read_cb, NULL, event_cb, NULL);
    bufferevent_enable(bev, EV_READ | EV_WRITE);  // 启用读写事件
}

// 监听错误回调函数
void accept_error_cb(struct evconnlistener *listener, void *ctx) {
    struct event_base *base = evconnlistener_get_base(listener);
    int err = EVUTIL_SOCKET_ERROR();
    fprintf(stderr, "Got an error %d (%s) on the listener. Shutting down.\n",
            err, evutil_socket_error_to_string(err));

    // 停止事件循环
    event_base_loopexit(base, NULL);
}

int main(int argc, char **argv) {
    if (argc < 2) {
        printf("Usage: %s <port>\n", argv[0]);
        return -1;
    }

    int port = atoi(argv[1]);

    // 创建 event_base
    struct event_base *base = event_base_new();
    if (!base) {
        fprintf(stderr, "Could not initialize libevent!\n");
        return -1;
    }

    // 创建 sockaddr_in 结构体，用于监听指定端口
    struct sockaddr_in sin;
    memset(&sin, 0, sizeof(sin));
    sin.sin_family = AF_INET;
    sin.sin_addr.s_addr = htonl(INADDR_ANY);  // 监听所有本地 IP 地址
    sin.sin_port = htons(port);

    // 创建 evconnlistener，用于监听传入连接
    struct evconnlistener *listener = evconnlistener_new_bind(
        base, accept_conn_cb, NULL,
        LEV_OPT_CLOSE_ON_FREE | LEV_OPT_REUSEABLE, -1,
        (struct sockaddr*)&sin, sizeof(sin));

    if (!listener) {
        perror("Could not create a listener");
        event_base_free(base);
        return -1;
    }

    // 设置错误回调
    evconnlistener_set_error_cb(listener, accept_error_cb);

    // 启动事件循环，等待连接和处理事件
    event_base_dispatch(base);

    // 释放资源
    evconnlistener_free(listener);
    event_base_free(base);

    return 0;
}

7 小结 

bufferevent 是 libevent 中用于高效管理异步 I/O 的高级接口，它通过事件驱动机制和双缓冲设计，大大简化了异步 I/O 操作的复杂性。

    bufferevent 的实现优势：

• 缓冲管理：通过 evbuffer 实现高效的读写缓冲管理，减少数据拷贝操作。
• 异步 I/O 简化：通过封装底层的事件循环与 I/O 操作，用户可以专注于处理高层逻辑，而无需关心具体的异步 I/O 操作。
• 高效的事件驱动模型：bufferevent 通过事件驱动模型实现了异步 I/O，使得它可以轻松处理高并发的网络连接。
• 灵活性：支持多种底层 I/O 模型，如 SSL/TLS 和自定义 I/O，使得 bufferevent 可以应对不同的应用场景。