python selectors模块使用

selectors — High-level I/O multiplexing

SelectorKeys对象

用于关联file object和其底层fd的namedtuple,如：

SelectorKey(fileobj=<socket.socket fd=5, family=AddressFamily.AF_INET, type=2049, proto=0, laddr=('192.168.18.129', 35990), raddr=('23.23.154.131', 80)>, fd=5, events=2, data=<bound method Select.connected of <__main__.Select object at 0x7f4477694d30>>)

对象属性

• 1 fileobj：被注册的文件对象
• 2 fd : 文件对象的基础文件描述符，整型。等同fileobj.fileno()
• 3 events: 需要等待的fileobj的事件(读或写事件)
• 4 data: 和fileobj关联的opaque object，一半是函数名。主要用于回调

该对象可由BaseSelector的几个方法返回:

• 1 register(fileobj, events, data=None)
• 2 unregister(fileobj）
• 3 select()
• 4 get_key(fileobj)

Select 对象

method说明

• 1 register(fileobj, events, data=None)：
  注册fileobj,监控fileobje的I/O事件;返回与fileobje关联的SelectorKey对象
• 2 unregister(fileobj):
  注销一个filobj,取消对它的监控。注销fileobj之前应先close();返回返回与fileobje关联的SelectorKey对象
• 3 select()：
  
  等待注册的fileobje的读或写事件准备好，returns a list of (key, events) tuples, one for each ready file object
• 4 fileno(): returns the file descriptor of selector,selector对象本身也是file object
• 5 get_map():返回file object对象和其关联的SelectorKey 的映射，key:file object的fd; value: 关联的SelectorKey

使用selector的I/O multiplexing 实现单线程并发

什么是I/O multiplexing ?

首先，网络上称它为“多路复用I/O”，但我认为I/O多路转接更好。
selctor本身是一个file object,它存储了多个已经注册为读或写事件的文件句柄，一旦某个文件句柄准备就绪，就可通知用户程序执行I/O操作。这种利用selector一个文件对象转换多个文件句柄的I/O操作的方式就是某种意义上的“复用”

协程的使用

#!/usr/bin/python3

import socket
from urllib.parse import urlparse
# DefaultSelector默认选择器，使用当前平台上可用的最高效的实现(select/poll/epoll)
from selectors import DefaultSelector, EVENT_READ, EVENT_WRITE


class Client(object):
    '''负责连接、 发送和接受'''
    def __init__(self, url):
        url = urlparse(url)
        self.host, self.path = url.netloc, [url.path, '/'][url.path == '']
        self.sock = socket.socket()
        self.sock.setblocking(False)
        self.data = b''
        try:
            self.sock.connect((self.host, 80))
        except BlockingIOError as ret:  # noqa
            pass

    def send_request(self):
        self.sock.send(
            "GET {} HTTP/1.1\r\nHost: {}\r\nConnection:close\r\n\r\n".format(
                self.path, self.host).encode('utf-8'))

    def recv_response(self):
        rev = self.sock.recv(1024)
        if rev:
            self.data += rev
            return True
        return False

    def end(self):
        html_body = self.data.split(b'\r\n\r\n')[1]
        self.sock.close()
        print(html_body)
        print('\n')
        return html_body


'''select + callback + eventLoop：单线程并发
只处理准备好socket事件，无需等待I/O;省去了线程切换的开销
回调模式:底层调用高层'''


class Select(object):
    '''创建selector,负责注册和取消注册socket读写事件、'''

    def __init__(self):
        self.selector = DefaultSelector()
        self.clients = {}

    def startListening(self, client):
        # add client,monitor socked wtite event
        self.clients[client.sock] = client
        # 注册socket写事件
        self.selector.register(client.sock, EVENT_WRITE, self.connected)

    def connected(self, key):
        # connect host,and monitor socket read event
        self.selector.unregister(key.fileobj)
        sock = key.fileobj
        self.clients[sock].send_request()
        self.selector.register(sock, EVENT_READ, self.do_read)

    def do_read(self, key):
        # read data ,if data is empty, then unregister socket and end
        sock = key.fileobj
        client = self.clients[sock]
        if client.recv_response():
            return
        self.selector.unregister(sock)
        client.end()


def event_loop():
    while True:
        events = sel.selector.select()
        for key, mask in events:
            print(key, mask)
            callback = key.data
            callback(key)


if __name__ == '__main__':
    urls = ['http://www.baidu.com', 'http://httpbin.org/']
    sel = Select()
    for url in urls:
        sel.startListening(Client(url))
    event_loop()
    sel.selector.close()