【代码】基于mmap和fcntl的一写多读实现

最新推荐文章于 2021-01-31 18:07:30 发布

原创最新推荐文章于 2021-01-31 18:07:30 发布 · 351 阅读

1 ·

CC 4.0 BY-SA版权

本文探讨如何结合mmap的跨进程访问优势和fcntl的共享内存保护锁，来实现一写多读的机制。虽然mmap与fcntl在某些情况下可能不兼容，但通过简单的锁文件策略，同样可以达成目标。文中提到了消息发布者和订阅者的应用场景。

mmap 用于跨任意进程快速访问，
fcntl 用于共享内存区域的保护锁（读共享，写互斥）
二者可能不能共用： Shared mmap co-ordination using fcntl locks?

所以单独用锁文件就好了，简单有效

message publisher

import mmap

try:
    import fcntl
except ImportError:
    fcntl = None
from time import sleep, time

with open('db', "w") as f:
    for i in range(1000):
        f.write(1 * 1024 * 1024 * '\0')
    f.close()
end = "a".encode() * 200 * 1024 * 1024

with open('db', "a+") as f:
    m = mmap.mmap(f.fileno(), 0, access=mmap.ACCESS_WRITE)
    j = 0
    while True:
        f.seek(0, 0)
        with open('.db_lock', 'w') as lock:
            fcntl.lockf(lock, fcntl.LOCK_EX)
            ss = [str(i * j).encode() + b', ' for i in range(10)]
            write_length = str(sum(len(s1) for s1 in ss))
            t0 = time()
            m.seek(0, 0)
            m.write(write_length.encode())
            m.write(b':')
            for i in range(10):
                m.write(ss[i])
                sleep(0.5)
            m.write(end)
            t1 = time()
            print(f'write: {write_length}, cost{t1-t0}')

        j += 1
        j %= 10

message subscriber

import mmap
from time import sleep, time

try:
    import fcntl
except ImportError:
    fcntl = None

with open('db', "rb+") as f:
    m = mmap.mmap(f.fileno(), 0, access=mmap.ACCESS_READ)
    line = None
    while True:
        m.seek(0, 0)
        with open('.db_lock', 'w+') as lock:
            fcntl.lockf(lock, fcntl.LOCK_EX | fcntl.LOCK_SH)
            t0 = time()
            size_pos = m.find(b':')
            m.seek(0, 0)
            use_length = int(m.read(size_pos))
            m.seek(1, 1)
            newline = m.read(use_length)
            if line != newline:
                print(newline[:10])
                t1 = time()
                print(f"read cost {t1 - t0}")
            else:
                sleep(0.01)
        line = newline