利用posix_fadvise清理系统中的文件缓存

本文介绍如何利用posix_fadvise函数清除特定文件的系统缓存,以准确评估程序性能。提供了示例代码及验证方法。

利用posix_fadvise清理系统中的文件缓存

分类: linux 操作系统 1549人阅读 评论(0) 收藏 举报

当我们需要对某段读写文件并进行处理的程序进行性能测试时,文件会被系统cache住从而影响I/O的效率,必须清理cache中的对应文件的才能正确的进行性能测试。通常清理内存可以采用下面的这条命令,但这条命令只有root才能使用,另外一方面这个会清理所有的cache,也许会影响其他程序的性能。

echo 3>/proc/sys/vm/drop_caches

linux下有一个posix_fadvise函数可以用来对cache中的文件进行清理,有关posix_fadvise的详细说明查看man手册。

  1. int posix_fadvise(int fd, off_t offset, off_t len, int advice);  

”Programs  can  use  posix_fadvise  to  announce an intention to access file data in a specific pattern in the future, thus allowing the kernel to perform appropriate optimisations”

fd是文件的描述符,用于清理对应文件cache的advice值选取POSIX_FADV_DONTNEED,利用此函数编写下面程序进行文件的清理。

  1. int clear_file_cache(const char *filename)   
  2. {   
  3.     struct stat st;   
  4.     if(stat(filename , &st) < 0) {   
  5.         fprintf(stderr , "stat localfile failed, path:%s\n",filename);   
  6.         return -1;   
  7.     }   
  8.   
  9.     int fd = open(filename, O_RDONLY);   
  10.     if( fd < 0 ) {   
  11.         fprintf(stderr , "open localfile failed, path:%s\n",filename);   
  12.         return -1;   
  13.     }   
  14.   
  15.     //clear cache by posix_fadvise  
  16.   
  17.     if( posix_fadvise(fd,0,st.st_size,POSIX_FADV_DONTNEED) != 0) {   
  18.         printf("Cache FADV_DONTNEED failed, %s\n",strerror(errno));   
  19.     }   
  20.     else {   
  21.         printf("Cache FADV_DONTNEED done\n");   
  22.     }   
  23.   
  24.     return 0;   
  25. }  

此外,linux-ftools这个工具也可以帮助清理并查看文件的内存状态,主页上也有详细的使用说明。编译后我们利用fincore这个工具来查看文件在内存中的状态,有关fincore的实现可以在linux下man mincore,mincore是根据缓存buffer指针来其指向的缓冲区判断在cache中的状态,fincore就是在mincore的基础上直接操作文件,就是通过对文件mmap获得指针,再调用mincore来判断。

首先我们通过cp命令拷贝了一个相对有点容量的文件,然后利用fincore查看文件在内存中的cache情况,可以看到大概cache了99.55%。

image

接着执行上面那段代码的运行程序,之后再执行命令查看该文件的缓存状态

image

可以看到,该文件在内存中已经没有被cache了。实际的清理效果也可以通过一些占用I/O的读文件程序来测试,一般程序第二遍运行时候由于文件已经被cache,实际程序运行的速度会比较快,通过上面的posix_fadivse清理之后,又会恢复和第一遍差不多的时间。


checking for OS + Linux 5.15.0-139-generic x86_64 checking for C compiler ... not found ./configure: error: C compiler /usr/bin/aarch64-linux-gcc is not found checking for -Wl,-E switch ... not found checking for gcc builtin atomic operations ... not found checking for C99 variadic macros ... not found checking for gcc variadic macros ... not found checking for gcc builtin 64 bit byteswap ... not found checking for unistd.h ... not found checking for inttypes.h ... not found checking for limits.h ... not found checking for sys/filio.h ... not found checking for sys/param.h ... not found checking for sys/mount.h ... not found checking for sys/statvfs.h ... not found checking for crypt.h ... not found checking for Linux specific features checking for epoll ... not found checking for O_PATH ... not found checking for sendfile() ... not found checking for sendfile64() ... not found checking for sys/prctl.h ... not found checking for prctl(PR_SET_DUMPABLE) ... not found checking for prctl(PR_SET_KEEPCAPS) ... not found checking for capabilities ... not found checking for crypt_r() ... not found checking for sys/vfs.h ... not found checking for nobody group ... not found checking for nogroup group ... found checking for poll() ... not found checking for /dev/poll ... not found checking for kqueue ... not found checking for crypt() ... not found checking for crypt() in libcrypt ... not found checking for F_READAHEAD ... not found checking for posix_fadvise() ... not found checking for O_DIRECT ... not found checking for F_NOCACHE ... not found checking for directio() ... not found checking for statfs() ... not found checking for statvfs() ... not found checking for dlopen() ... not found checking for dlopen() in libdl ... not found checking for sched_yield() ... not found checking for sched_yield() in librt ... not found checking for sched_setaffinity() ... not found checking for SO_SETFIB ... not found checking for SO_REUSEPORT ... not found checking for SO_ACCEPTFILTER ... not found checking for SO_BINDANY ... not found checking for IP_TRANSPARENT ... not found checking for IP_BINDANY ... not found checking for IP_BIND_ADDRESS_NO_PORT ... not found checking for IP_RECVDSTADDR ... not found checking for IP_SENDSRCADDR ... not found checking for IP_PKTINFO ... not found checking for IPV6_RECVPKTINFO ... not found checking for TCP_DEFER_ACCEPT ... not found checking for TCP_KEEPIDLE ... not found checking for TCP_FASTOPEN ... not found checking for TCP_INFO ... not found checking for accept4() ... not found checking for eventfd() ... not found checking for eventfd() (SYS_eventfd) ... not found checking for int size ..../configure: 33: auto/types/sizeof: END: not found checking for long size ..../configure: 33: auto/types/sizeof: END: not found checking for long long size ..../configure: 33: auto/types/sizeof: END: not found checking for void * size ..../configure: 33: auto/types/sizeof: END: not found checking for uint32_t ... uint32_t not found, u_int32_t not found ./configure: error: can not define uint32_t
11-24
评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值