【FastDFS】分布式文件系统FastDFS之FastDHT文件去重

概述

FastDFS除了提供了与Nginx的集成，已提供了去重的文件解决方案。该解决方案FastDFS的作者yuqing也在github上以FastDHT分支贡献出来了。

FastDHT is a high performance distributed hash table (DHT) which based key value pairs. It can store mass key value pairs such as filename mapping, session data and user related data.
FastDHT是基于键值对的高性能分布式哈希表（DHT）。它可以存储大量键值对，例如文件名映射，会话数据和与用户相关的数据。

FastDHT安装

由于github上下载来的安装包文件在安装时或多或少会出现bug，此处建议使用小编给大家处理过的安装包进行安装，以下所有安装包获取方式：

百度网盘：
链接：https://pan.baidu.com/s/1rnwVzlJDA0ykxBzbSOsDWQ
提取码：thmn

蓝奏云：
https://wws.lanzous.com/b01huf3ah
密码:thmn

1. 安装BerkeleyDB 下载db-4.7.25.tar.gz

# 解压
[root@CentOSX ~]# tar -zxf db-4.7.25.tar.gz
[root@CentOSX ~]# cd db-4.7.25
[root@CentOSX db-4.7.25]# cd build_unix/
# 编译安装
[root@CentOSX build_unix]# ./../dist/configure
[root@CentOSX build_unix]# make
[root@CentOSX build_unix]# make install

2. 安装FastDHT

[root@CentOSX ~]# tar zxf FastDHT_v2.01.tar.gz
[root@CentOSX ~]# cd FastDHT
[root@CentOSX FastDHT]# ./make.sh
[root@CentOSX FastDHT]# ./make.sh install

安装结束后会在/etc目录下产生/fdht文件夹

[root@CentOSX FastDHT]# tree /etc/fdht/
/etc/fdht/
├── fdht_client.conf
├── fdhtd.conf
└── fdht_servers.conf

3. 修改fdhtd.conf

[root@CentOSX ~]# mkdir /data/fastdht
[root@CentOSX ~]# vi /etc/fdht/fdhtd.conf
base_path=/data/fastdht

4. 修改fdht_servers.conf

[root@CentOSX ~]# vi /etc/fdht/fdht_servers.conf
group_count = 3
group0 = CentOSA:11411
group1 = CentOSB:11411
group2 = CentOSC:11411

5. 修改fdht_client.conf配置文件

[root@CentOSX ~]# vi /etc/fdht/fdht_client.conf
base_path=/tmp/

6. 启动FDHT服务

[root@CentOSX ~]# fdhtd /etc/fdht/fdhtd.conf start | stop | restart
[root@CentOSX ~]# ps -axu| grep fd
Warning: bad syntax, perhaps a bogus '-'? See /usr/share/doc/procps-3.2.8/FAQ
root      29127  0.0  0.8 195200 17504 ?        Sl   00:38   0:00 fdhtd /etc/fdht/fdhtd.conf start
root      29193  0.0  0.0 103320   884 pts/0    S+   00:39   0:00 grep fd
root     128672  0.2  0.1 275856  2204 ?        Sl   00:14   0:03 /usr/bin/fdfs_trackerd /etc/fdfs/tracker.conf
root     128710  1.8  3.2  85584 67188 ?        Sl   00:14   0:27 /usr/bin/fdfs_storaged /etc/fdfs/storage.conf

操作FastDHT服务

• 设置值

[root@CentOSX ~]# fdht_set /etc/fdht/fdht_client.conf jiangzz:user001 name='jiangzz',age=18;
This is FastDHT client test program v2.01

Copyright (C) 2008, Happy Fish / YuQing

FastDHT may be copied only under the terms of the GNU General
Public License V3, which may be found in the FastDHT source kit.
Please visit the FastDHT Home Page http://www.csource.org/
for more detail.

success set key count: 2, fail count: 0

• 读取值

[root@CentOSX ~]# fdht_get /etc/fdht/fdht_client.conf jiangzz:user001 name,age
This is FastDHT client test program v2.01

Copyright (C) 2008, Happy Fish / YuQing

FastDHT may be copied only under the terms of the GNU General
Public License V3, which may be found in the FastDHT source kit.
Please visit the FastDHT Home Page http://www.csource.org/
for more detail.

name=jiangzz
age=18

success get key count: 2, fail count: 0

• 删除值

[root@CentOSX ~]# fdht_delete /etc/fdht/fdht_client.conf jiangzz:user001 name;
This is FastDHT client test program v2.01

Copyright (C) 2008, Happy Fish / YuQing

FastDHT may be copied only under the terms of the GNU General
Public License V3, which may be found in the FastDHT source kit.
Please visit the FastDHT Home Page http://www.csource.org/
for more detail.

success delete keys: name

success delete key count: 1, fail count: 0

Nginx集成FastDHT

1. 修改/etc/fdfs/storage.conf配置文件

[root@CentOSX ~]# vi /etc/fdfs/storage.conf

check_file_duplicate=1
keep_alive=1

# 下面这句属于C语言的导包语句，必须加进去，否则报错
#include /etc/fdht/fdht_servers.conf

2. 分别启动fdhtd服务、fastfs服务

[root@CentOSX usr]# /usr/local/bin/fdhtd /etc/fdht/fdhtd.conf restart
[root@CentOSX usr]# /etc/init.d/fdfs_trackerd restart
[root@CentOSX usr]# /etc/init.d/fdfs_storaged restart

• 上传文件测试

[root@CentOSA ~]# fdfs_upload_file /etc/fdfs/client.conf install.log
group2/M00/00/00/wKikgl0qC4KAErBTAAAixXWAIyY133.log
[root@CentOSA ~]# fdfs_upload_file /etc/fdfs/client.conf install.log
group2/M00/00/00/wKikgl0qDAqAa0XwAAAixWB5m1c851.log


[root@CentOSB ~]# ls -l /data/fdfs/storage/store01/data/00/00/
total 20
lrwxrwxrwx. 1 root root   72 Jul 14 00:49 wKikgl0qC4KAErBTAAAixXWAIyY133.log -> /data/fdfs/storage/store01/data/00/00/wKikgl0qC4KARrYBAAAixZ60QfI755.log
-rw-r--r--. 1 root root 8901 Jul 14 00:49 wKikgl0qC4KARrYBAAAixZ60QfI755.log
lrwxrwxrwx. 1 root root   72 Jul 14 00:51 wKikgl0qDAqAa0XwAAAixWB5m1c851.log -> /data/fdfs/storage/store01/data/00/00/wKikgl0qC4KARrYBAAAixZ60QfI755.log

可以看出系统产生了wKikgl0qC4KAErBTAAAixXWAIyY133.log的两个链接