nfs 共享session方式 session_start 慢 问题解决

文章详细介绍了PHP应用在NFS共享环境下遇到会话启动缓慢的问题,通过排查发现是NFS服务未开启导致。文章进一步解释了NFS配置、优化及基本命令,最终成功解决了PHP脚本在不同服务器间读取会话数据的问题。

       近几天php搭建了个新的运行平台,但出了个问题,就是显示登录状态的页面打开卡的时间有点长。这个很明显是读取会话状态的操作时产生,但就是搞不明白是怎么回事。

       因为php方面有所改动,在访问会话时产生,很自然想到会不会是PHP的问题,或者NFS的负载高了?

       首先是重装一次PHP,换用原来一模一样的环境,问题依旧,再使用PHP性能检测工具,问题确实是出现在session_start()的时候。 是不是性能问题呢? 于是乎又是一番对NFS的检查,优化。。。均告无效。

  上网搜索,此类问题均被作为PHP BUG来提问 =。= 

      很偶然的情况下,我想,会不会是nfslock服务停掉了的原因呢?  启动服务后一切正常。哈哈。

      这才想起,做初始化脚本时,认为一般系统是不运行NFS的,当有需要时再开,因此关掉了很多认为少用到的服务。

 

 

关键字:Very Slow session_start on nfs-mount   Session_start Is Slow

 

 

另外有一种情况,运行PHP时,使用NFS方式测试性能下降明显,原因是open_basedir 的设置问题。单机环境去掉即可。或:     ../:./:/tmp:../../

 

 

 

至于设置多个domain,写多几行就行。

session.cookie_domain =abc.com
session.cookie_domain =abcd.com

 

 

 

 

 

 

 

 

 

 

 

 

服务器是使用session NFS共享方式实现了的!session NFS共享方式配置如下:

http://www.linuxdiyf.com/viewarticle.php?id=161086

 

首先,修改 php.ini的 session.save_path 选项修改如下:

session.save_path = "2;/tmp/php_sess" (去掉前面分号)

意为把session存放在 "/tmp/php_sess" 目录下,并且分成 2 级子目录,每级子目录又分别有 16 个子目录。


2.假设php的主目录为 /usr/local/server/php/,则新建一个文件 /usr/local/server/php/include/php/ext/session/mod_files.sh,其内容如下:
#! /bin/sh
# NAME
# mod_files.sh - Update of the php-source/ext/session/mod_files.sh
#
# SYNOPSIS
# mod_files.sh basedir depth [numberofsubdirs]
#
# DESCRIPTION
# this script creates the directories tree used by php to store the session files
# (see php.ini - 'session.save_path' option)
#
# Example: if you want php to store the session files in a directory tree
# of 3 levels of depth containing 32 directories in each directory,
# first, put the setting bellow in the php.ini file:
#
# session.save_path = "3;/tmp/session"
#
# Now create the basedir directory: 'mkdir /tmp/session'
#
# Then, call this scrip with the following arguments:
#
# ./mod_files.sh ./mod_files.sh /tmp/session 3 32

if test "$2" = ""; then
echo "usage: $0 basedir depth [numberofsubdirs]"
echo "numberofsubdirs: if unset, defaults to 16. if 32, 32 subdirs, if 64, 64 subdirs."
exit 1
fi

if test "$2" = "0"; then
exit 0
fi

hash_chars="0 1 2 3 4 5 6 7 8 9 a b c d e f"
if [ ! -z $3 ] ; then
if test "$3" -a "$3" -eq "32"; then
hash_chars="$hash_chars g h i j k l m n o p q r s t u v"
if test "$3" -eq "64"; then
hash_chars="$hash_chars w x y z A B C D E F G H I J K L M N O P Q R S T U V W X Y Z - ,"
fi
fi
fi

for i in $hash_chars; do
newpath="$1/$i"
mkdir $newpath || exit 1
sh $0 $newpath `expr $2 - 1` $3
done

 

3.设置为可执行之后,运行以下命令来创建哈希目录:

#cd /usr/local/server/php/include/php/ext/session/
#./mod_files.sh /tmp/php_sess 2 16


4现在,就开始设置 NFS 共享了。假定有3台主机,ip分别为192.168.0.1(主机名svr1)、192.168.0.2(主机名svr2)、192.168.0.3(主机名svr3),现在让192.168.0.1来提供NFS共享服务,配置 /etc/exports,加入如下内容:
/tmp/php_sess/ svr*(rw,no_root_squash) #(意思是:允许svr*所以的机器来访问)

 

5.然后重启 nfs 服务,即可对另外两台主机提供NFS共享了。
在 svr2、svr3 上执行以下命令来挂在NFS:
#mkdir /tmp/php_sess
#mount svr1:/tmp/php_sess /tmp/php_sess

 


按上面的配置完成 session NFS 共享,但是问题来了,
我在A服务器上使用PHP脚本如下:
sei.php(文件名)

session_start();
$_SESSION['ID'] ='ABC';
echo $_SESSION['ID'];
?>

然后在B服务下使用PHP脚本如下:
sei.php(文件名)

session_start();
$_SESSION['ID'];
echo $_SESSION['ID'];
?>

测试:A服务能正常显示ABC 但是B服务不能得到A服务访问的session,不显示ABC,显示全是空白。按道理说,按上面session nfs共享配置,我A服务产生了 ABC,我B服务器能自动读取ABC,应该也显示ABC才对啊!!(session NFS共享测试,是不是我PHP脚本有问题??)
大家帮帮忙。。。

 

 

 

 

 

 

=================================================================
nfs配置+优化+相关基本命令 

NFS配置步骤和优化(转载)
nfs配置                                          
1、NFS包

NFS需要5个RPM,分别是:
setup-*:    共享NFS目录在/etc/exports中定义
initscripts-*:   包括引导过程中装载网络目录的基本脚本
nfs-utils-*:    包括基本的NFS命令与监控程序
portmap-*:   支持安全NFS RPC服务的连接
quota-*:    网络上共享的目录配额,包括rpc.rquotad (这个包不是必须的)

2、基本监控程序

要顺利运行NFS,至少需要五个Linux服务,它们各有不同的功能,有的负责装载服务,有的保证远程命令指向正确的位置。这些服务通过/etc/rc.d/init.d目录中的nfs,nfslock和portmap脚本启动。下面简单介绍每个监控程序:

(1) 基本NFS
rpc.nfsd是NFS服务器监控程序,它通过/etc/rc.d/init.d目录中的nfs脚本启动。NFS监控程序还启动rpc.mountd装载监控程序,并导出共享目录。

(2) RPC装载
可以用mount命令连接本地目录或网络目录,但还需要一个装载NFS目录的特殊监控程序rpc.mountd

(3) 端口映射器
portmap监控程序只是定向RPC通信数据流,但它对于NFS服务很重要。如果不运行portmap,则NFS客户机无法找到从NFS服务器共享的目录。

(4) 重新启动与statd
当NFS服务需要中断或者重新启动时,rpc.statd监控程序和rpc.lockd在服务器重新启动之后使客户机恢复NFS连接。

(5) 锁定
通过共享NFS目录打开文件时,锁定可以使用户不能覆盖同一个文件。锁定通过nfslock脚本并使用rpc.lockd监控程序启动运行。

3、配置NFS

共享的NFS目录在/etc/exports中列出,这个文件控制对目录的共享。书写规则是:(每个共享规则一行)

共享目录 主机(参数)

例如:/mnt/cdrom *.abc.com(ro,sync) master.abc.com(rw,sync)

上面的规则代表将/mnt/cdrom目录以只读同步方式共享给*.abc.com域,并且以读写同步方式共享给master.abc.com主机。任何共享目录都要指定sync或async,也就是指定文件写入磁盘之前共享NFS目录是否响应命令。

下面是一些NFS共享的常用参数:
ro:只读访问
rw:读写访问
sync:所有数据在请求时写入共享
async:NFS在写入数据前可以相应请求
secure:NFS通过1024以下的安全TCP/IP端口发送
insecure:NFS通过1024以上的端口发送
wdelay:如果多个用户要写入NFS目录,则归组写入(默认)
no_wdelay:如果多个用户要写入NFS目录,则立即写入,当使用async时,无需此设置。
hide:在NFS共享目录中不共享其子目录
no_hide:共享NFS目录的子目录
subtree_check:如果共享/usr/bin之类的子目录时,强制NFS检查父目录的权限(默认)
no_subtree_check:和上面相对,不检查父目录权限
all_squash:共享文件的UID和GID映射匿名用户anonymous,适合公用目录。
no_all_squash:保留共享文件的UID和GID(默认)
root_squash:root用户的所有请求映射成如anonymous用户一样的权限(默认)
no_root_squash:root用户具有根目录的完全管理访问权限
anonuid=xxx:指定NFS服务器/etc/passwd文件中匿名用户的UID
anongid=xxx:指定NFS服务器/etc/passwd文件中匿名用户的GID

4、启动NFS

# service portmap start  
# service nfs start

或者
#/etc/init.d/nfs start
#/etc/init.d/portmap start
检查NFS的运行级别:
# chkconfig --list portmap
# chkconfig --list nfs

根据需要设置在相应的运行级别自动启动NFS:
# chkconfig --level 235 portmap on
# chkconfig --level 235 nfs on

另外,还需要查看系统的iptables、/etc/hosts.allow、/etc/hosts.deny是否设置了正确的NFS访问规则。

参考:
nfs-howto

nfs优化                                          

1.设置块大小
mount命令的risize和wsize指定了server端和client端的传输的块大小。

mount -t nfs -o rsize=8192,wsizevb=8192,timeo=14,intr client:/partition /partition

如果未指定,系统根据nfs version来设置缺省的risize和wsize大小。大多数情况是4K对于nfs v2,最大是8K,对于v3,通过server端kernel设置risize和wsize的限制

vi /usr/src/linux2.4.22/include/linux/nfsd/const.h
修改常量: NFSSVC_MAXBLKSIZE

所有的2.4的的client都支持最大32K的传输块。系统缺省的块可能会太大或者太小,这主要取决于你的kernel和你的网卡,太大或者太小都有可能导致nfs速度很慢。
具体的可以使用Bonnie,Bonnie++,iozone等benchmark来测试不同risize和wsize下nfs的速度。当然,也可以使用dd来测试。

#time dd if=/dev/zero of=/testfs/testfile bs=8k count=1024  测试nfs写
#time dd if=/testfs/testfile of=/dev/null bs=8k        测试nfs读

测试时文件的大小至少是系统RAM的两倍,每次测试都使用umount 和mount对/testfs进行挂载,通过比较不同的块大小,得到优化的块大小。


2.网络传输包的大小
网络在包传输过程,对包要进行分组,过大或者过小都不能很好的利用网络的带宽,所以对网络要进行测试和调优。可以使用ping -s 2048 -f hostname进行ping,尝试不同的package size,这样可以看到包的丢失情况。同时,可以使用nfsstat -o net 测试nfs使用udp传输时丢包的多少。因为统计不能清零,所以要先运行此命令记住该值,然后可以再次运行统计。如果,经过上面的统计丢包很多。那么可以看看网络传输包的大小。使用下面的命令:

#tracepath node1/端口号
#ifconfig eth0

比较网卡的mtu和刚刚的pmtu,使用#ifconfig eth0 mtu 16436设置网卡的mtu和测试的一致。 当然如果risize和wsize比mtu的值大,那么的话,server端的包传到client端就要进行重组,这是要消耗client端的cpu资源。此外,包重组可能导致网络的不可信和丢包,任何的丢包都会是的rpc请求重新传输,rpc请求的重传有会导致超时,严重降低nfs的性能。
可以通过查看

/proc/sys/net/ipv4/ipfrag_high_thresh
/proc/sys/net/ipv4/ipfrag_low_thresh

了解系统可以处理的包的数目,如果网络包到达了ipfrag_high_thresh,那么系统就会开始丢包,直到包的数目到达ipfrag_low_thresh。

3.nfs挂载的优化
timeo:  如果超时,客户端等待的时间,以十分之一秒计算
retrans: 超时尝试的次数。
bg:    后台挂载,很有用
hard:   如果server端没有响应,那么客户端一直尝试挂载
wsize:  写块大小
rsize:  读块大小
intr:   可以中断不成功的挂载
noatime: 不更新文件的inode访问时间,可以提高速度
async:  异步读写

4.nfsd的个数
缺省的系统在启动时,有8个nfsd进程
#ps -efl|grep nfsd
通过查看/proc/net/rpc/nfsd文件的th行,第一个是nfsd的个数,后十个是线程是用的时间数,第二个到第四个值如果很大,那么就需要增加nfsd的个数。
具体如下:

#vi /etc/init.d/nfs

找到RPCNFSDCOUNT,修改该值,一般和client端数目一致。

#service nfs restart
#mount -a

5.nfsd的队列长度
对于8个nfsd进程,系统的nfsd队列长度是64k大小,如果是多于8个,就要相应的增加相应的队列大小,具体的在

/proc/sys/net/core/rwmem_default
/proc/sys/net/core/wwmem_default
/proc/sys/net/core/rmmem_max
/proc/sys/net/core/wmmem_max

队列的长度最好是每一个nfsd有8k的大小。这样,server端就可以对client的请求作排队处理。如果要永久更改此值

#vi /etc/sysctl.conf
net.core.rmmem_default=数目
net.core.wmmem_default=数目
net.core.rmmem_max=数目
net.core.wmmem_max=数目
#service nfs restart


++
查看被导出资源
showmount -e nfsserver_name(or nfsserver IP address)
重新加载配置:
exportfs -rv
停止现在发布的目录
exportfs -a

 

 

===============================================================================
Centos64bit_5.3 默认值:
[root@qy init.d]# cat /proc/sys/net/core/rmem_default
126976         124K
[root@qy init.d]# cat /proc/sys/net/core/rmem_max
131071         127.xxxK

 

#include <dirent.h> #include <sys/statfs.h> #include "common.h" #include "storage.h" #include "iot_info.h" #include "device.h" #ifdef HEALTH_INFO_SUPPORT #include "health_info.h" #endif void storage_session_free(STORAGE_SESSION *storage_session); void storage_server_timeout_handle_immediately(STORAGE_SERVER *storage_server); /*---------------------------------------------------------*/ void storage_delete_posted_session(STORAGE_SERVER *storage_server, char *dev_id, int storage_type) { STORAGE_SESSION *session, *nsession; if (!storage_server || !dev_id) { DBG_ERR("arg NULL\n"); return; } if (STORAGE_STREAMING == storage_type) { list_for_each_entry_safe(session, nsession, &storage_server->stream_session, list) { if (!strncmp(session->dev_id, dev_id, LEN_MAX_ID) && !session->uploading_flag && (!session->stream_send_to_cloud_flag || session->delete_flag)) { storage_session_free(session); } } } else if (STORAGE_SNAPSHOT == storage_type) { list_for_each_entry_safe(session, nsession, &storage_server->snapshot_session, list) { DBG_DBG("%s %s %d %d %d\n", dev_id,session->dev_id, session->uploading_flag, session->snapshot_send_to_cloud_flag, session->delete_flag); if (!strncmp(session->dev_id, dev_id, LEN_MAX_ID) && !session->uploading_flag && (!session->snapshot_send_to_cloud_flag || session->delete_flag)) { storage_session_free(session); } } } } static void storage_session_telemetry_failure_count(STORAGE_SESSION *storage_session) { DEV_INFO *dev_info; MEDIACENTER_CTX *mediacenter_ctx; if (!storage_session) { DBG_ERR("arg NULL\n"); return; } mediacenter_ctx = storage_session->storage_server->worker_ctx->top_ctx; dev_info = get_dev_info(mediacenter_ctx, storage_session->dev_id, storage_session->dev_ip); if (dev_info && storage_session->post_success && (dev_info->telemetry.relay_clips_failure_count > 0)) { dev_info->telemetry.relay_clips_failure_count--; #ifdef HEALTH_INFO_SUPPORT dev_update_health_info(dev_info->dev_id); #endif } return; } void storage_delete_sessions_of_device(MEDIACENTER_CTX *mediacenter_ctx, char *dev_id) { WORKER_CTX *worker_ctx = NULL; STORAGE_SERVER *storage_server = NULL; STORAGE_SESSION *session = NULL, *nsession = NULL; if (!mediacenter_ctx || !dev_id) { DBG_ERR("arg NULL\n"); return; } if (list_empty(&mediacenter_ctx->worker_list)) { DBG_ERR("worker list empty\n"); return; } worker_ctx = (WORKER_CTX *)list_first_entry(&mediacenter_ctx->worker_list, STORAGE_SESSION, list); if (worker_ctx) { storage_server = worker_ctx->storage_server; if (storage_server) { list_for_each_entry_safe(session, nsession, &storage_server->stream_session, list) { if (!strncmp(session->dev_id, dev_id, LEN_MAX_ID)) { storage_session_free(session); } } list_for_each_entry_safe(session, nsession, &storage_server->snapshot_session, list) { if (!strncmp(session->dev_id, dev_id, LEN_MAX_ID)) { storage_session_free(session); } } } } } int storage_session_get_push_status(STORAGE_SESSION *session) { DEV_INFO *dev_info; MEDIACENTER_CTX *mediacenter_ctx; if (!session) { DBG_ERR("arg NULL\n"); return -1; } mediacenter_ctx = session->storage_server->worker_ctx->top_ctx; dev_info = get_dev_info(mediacenter_ctx, session->dev_id, session->dev_ip); if (dev_info) { if (STORAGE_STREAMING == session->storage_type) { return dev_info->alarm_stream_push_status; } else { return dev_info->alarm_snapshot_push_status; } } return -1; } void storage_session_set_push_status(STORAGE_SESSION *session, int status) { DEV_INFO *dev_info; MEDIACENTER_CTX *mediacenter_ctx; if (!session) { DBG_ERR("arg NULL\n"); return; } mediacenter_ctx = session->storage_server->worker_ctx->top_ctx; dev_info = get_dev_info(mediacenter_ctx, session->dev_id, session->dev_ip); if (dev_info) { DBG_INFO("It is alarm re-upload below\n"); if (STORAGE_STREAMING == session->storage_type) { dev_set_alarm_stream_status(dev_info, status); } else { dev_set_alarm_snapshot_status(dev_info, status); } } return; } void storage_session_reupload_timeout(struct uloop_timeout* tmo) { //MYDEBUG("STORAGE SESSION REUPLOAD CHECK TIMEOUT\n"); unsigned long long sent; STORAGE_SESSION *storage_session = container_of(tmo, STORAGE_SESSION, reupload_tmo); if (!storage_session) { DBG_ERR("arg NULL\n"); return; } uloop_timeout_set(&storage_session->reupload_tmo, storage_session->reupload_idle_time); if (storage_session->sock) { sent = storage_session->reupload_total_load - storage_session->sock->write_buf_length; if (sent <= storage_session->reupload_last_sent) { DBG_ERR("STORAGE SESSION REUPLOAD TIMEOUT\n"); tpsocket_free(storage_session->sock); } else { storage_session->reupload_last_sent = sent; storage_session->reupload_last_load = storage_session->reupload_total_load; } } } bool storage_session_write_list_force(STORAGE_SESSION *storage_session, struct list_head *head, bool force) { struct tpsocket_buf *buf; unsigned long long to_send = 0; if (!storage_session || !head) { DBG_ERR("arg NULL\n"); return false; } /* data length */ list_for_each_entry(buf, head, list) { to_send += tpbuf_data_len(buf); } /* move bufs to sock */ if (false == tpsocket_write_list_force(storage_session->sock, head, force)) { DBG_ERR("tpsocket_write_list error\n"); return false; } /* calculate total load */ storage_session->reupload_total_load += to_send; return true; } void storage_session_stream_add_tail(STORAGE_SESSION *storage_session) { struct list_head head; if (!storage_session) { DBG_ERR("arg NULL\n"); return; } if (storage_session->chunked) { INIT_LIST_HEAD(&head); tpsocket_chunk_encode(&head);//generate chunk end '0\r\n' tpsocket_write_list_force(storage_session->sock, &head, true); } tpsocket_write_list_force(storage_session->sock, &head, true); } void storage_session_snapshot_add_tail(STORAGE_SESSION *storage_session) { struct tpsocket_buf *mybuf = NULL; struct list_head head; if (!storage_session) { DBG_ERR("arg NULL\n"); return; } INIT_LIST_HEAD(&head); mybuf = tpbuf_snprintf(LEN_MAX_128, "\r\n"); if (mybuf) { list_add_tail(&mybuf->list, &head); } if (storage_session->region_buf_len) { mybuf = tpbuf_snprintf(LEN_MAX_128*4, "%s", storage_session->region_buf); if (mybuf) { list_add_tail(&mybuf->list, &head); } } mybuf = tpbuf_snprintf(LEN_MAX_128, "--%s--", SNAPSHOT_FILE_BOUNDARY);//generate snapshot multi-part end if (mybuf) { list_add_tail(&mybuf->list, &head); } if (storage_session->chunked) { tpsocket_chunk_encode(&head); } tpsocket_write_list_force(storage_session->sock, &head, true); } int storage_session_set_region(STORAGE_SESSION *storage_session, TRIGGER_REGION *region) { if (!storage_session || !region) { DBG_ERR("arg NULL\n"); return -1; } if (0 == region->x1 && 0 == region->y1 && 0 == region->x2 && 0 == region->y2) { } else { storage_session->region_buf_len = snprintf(storage_session->region_buf, LEN_MAX_128*4, "--%s\r\n" "Content-Disposition: form-data;name=\"region\"\r\n\r\n" "{\"top\": %d, \"bottom\": %d, \"left\": %d, \"right\": %d}\r\n", SNAPSHOT_FILE_BOUNDARY, region->y1, region->y2, region->x1, region->x2); MYDEBUG("%s", storage_session->region_buf); } return 0; } int storage_session_set_event_timestamp(STORAGE_SESSION *storage_session, time_t timestamp) { if (!storage_session) { DBG_ERR("arg NULL\n"); return -1; } //storage_session->event_timestamp = timestamp - common_get_time_diff(); storage_session->event_timestamp = timestamp; return 0; } int storage_session_set_event_type(STORAGE_SESSION *storage_session, enum EVENT_TYPE event_type) { if (!storage_session) { DBG_ERR("arg NULL\n"); return -1; } storage_session->event_type = event_type; return 0; } void storage_update_list(STORAGE_SERVER *storage_server, int flag) { STORAGE_SESSION *storage_session; char list_path[LEN_MAX_PATH] = {0}; FILE *fd; if (!storage_server) { DBG_ERR("arg NULL\n"); return; } if (STORAGE_STREAMING == flag) { memset(list_path, 0, LEN_MAX_PATH); snprintf(list_path, LEN_MAX_PATH, "%s/%s", STORAGE_ROOT_PATH, STORAGE_STREAMING_LIST); fd = fopen(list_path, "w"); if (fd) { list_for_each_entry(storage_session, &storage_server->stream_session, list) { fwrite(storage_session, sizeof(*storage_session), 1, fd); } fclose(fd); } } if (STORAGE_SNAPSHOT == flag) { memset(list_path, 0, LEN_MAX_PATH); snprintf(list_path, LEN_MAX_PATH, "%s/%s", STORAGE_ROOT_PATH, STORAGE_SNAPSHOT_LIST); fd = fopen(list_path, "w"); if (fd) { list_for_each_entry(storage_session, &storage_server->snapshot_session, list) { fwrite(storage_session, sizeof(*storage_session), 1, fd); } fclose(fd); } } } void storage_session_add_session(STORAGE_SESSION *storage_session) { STORAGE_SERVER *storage_server; STORAGE_SESSION *session, *nsession; if (!storage_session) { MYDEBUG("arg NULL\n"); return; } storage_server = storage_session->storage_server; if (STORAGE_STREAMING == storage_session->storage_type) { MYDEBUG("storage stream session = %p\n", storage_session); if (false == storage_session->storage_success) { storage_session_free(storage_session); return; } list_for_each_entry_safe(session, nsession, &storage_server->stream_session, list) { if (!strncmp(session->dev_id, storage_session->dev_id, LEN_MAX_ID) && !session->uploading_flag) { if (!strncmp(session->stream_path, storage_session->stream_path, LEN_MAX_PATH)) memset(session->stream_path, 0, LEN_MAX_PATH);//if old session's path is the same as new, do not delete file storage_session_free(session); } } if (get_dev_info(storage_server->worker_ctx->top_ctx, storage_session->dev_id, NULL)) { list_add_tail(&storage_session->list, &storage_server->stream_session); storage_update_list(storage_server, STORAGE_STREAMING); } else { //if device not exist, free session storage_session_free(storage_session); } } else { MYDEBUG("storage snapshot session = %p\n", storage_session); if (false == storage_session->storage_success) { storage_session_free(storage_session); return; } list_for_each_entry_safe(session, nsession, &storage_server->snapshot_session, list) { if (!strncmp(session->dev_id, storage_session->dev_id, LEN_MAX_ID) && !session->uploading_flag) { if (!strncmp(session->snapshot_path, storage_session->snapshot_path, LEN_MAX_PATH)) memset(session->snapshot_path, 0, LEN_MAX_PATH); storage_session_free(session); } } if (get_dev_info(storage_server->worker_ctx->top_ctx, storage_session->dev_id, NULL)) { list_add_tail(&storage_session->list, &storage_server->snapshot_session); storage_update_list(storage_server, STORAGE_SNAPSHOT); } else { //if device not exist, free session storage_session_free(storage_session); } } return; } bool storage_session_add_fixed_header(STORAGE_SESSION *storage_session) { struct tpsocket_buf *buf; if (!storage_session) { DBG_ERR("arg NULL\n"); return false; } /* add */ buf = tpbuf_snprintf(512, "%x\r\n" "--data-boundary--\r\n" "Content-Length: 218\r\n" "Content-Type: text/http-header\r\n" "\r\n" "HTTP/1.0 200 OK\r\n" "Content-Type: multipart/x-mixed-replace;boundary=data-boundary--\r\n" "X-Encrypt-Type: PLAIN\r\n" "Pragma: no-cache\r\n" "Cache-Control: no-cache\r\n" "Content-Length: -1\r\n" "Transfer-Encoding: chunked\r\n" "Connection: close\r\n\r\n" "\r\n", 292); if (buf) { list_add_tail(&buf->list, &storage_session->headers); } return true; } bool storage_stream_consumer_post(STORAGE_SESSION *storage_session) { struct list_head head; struct tpsocket_buf *buf; DEV_INFO *dev_info; char *request = NULL; if (!storage_session) { DBG_ERR("arg NULL\n"); return false; } dev_info = get_dev_info(storage_session->storage_server->worker_ctx->top_ctx, storage_session->dev_id, NULL); if (!dev_info) { DBG_ERR("get dev info failed\n"); return false; } INIT_LIST_HEAD(&head); char url[256] = {0}; request = strstr(dev_info->relay_storage_url, "/v1"); snprintf(url, 256, "%sactivity?deviceid=%s" "&eventid=%s" "&type=mixed" "&resolution=HD", request, storage_session->dev_id, storage_session->event_id); //MYDEBUG("storage token:%s\n", storage_session->xtoken); //MYDEBUG("storage cookie:%s\n", storage_session->cookie); buf = tpbuf_snprintf(1024, "POST %s HTTP/1.1\r\n" "User-Agent: %s/%s %s\r\n" // MODEL/FW_VERSION PROGRAM_NAME/PROGRAM_VERSION "Host: %s\r\n" "Transfer-Encoding: chunked\r\n" //http chunked transfer encoding "Content-Type: multipart/x-mixed-replace;boundary=%s\r\n" "X-token: %s\r\n" // X-token "X-Audio-Codec: %s\r\n" "Cookie: %s\r\n" "\r\n", url, dev_info->dev_model?dev_info->dev_model:"*", dev_info->sw_ver?dev_info->sw_ver:"*", THIS_PROCESS, storage_session->host, "data-boundary--", storage_session->xtoken, dev_info->audio_fmt, storage_session->cookie); list_add_tail(&buf->list, &head); /* header */ list_splice_tail_init(&storage_session->headers, &head); /* ready to write */ debug_show_tpbuf_list(&head, "STORAGE STREAM POST:", 3); if (false == storage_session_write_list_force(storage_session, &head, false)) { DBG_ERR("storage_session_write_list_force error\n"); tpsocket_free_buf(&head, NULL, 0); return false; } return true; } bool storage_snapshot_consumer_post(STORAGE_SESSION *storage_session) { struct list_head head; struct tpsocket_buf *buf; DEV_INFO *dev_info; char *request = NULL; if (!storage_session) { DBG_ERR("arg NULL\n"); return false; } INIT_LIST_HEAD(&head); dev_info = get_dev_info(storage_session->storage_server->worker_ctx->top_ctx, storage_session->dev_id, NULL); if (!dev_info) { DBG_ERR("get dev info failed\n"); return false; } /* add POST header */ char url[256] = {0}; int sub_header_len; char sub_header[512] = {0}; request = strstr(dev_info->storage_url, "/v1"); snprintf(url, 256, "%sdevice/%s/activity/%s/snapshot", request, storage_session->dev_id, storage_session->event_id); sub_header_len = snprintf(sub_header, 512, "--%s\r\n" "Content-Disposition: form-data;name=\"snapshot\";filename=\"example.jpg\"\r\n" "Content-Type: image/jpeg\r\n\r\n", SNAPSHOT_FILE_BOUNDARY); int content_len = sub_header_len+storage_session->content_len+strlen("\r\n")+storage_session->region_buf_len+ strlen("--")+strlen(SNAPSHOT_FILE_BOUNDARY)+strlen("--"); buf = tpbuf_snprintf(1024, "POST %s HTTP/1.1\r\n" "User-Agent: %s/%s %s\r\n" // MODEL/FW_VERSION PROGRAM_NAME/PROGRAM_VERSION "X-token: %s\r\n" // X-token "Host: %s\r\n" "Content-Length: %d\r\n" "Content-Type: multipart/form-data;boundary=%s\r\n" "\r\n" "%s",//sub_header url, dev_info->dev_model?dev_info->dev_model:"*", dev_info->sw_ver?dev_info->sw_ver:"*", THIS_PROCESS, storage_session->xtoken, storage_session->host, content_len, SNAPSHOT_FILE_BOUNDARY, sub_header); list_add_tail(&buf->list, &head); debug_show_tpbuf_list(&head, "STORAGE SNAPSHOT POST:", 3); if (false == storage_session_write_list_force(storage_session, &head, false)) { DBG_ERR("storage_session_write_list_force error\n"); tpsocket_free_buf(&head, NULL, 0); return false; } return true; } bool storage_file_produce(STORAGE_SESSION *storage_session, struct list_head *buf) { if (!storage_session) { DBG_ERR("arg NULL\n"); return false; } if (!storage_session->sock) { DBG_ERR("%p arg NULL, file = %p\n", storage_session, storage_session->file); return false; } if (tpsocket_writable(storage_session->sock)) { if (!list_empty(buf) && storage_session->chunked && !tpsocket_chunk_encode(buf)) { tpsocket_free_buf(buf, "PRODUCE", 0); return false; } storage_session_write_list_force(storage_session, buf, true); } return true; } bool storage_read_file(struct tpsocket_handler*handler, struct list_head*buf, int event) { STORAGE_SESSION *storage_session = handler->priv; //MYDEBUG("storage read file event = %s, sock = %p, consumer = %p\n", tpsocket_event_name(event), sock, storage_session); switch(event) { case TPSOCKET_EVENT_CONNECTED: break; case TPSOCKET_EVENT_REQ_HEAD: break; case TPSOCKET_EVENT_RSP_HEAD: break; case TPSOCKET_EVENT_UPGRADE: break; case TPSOCKET_EVENT_SUB_HEAD: case TPSOCKET_EVENT_STREAM: case TPSOCKET_EVENT_MESSAGE: if (false == storage_file_produce(storage_session, buf)) { return false; } goto out; //break; case TPSOCKET_EVENT_RESET: break; case TPSOCKET_EVENT_CLOSED: if (storage_session) { storage_session->file = NULL; if (storage_session->sock) { if (!list_empty(buf)) { if (storage_session->chunked) { tpsocket_chunk_encode(buf); tpsocket_write_list_force(storage_session->sock, buf, true); } } if (STORAGE_STREAMING == storage_session->storage_type) { storage_session_stream_add_tail(storage_session); } else { storage_session_snapshot_add_tail(storage_session); } } } break; case TPSOCKET_EVENT_ERROR: default: break; } tpsocket_free_buf(buf, tpsocket_event_name(event), 0); out: return true; } bool storage_start_read(STORAGE_SESSION *storage_session) { if (!storage_session) { DBG_ERR("arg NULL\n"); return false; } struct tpsocket_handler client = { .cb = storage_read_file, .flowcontrol = 10*1024, .priv = storage_session, .read_buf_max = STORAGE_MEMORY_CACHE_LIMIT, }; if (STORAGE_STREAMING == storage_session->storage_type) { storage_session->file = tpsocket_new(NULL, "stream", NULL, storage_session->stream_path, TPSOCKET_TYPE_FILE, &client); if (!storage_session->file) { DBG_ERR("open %s failed\n", storage_session->stream_path); return false; } } else { storage_session->file = tpsocket_new(NULL, "read", NULL, storage_session->snapshot_path, TPSOCKET_TYPE_FILE, &client); if (!storage_session->file) { DBG_ERR("open %s failed\n", storage_session->snapshot_path); return false; } } DBG_DBG("session = %p, sock = %p, open file = %p\n", storage_session, storage_session->sock, storage_session->file); return true; } bool tpsocket_event_storage_stream_to_cloud(struct tpsocket_handler *handler, struct list_head *buf, int event) { //debug_show_tpbuf_list(buf, "TEST", 3); struct tpsocket_fd *sock = container_of(handler, struct tpsocket_fd, handler); STORAGE_SESSION *storage_session = handler->priv; int err_code = 0; if (event != TPSOCKET_EVENT_WRITABLE) { MYDEBUG("storage stream to cloud event = %s, sock = %p, session = %p\n", tpsocket_event_name(event), sock, storage_session); if (storage_session) { DBG_DBG("Device Id = %s, session sock = %p\n", storage_session->dev_id, storage_session->sock); } } switch(event) { case TPSOCKET_EVENT_LISTEN: break; case TPSOCKET_EVENT_SHUTDOWN: break; case TPSOCKET_EVENT_ACCEPT: break; case TPSOCKET_EVENT_CONNECTED: MYDEBUG("STORAGESTREAM2CLOUD: %s:%s Connected\n", sock->addr, sock->port); if (false == storage_stream_consumer_post(storage_session)) return false; if (false == storage_start_read(storage_session)) { DBG_ERR("read stream file failed\n"); return false; } break; case TPSOCKET_EVENT_REQ_HEAD: break; case TPSOCKET_EVENT_RSP_HEAD: break; case TPSOCKET_EVENT_SUB_HEAD: break; case TPSOCKET_EVENT_UPGRADE: break; case TPSOCKET_EVENT_WRITABLE: if (storage_session->file && !list_empty(&(storage_session->file->parser.buf))) { if (storage_session->chunked && !tpsocket_chunk_encode(&(storage_session->file->parser.buf))) { tpsocket_free_buf(&(storage_session->file->parser.buf), tpsocket_event_name(event), 0); break; } storage_session_write_list_force(storage_session, &(storage_session->file->parser.buf), true); storage_session->file->read_buf_length = 0; } break; case TPSOCKET_EVENT_STREAM: return true; case TPSOCKET_EVENT_MESSAGE: err_code = common_check_error_code(buf); if (!err_code) { DBG_DBG("storage stream push success\n"); storage_session->post_success = 1; #ifndef TAPO_CAMERA } else if (err_code == -98400) { //If the recording/relay server finds out that the device is not eligible for cloud storage, // it returns an error code of -98400 upon which the firmware updates the cloud storage feature of that device. DEV_INFO *dev_info = get_dev_info(storage_session->storage_server->worker_ctx->top_ctx, storage_session->dev_id, NULL); if (dev_info) { dev_info->cloud_storage = false; } // mark as success here to avoid retrying. storage_session->post_success = 1; #endif } tpsocket_free_buf(buf, tpsocket_event_name(event), 0); common_tpsocket_close_immediately(sock); break; case TPSOCKET_EVENT_RESET: break; case TPSOCKET_EVENT_KEEPALIVE: common_tpsocket_close_immediately(sock); break; case TPSOCKET_EVENT_CLOSED: if (storage_session) { tpsocket_unbind(sock, &storage_session->sock); uloop_timeout_cancel(&storage_session->reupload_tmo); tpsocket_free2(&storage_session->file, storage_session); if (storage_session->post_success) { storage_session->stream_send_to_cloud_flag = 0; } storage_session->uploading_flag = 0; storage_session_set_push_status(storage_session, 0); /* handle timeout immediately */ storage_server_timeout_handle_immediately(storage_session->storage_server); } break; case TPSOCKET_EVENT_ERROR: default: break; } tpsocket_free_buf(buf, tpsocket_event_name(event), 0); return true; } bool tpsocket_event_storage_snapshot_to_cloud(struct tpsocket_handler *handler, struct list_head *buf, int event) { //debug_show_tpbuf_list(buf, "TEST", 3); struct tpsocket_fd *sock = container_of(handler, struct tpsocket_fd, handler); STORAGE_SESSION *storage_session = handler->priv; if (event != TPSOCKET_EVENT_WRITABLE) { MYDEBUG("storage snapshot to cloud event = %s, sock = %p, session = %p\n", tpsocket_event_name(event), sock, storage_session); if (storage_session) { DBG_DBG("Device Id = %s, session sock = %p\n", storage_session->dev_id, storage_session->sock); } } switch(event) { case TPSOCKET_EVENT_LISTEN: break; case TPSOCKET_EVENT_SHUTDOWN: break; case TPSOCKET_EVENT_ACCEPT: break; case TPSOCKET_EVENT_CONNECTED: MYDEBUG("STORAGESNAPSHOT2CLOUD: %s:%s Connected\n", sock->addr, sock->port); if (false == storage_snapshot_consumer_post(storage_session)) return false; if (false == storage_start_read(storage_session)) { DBG_ERR("read snapshot file failed\n"); return false; } break; case TPSOCKET_EVENT_REQ_HEAD: break; case TPSOCKET_EVENT_RSP_HEAD: break; case TPSOCKET_EVENT_SUB_HEAD: break; case TPSOCKET_EVENT_UPGRADE: break; case TPSOCKET_EVENT_WRITABLE: if (storage_session->file && !list_empty(&(storage_session->file->parser.buf))) { if (storage_session->chunked && !tpsocket_chunk_encode(&(storage_session->file->parser.buf))) { tpsocket_free_buf(&(storage_session->file->parser.buf), tpsocket_event_name(event), 0); break; } storage_session_write_list_force(storage_session, &(storage_session->file->parser.buf), true); storage_session->file->read_buf_length = 0; } break; case TPSOCKET_EVENT_STREAM: return true; case TPSOCKET_EVENT_MESSAGE: if (!common_check_error_code(buf)) { DBG_DBG("storage snapshot push success\n"); storage_session->post_success = 1; } tpsocket_free_buf(buf, tpsocket_event_name(event), 0); common_tpsocket_close_immediately(sock); break; case TPSOCKET_EVENT_RESET: break; case TPSOCKET_EVENT_KEEPALIVE: common_tpsocket_close_immediately(sock); break; case TPSOCKET_EVENT_CLOSED: if (storage_session) { tpsocket_unbind(sock, &storage_session->sock); uloop_timeout_cancel(&storage_session->reupload_tmo); tpsocket_free2(&storage_session->file, storage_session); if (storage_session->post_success) { storage_session->snapshot_send_to_cloud_flag = 0; //storage_session_free(storage_session); } else { storage_session->retry_count++; //list_add_tail(&storage_session->list, &storage_session->storage_server->snapshot_session); } storage_session->uploading_flag = 0; storage_session_set_push_status(storage_session, 0); /* handle timeout immediately */ storage_server_timeout_handle_immediately(storage_session->storage_server); } break; case TPSOCKET_EVENT_ERROR: default: break; } tpsocket_free_buf(buf, tpsocket_event_name(event), 0); return true; } bool storage_stream_upload(STORAGE_SESSION *storage_session) { DEV_INFO *dev_info = NULL; STORAGE_SERVER *storage_server; if (!storage_session) { DBG_ERR("arg NULL\n"); return false; } if (storage_session->sock) { DBG_DBG("sock exist\n"); return true; } storage_server = storage_session->storage_server; uloop_timeout_set(&storage_session->reupload_tmo, storage_session->reupload_idle_time); dev_info = get_dev_info(storage_server->worker_ctx->top_ctx, storage_session->dev_id, NULL); if (!dev_info) { DBG_ERR("Not find device\n"); return false; } /* create socket */ struct tpsocket_handler storage_stream_to_cloud = { .cb = tpsocket_event_storage_stream_to_cloud, .cer = MEDIACENTER_CERTFILE, .write_buf_max = STORAGE_MEMORY_CACHE_LIMIT, }; storage_session->retry_count++; if ((storage_session->sock = tpsocket_from_url(dev_info->relay_storage_url, &storage_stream_to_cloud))) { //if (storage_session->sock = tpsocket_from_url("http://192.168.137.103:8010", &storage_stream_to_cloud)) { DBG_DBG("Create session sock = %p, session = %p\n", storage_session->sock, storage_session); storage_session->sock->handler.priv = storage_session; storage_session->uploading_flag = 1; storage_session_set_push_status(storage_session, 1); } return true; } void storage_set_iot_info(void *arg, const char *host, const char *cookie, const char *token, const char *event_id) { STORAGE_SESSION *storage_session; #ifdef TAPO_CAMERA if (!arg || !host || !token || !event_id) { #else if (!arg || !host || !cookie || !token || !event_id) { #endif DBG_ERR("arg NULL\n"); return; } storage_session = (STORAGE_SESSION*)arg; snprintf(storage_session->host, LEN_MAX_HOST, "%s", host); snprintf(storage_session->cookie, LEN_MAX_COOKIE, "%s", cookie); snprintf(storage_session->xtoken, LEN_MAX_TOKEN, "%s", token); snprintf(storage_session->event_id, LEN_MAX_EVENT_ID, "%s", event_id); MYDEBUG("host: %s\n", storage_session->host); //MYDEBUG("cookie: %s\n", storage_session->cookie); //MYDEBUG("xtoken: %s\n", storage_session->xtoken); MYDEBUG("event_id: %s\n", storage_session->event_id); return; } void storage_get_iot_info_cb(void *priv, void *iot_info, char *host, char *cookie, char *token, char *event_id, int err_code) { STORAGE_SESSION *storage_session; if (!priv) { DBG_ERR("arg NULL\n"); return; } storage_session = (STORAGE_SESSION*)priv; if (err_code) { DBG_ERR("storage get iot info failed\n"); goto out; } if (!host || !cookie || !token || !event_id) { DBG_ERR("storage get iot info param error\n"); goto out; } storage_set_iot_info(storage_session, host, cookie, token, event_id); MYDEBUG("Have got iot info, reupload now\n"); if (STORAGE_STREAMING == storage_session->storage_type) { storage_stream_upload(storage_session); return; } out: storage_session->retry_count++; return; } int storage_stream_reupload(STORAGE_SESSION *storage_session) { IOT_INFO *iot_info; if (!storage_session) { DBG_ERR("arg NULL\n"); return -1; } if (storage_session->event_id[0]) { MYDEBUG("Have got iot info, reupload now\n"); if (false == storage_stream_upload(storage_session)) { storage_session->stream_send_to_cloud_flag = 0; } } else { //find event id iot_info = iot_info_find(storage_session->storage_server->worker_ctx->iot_info_server, storage_session->dev_id , storage_session->count_id); if (iot_info) { if (iot_info->ready) { #ifdef TAPO_CAMERA storage_get_iot_info_cb((void*)storage_session, NULL, iot_info->host, iot_info->cookie, iot_info->xtoken, iot_info->alarm_id, 0); #else storage_get_iot_info_cb((void*)storage_session, NULL, iot_info->host, iot_info->cookie, iot_info->token, iot_info->event_id, 0); #endif if (IOT_INFO_CONSUMER_NUM == iot_info->consumer_used_count) { iot_info_free(iot_info); } } else { storage_session->retry_count++; } } else { storage_session->stream_send_to_cloud_flag = 0; } } return 0; } void storage_force_delete_session(STORAGE_SERVER *storage_server) { struct statfs diskInfo; STORAGE_SESSION *stream_session, *nstream_session; int force = 0; if (!storage_server) { DBG_ERR("arg NULL\n"); return; } retry: if (force++ < 2 && !statfs(STORAGE_ROOT_PATH, &diskInfo)) { int free_size = (diskInfo.f_bsize * diskInfo.f_bfree) >> 20; if (free_size < 20) { MYDEBUG("FREE MEMORY %u MB!\n", free_size); if (free_size < 8) { list_for_each_entry_safe(stream_session, nstream_session, &storage_server->stream_session, list) { if (!stream_session->iot_consumer) { if ((force == 1 && stream_session->delete_flag) || (force == 2 && !stream_session->uploading_flag)) { storage_session_free(stream_session); } } } goto retry; } } } } void storage_check_list_to_upload(STORAGE_SERVER *storage_server) { STORAGE_SESSION *stream_session, *nstream_session; int ret = 0; if (!storage_server) { DBG_ERR("arg NULL\n"); return; } list_for_each_entry_safe(stream_session, nstream_session, &storage_server->stream_session, list) { if (!stream_session->iot_consumer) {//make sure alarm push finished //MYDEBUG("find storage stream session = %p\n", stream_session); if (stream_session->uploading_flag) { MYDEBUG("stream session %p uploading push busy ...\n", stream_session); break; } ret = storage_session_get_push_status(stream_session); if (1 == ret) { MYDEBUG("this stream session %p is ready, but other session of the same device push busy ...\n", stream_session); break; } else if (-1 == ret) { MYDEBUG("device not found\n"); break; } if (stream_session->stream_send_to_cloud_flag && (stream_session->retry_count < 3)) { MYDEBUG("stream session %p do upload!\n", stream_session); storage_stream_reupload(stream_session); } else { //DBG_DBG("stream session %p can be deleted\n", stream_session); stream_session->delete_flag = true; } } else { MYDEBUG("waitting iot session %p closed\n", stream_session->iot_consumer); } } } void storage_server_timeout_handle_immediately(STORAGE_SERVER *storage_server) { MYDEBUG("storage_server_timeout_handle_immediately ...\n"); if (!storage_server) { DBG_ERR("arg NULL\n"); return; } uloop_timeout_set(&storage_server->tmo, 100); } void storage_server_timeout_cb(struct uloop_timeout *tmo) { //MYDEBUG("storage server timeout cb ...\n"); STORAGE_SERVER *storage_server = container_of(tmo, STORAGE_SERVER, tmo); if (!storage_server) { DBG_ERR("storage_server NULL\n"); return; } storage_force_delete_session(storage_server); storage_check_list_to_upload(storage_server); uloop_timeout_set(tmo, 10*1000); } void storage_session_set_send_flag(void *arg, int flag) { STORAGE_SESSION *storage_session = NULL; if (!arg) { DBG_ERR("arg NULL\n"); return; } storage_session = (STORAGE_SESSION*)arg; if (STORAGE_STREAMING == storage_session->storage_type) { storage_session->stream_send_to_cloud_flag = flag; } else { storage_session->snapshot_send_to_cloud_flag = flag; } return; } void storage_session_set_iot_consumer_null(void *arg) { STORAGE_SESSION *storage_session; if (!arg) { DBG_ERR("arg NULL\n"); return; } storage_session = (STORAGE_SESSION*)arg; storage_session->iot_consumer = NULL; storage_update_list(storage_session->storage_server, storage_session->storage_type); return; } int storage_delete_empty_dir(char *filedir) { int count = 0; DIR *dir = NULL; struct dirent *Dirent = NULL; //char cmd[128] = {0}; if(!filedir){ DBG_ERR("arg NULL\n"); return -1; } dir = opendir(filedir); if(dir == NULL){ DBG_NOTICE("opendir:%s error:%s\n", filedir, strerror(errno)); return -1; } while((Dirent = readdir(dir)) != NULL){ if(0 != strncmp(Dirent->d_name, ".", sizeof(".")) && 0 != strncmp(Dirent->d_name, "..", sizeof(".."))) { count++; } } closedir(dir); if(!count){ //memset(cmd, '\0', 128); //snprintf(cmd, 128, "rm -rf %s", filedir); //if (system(cmd) < 0) { if (tpcom_system("rm", ("-rf"), ("%s", filedir)) < 0) { DBG_ERR("remove:%s error:%s\n", filedir, strerror(errno)); return -1; } DBG_DBG("remove dir:%s ok\n", filedir); } else { DBG_DBG("has %d file in %s\n", count, filedir); } return 0; } void storage_delete_file(char *file) { char *pos; char dev_id[LEN_MAX_ID] = {0}; char count_id[LEN_MAX_ID] = {0}; char dir[LEN_MAX_PATH] = {0}; if (!file) { DBG_ERR("arg NULL\n"); return; } if (!file[0]) { DBG_DBG("No file to delete\n"); return; } pos = file+strlen(STORAGE_ROOT_PATH); DBG_DBG("pos = %s\n", pos); sscanf(pos, "/%[^/]/%[^/]/%*s", dev_id, count_id); DBG_DBG("dev_id = %s, count_id = %s\n", dev_id, count_id); /* delete file */ remove(file); /* delete count id dir */ memset(dir, 0, LEN_MAX_PATH); snprintf(dir, LEN_MAX_PATH, "%s/%s/%s", STORAGE_ROOT_PATH, dev_id, count_id); storage_delete_empty_dir(dir); /* delete device dir */ memset(dir, 0, LEN_MAX_PATH); snprintf(dir, LEN_MAX_PATH, "%s/%s", STORAGE_ROOT_PATH, dev_id); storage_delete_empty_dir(dir); } void storage_move_file(char *file) { char *pos; char cmd[512] = {0}; char dev_id[LEN_MAX_ID] = {0}; char count_id[LEN_MAX_ID] = {0}; char dir[LEN_MAX_PATH] = {0}; char count_path[LEN_MAX_PATH] = {0}; if (!file || !file[0]) { DBG_ERR("arg NULL\n"); return; } pos = file+strlen(STORAGE_ROOT_PATH); MYDEBUG("pos = %s\n", pos); sscanf(pos, "/%[^/]/%[^/]/%*s", dev_id, count_id); MYDEBUG("dev_id = %s, count_id = %s\n", dev_id, count_id); /* remove file */ snprintf(count_path, LEN_MAX_PATH, "%s/%s/%s/", NFS_ROOT, dev_id, count_id); if (access(count_path, F_OK)) { //snprintf(cmd, 512, "mkdir -p %s", count_path); //system(cmd); tpcom_system("mkdir", ("-p"), ("%s", count_path)); } memset(cmd, 0, 512); snprintf(cmd, 512, "mv %s %s", file, count_path); DEBUG("cmd: %s\n", cmd); //system(cmd); tpcom_system("mv", (file), ("%s", count_path)); if (!access(file, F_OK)) { DBG_ERR("%s failed\n", cmd); remove(file); } /* delete count id dir */ memset(dir, 0, LEN_MAX_PATH); snprintf(dir, LEN_MAX_PATH, "%s/%s/%s", STORAGE_ROOT_PATH, dev_id, count_id); storage_delete_empty_dir(dir); /* delete device dir */ memset(dir, 0, LEN_MAX_PATH); snprintf(dir, LEN_MAX_PATH, "%s/%s", STORAGE_ROOT_PATH, dev_id); storage_delete_empty_dir(dir); } void storage_session_free(STORAGE_SESSION *storage_session) { MYDEBUG("STORAGE SESSION FREE %p\n", storage_session); if (!storage_session) { return; } list_del(&storage_session->list); uloop_timeout_cancel(&storage_session->reupload_tmo); tpsocket_free_buf(&storage_session->headers, NULL, 0); if (storage_session->sock) { storage_session_set_push_status(storage_session, 0); } tpsocket_free2(&storage_session->sock, storage_session); tpsocket_free2(&storage_session->file, storage_session); if (storage_session->iot_consumer) { alarm_consumer_set_storage_session_null(storage_session->iot_consumer); } /* telemetry */ storage_session_telemetry_failure_count(storage_session); /* delete file */ if (STORAGE_STREAMING == storage_session->storage_type) { if (STORAGE_DEBUG && !access("/nfsroot", F_OK)) { storage_move_file(storage_session->stream_path); } else { storage_delete_file(storage_session->stream_path); } } else { if (STORAGE_DEBUG && !access("/nfsroot", F_OK)) { storage_move_file(storage_session->snapshot_path); } else { storage_delete_file(storage_session->snapshot_path); } } free(storage_session); } bool tpsocket_event_consumer_to_storage(struct tpsocket_handler *handler, struct list_head *buf, int event) { struct tpsocket_fd *sock = container_of(handler, struct tpsocket_fd, handler); STORAGE_SESSION *storage_session = (STORAGE_SESSION*)handler->priv; if (STORAGE_SNAPSHOT == storage_session->storage_type) { MYDEBUG("storage snpashot cb: sock=%p, event=%d\n", sock, event); } switch(event) { case TPSOCKET_EVENT_CONNECTED: break; case TPSOCKET_EVENT_REQ_HEAD: break; case TPSOCKET_EVENT_RSP_HEAD: break; case TPSOCKET_EVENT_SUB_HEAD: break; case TPSOCKET_EVENT_UPGRADE: break; case TPSOCKET_EVENT_MESSAGE: break; case TPSOCKET_EVENT_STREAM: break; case TPSOCKET_EVENT_RESET: break; case TPSOCKET_EVENT_CLOSED: if (storage_session) { storage_session->file = NULL; list_del_init(&storage_session->list); storage_session_add_session(storage_session); } break; case TPSOCKET_EVENT_ERROR: MYDEBUG("tpsocket_event_consumer_to_storage ERROR %p\n", storage_session); storage_session->storage_success = false; default: break; } tpsocket_free_buf(buf, tpsocket_event_name(event), 0); return true; } STORAGE_SESSION *storage_session_new(char *path, STORAGE_SERVER *storage_server) { STORAGE_SESSION *storage_session; if (!path || ! storage_server) { DBG_ERR("arg NULL\n"); return NULL; } storage_session = (STORAGE_SESSION*)malloc(sizeof(*storage_session)); if (!storage_session) { DBG_ERR("malloc failed\n"); return NULL; } memset(storage_session, 0, sizeof(*storage_session)); INIT_LIST_HEAD(&storage_session->list); INIT_LIST_HEAD(&storage_session->headers); storage_session->storage_server = storage_server; storage_session->reupload_idle_time = 10000; storage_session->reupload_tmo.cb = storage_session_reupload_timeout; storage_session->storage_success = true; storage_session->delete_flag = false; /* create socket */ struct tpsocket_handler client = { .cb = tpsocket_event_consumer_to_storage, .flowcontrol = 500000, .priv = storage_session }; storage_session->file = tpsocket_new(NULL, "write", NULL, path, TPSOCKET_TYPE_FILE, &client); if (!(storage_session->file)) { DBG_ERR("tpsocket_new failed\n"); free(storage_session); return NULL; } //consumer->chunked = 0; //consumer->store = 1; return storage_session; } int storage_find_file_in_list(char *dev_id, char *timestamp_str, struct list_head *head) { STORAGE_SESSION *session; int timestamp; if (!dev_id || !timestamp_str || !head) { DBG_ERR("arg NULL\n"); return -1; } timestamp = atoi(timestamp_str); list_for_each_entry(session, head, list) { if (!strncmp(session->dev_id, dev_id, LEN_MAX_ID) && (session->event_timestamp == timestamp)) { return 0; } } return -1; } void storage_check_file(STORAGE_SERVER *storage_server) { if (!storage_server) { DBG_ERR("arg NULL\n"); return; } DIR *dir = NULL, *dev_dir = NULL; struct dirent *root_dirent = NULL, *dev_dirent = NULL; char dev_id[LEN_MAX_ID] = {0}, timestamp[LEN_MAX_ID] = {0}; char dev_path[LEN_MAX_PATH] = {0}, event_path[LEN_MAX_PATH] = {0}, file_path[LEN_MAX_PATH] = {0}; dir = opendir(STORAGE_ROOT_PATH); if(dir == NULL){ DBG_NOTICE("opendir:%s error:%s\n", STORAGE_ROOT_PATH, strerror(errno)); return; } while((root_dirent = readdir(dir)) != NULL){ /* dev */ if(0 == strncmp(root_dirent->d_name, ".", sizeof(".")) || 0 == strncmp(root_dirent->d_name, "..", sizeof(".."))) { continue; } memset(dev_path, 0, LEN_MAX_PATH); snprintf(dev_path, LEN_MAX_PATH, "%s/%s", STORAGE_ROOT_PATH, root_dirent->d_name); dev_dir = opendir(dev_path); if(dev_dir == NULL){ DBG_NOTICE("opendir:%s error:%s\n", dev_path, strerror(errno)); continue; } memset(dev_id, 0, LEN_MAX_ID); snprintf(dev_id, LEN_MAX_ID, "%s", root_dirent->d_name); while ((dev_dirent = readdir(dev_dir)) != NULL) { /* event */ if(0 == strncmp(dev_dirent->d_name, ".", sizeof(".")) || 0 == strncmp(dev_dirent->d_name, "..", sizeof(".."))) { continue; } memset(timestamp, 0, LEN_MAX_ID); snprintf(timestamp, LEN_MAX_ID, "%s", dev_dirent->d_name); memset(event_path, 0, LEN_MAX_PATH); snprintf(event_path, LEN_MAX_PATH, "%s/%s/%s", STORAGE_ROOT_PATH, dev_id, timestamp); memset(file_path, 0, LEN_MAX_PATH); snprintf(file_path, LEN_MAX_PATH, "%s/%s/%s/%s", STORAGE_ROOT_PATH, dev_id, timestamp, STORAGE_STREAMING_FILE); if (!access(file_path, F_OK)) { DBG_NOTICE("streaming file: %s\n", file_path); if (storage_find_file_in_list(dev_id, timestamp, &storage_server->stream_session)) { storage_delete_file(file_path); } } memset(file_path, 0, LEN_MAX_PATH); snprintf(file_path, LEN_MAX_PATH, "%s/%s/%s/%s", STORAGE_ROOT_PATH, dev_id, timestamp, STORAGE_SNAPSHOT_FILE); if (!access(file_path, F_OK)) { DBG_NOTICE("snapshot file: %s\n", file_path); if (storage_find_file_in_list(dev_id, timestamp, &storage_server->snapshot_session)) { storage_delete_file(file_path); } } storage_delete_empty_dir(event_path); } closedir(dev_dir); } closedir(dir); return; } void storage_get_iot_info(STORAGE_SESSION *session, struct list_head *iot_info_head) { IOT_INFO *iot_info; if (!session || !iot_info_head) { DBG_ERR("arg NULL\n"); return; } list_for_each_entry(iot_info, iot_info_head, list) { if (!strncmp(session->dev_id, iot_info->dev_id, LEN_MAX_ID) && (session->count_id == iot_info->count_id)) { #ifdef TAPO_CAMERA storage_set_iot_info(session, iot_info->host, iot_info->cookie, iot_info->xtoken, iot_info->alarm_id); #else storage_set_iot_info(session, iot_info->host, iot_info->cookie, iot_info->token, iot_info->event_id); #endif break; } } } bool storage_check_version() { FILE *version_fd; char version_path[LEN_MAX_PATH] = {0}, version[4] = {0}; //char cmd[128] = {0}; snprintf(version_path, LEN_MAX_PATH, "%s/%s", STORAGE_ROOT_PATH, STORAGE_VERSION_FILE); if (access(version_path, F_OK)) { DBG_NOTICE("storage version file not exist\n"); goto failed; } if (!(version_fd = fopen(version_path, "r"))) { DBG_NOTICE("open storage version file failed\n"); goto failed; } if (1 != fread(version, 4, 1, version_fd)) { DBG_NOTICE("read storage version file error\n"); fclose(version_fd); goto failed; } fclose(version_fd); if (strncmp(version, STORAGE_VERSION, 4)) { DBG_NOTICE("old version\n"); goto failed; } return true; failed: /* delete storage */ //snprintf(cmd, 128, "rm -rf %s/*", STORAGE_ROOT_PATH); //system(cmd); tpcom_system("rm", ("-rf"), ("%s", STORAGE_ROOT_PATH)); tpcom_system("mkdir", ("-p"), ("%s", STORAGE_ROOT_PATH)); /* create version */ snprintf(version, 4, "%s", STORAGE_VERSION); version_fd = fopen(version_path, "w"); if (version_fd) { fwrite(version, 4, 1, version_fd); fclose(version_fd); } return false; } void storage_file_list_init(STORAGE_SERVER *storage_server) { FILE *iot_info_fd, *stream_fd, *snapshot_fd; char iot_info_path[LEN_MAX_PATH] = {0}, stream_path[LEN_MAX_PATH] = {0}, snapshot_path[LEN_MAX_PATH] = {0}; struct list_head list; IOT_INFO *iot_info, *niot_info; if (!storage_server) { DBG_ERR("arg NULL\n"); return; } /* check version */ if (false == storage_check_version()) { return; } /* iot info */ INIT_LIST_HEAD(&list); snprintf(iot_info_path, LEN_MAX_PATH, "%s/%s", STORAGE_ROOT_PATH, STORAGE_IOT_INFO_LIST); iot_info_fd = fopen(iot_info_path, "r"); if (iot_info_fd) { while (1) { iot_info = malloc(sizeof(IOT_INFO)); if (fread(iot_info, sizeof(IOT_INFO), 1, iot_info_fd) <= 0) { free(iot_info); break; } list_add_tail(&iot_info->list, &list); } fclose(iot_info_fd); } /* stream session */ snprintf(stream_path, LEN_MAX_PATH, "%s/%s", STORAGE_ROOT_PATH, STORAGE_STREAMING_LIST); stream_fd = fopen(stream_path, "r"); if (stream_fd) { while (1) { STORAGE_SESSION *stream_session = malloc(sizeof(STORAGE_SESSION)); if (fread(stream_session, sizeof(STORAGE_SESSION), 1, stream_fd) <= 0) { free(stream_session); break; } DBG_NOTICE("dev_id: %s, count_id: %d\n", stream_session->dev_id, stream_session->count_id); INIT_LIST_HEAD(&stream_session->list); INIT_LIST_HEAD(&stream_session->headers); stream_session->sock = NULL; stream_session->file = NULL; stream_session->uploading_flag = 0; stream_session->producer = NULL; stream_session->iot_consumer = NULL; stream_session->storage_server = storage_server; storage_session_add_fixed_header(stream_session); memset(&stream_session->reupload_tmo, 0, sizeof(struct uloop_timeout)); list_add_tail(&stream_session->list, &storage_server->stream_session); storage_get_iot_info(stream_session, &list); } fclose(stream_fd); } /* snapshot session */ snprintf(snapshot_path, LEN_MAX_PATH, "%s/%s", STORAGE_ROOT_PATH, STORAGE_SNAPSHOT_LIST); snapshot_fd = fopen(snapshot_path, "r"); if (snapshot_fd) { while (1) { STORAGE_SESSION *snapshot_session = malloc(sizeof(STORAGE_SESSION)); if (fread(snapshot_session, sizeof(STORAGE_SESSION), 1, snapshot_fd) <= 0) { free(snapshot_session); break; } DBG_NOTICE("dev_id: %s, count_id: %d\n", snapshot_session->dev_id, snapshot_session->count_id); INIT_LIST_HEAD(&snapshot_session->list); INIT_LIST_HEAD(&snapshot_session->headers); snapshot_session->sock = NULL; snapshot_session->file = NULL; snapshot_session->uploading_flag = 0; snapshot_session->producer = NULL; snapshot_session->iot_consumer = NULL; snapshot_session->storage_server = storage_server; memset(&snapshot_session->reupload_tmo, 0, sizeof(struct uloop_timeout)); list_add_tail(&snapshot_session->list, &storage_server->snapshot_session); storage_get_iot_info(snapshot_session, &list); } fclose(snapshot_fd); } /* free & delete */ list_for_each_entry_safe(iot_info, niot_info, &list, list) { free(iot_info); } /* check file */ storage_check_file(storage_server); return; } void storage_server_free(STORAGE_SERVER *storage_server) { STORAGE_SESSION *storage_session, *nstorage_session; if (!storage_server) { return; } uloop_timeout_cancel(&storage_server->tmo); list_for_each_entry_safe(storage_session, nstorage_session, &storage_server->stream_session, list) { storage_session_free(storage_session); } list_for_each_entry_safe(storage_session, nstorage_session, &storage_server->snapshot_session, list) { storage_session_free(storage_session); } if (storage_server->worker_ctx) { storage_server->worker_ctx->storage_server = NULL; } list_del_init(&storage_server->list); free(storage_server); } int storage_server_init(WORKER_CTX *worker_ctx) { STORAGE_SERVER *storage_server; if (!worker_ctx) { DBG_ERR("worker ctx NULL\n"); return -1; } /* create */ storage_server = (STORAGE_SERVER*)malloc(sizeof(*storage_server)); if (!storage_server) { DBG_ERR("malloc failed\n"); return -1; } memset(storage_server, 0, sizeof(*storage_server)); storage_server->worker_ctx = worker_ctx; INIT_LIST_HEAD(&storage_server->list); INIT_LIST_HEAD(&storage_server->stream_session); INIT_LIST_HEAD(&storage_server->snapshot_session); //timer to re-upload storage_server->tmo.cb = storage_server_timeout_cb; uloop_timeout_set(&storage_server->tmo, 10*1000); /* set */ worker_ctx->storage_server = storage_server; /* file list init */ storage_file_list_init(storage_server); return 0; }
最新发布
09-23
评论
成就一亿技术人!
拼手气红包6.0元
还能输入1000个字符
 
红包 添加红包
表情包 插入表情
 条评论被折叠 查看
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值