Xtoken

“我希望有一种模式,利用群体的智慧让最好的想法总能够脱颖而出”。

博弈模型

 

背景

本文为NEO社区理事会秘书长陶荣祺在全球创业周区块链创新与发展论坛上的主题演讲《Xtoken代观社区驱动群体智慧涌现投资决策范式》,应观众要求整理成文。

 

 

 

大家好,我是来自NEO理事会的陶荣祺。今天我讲的题目是《Xtoken代观社区驱动群体智慧涌现投资决策范式》。同时也会有很多的人参与到这个项目或者那个项目的投资里边去。请问到底什么样的投资决策才是正确的,什么样的项目才是有价值的?我作为一个社区的参与者始终在寻找这样问题的答案。今天讲的这个东西可能是最近几年经验的总结。

 

这段话来自于桥水基金的创始人Ray Dalio ,“我希望有一个模式,让最好的想法脱颖而出”。桥水基金是一个比较特殊的基金,其实这是我们每个人的想法,有没有一种模式或者最好的投资决策脱颖而出。那是怎么做的呢?他们的思维是一种精英的思维模式,并不是完全平等,也不是完全集权的。基因意味着什么?极度真实,极度透明,依靠算法进行决策。

 

 

什么叫做真实呢?在桥水基金内部他们的理念是所有的人需要表达出他们真正相信的,透明意味着所有的人需要能够获得他们需要的一切信息,并不会对桥水基金内部的人有任何的遮掩。依靠算法意味着可信权重作为算法的基础,依靠算法并不是依靠很多人的嘴巴,而是把它固化成算法。

 

我们看这些东西的时候再对比一下我们周围的模式会发现区别是很大的,比如说我们会发现社区里边关于投资的建议,请问某些大佬或者某些的意见领袖投资决策是不是沟通的人里边能够获得所需要的信息,他们的投资决策是不是以算法作为基础。这些问题在我看来全部都不是,这是桥水基金的做法。

 

 

这是一张内部做决策的矩阵列表,同时这张图是来自于桥水基金戴利奥的最新TED演讲。内部所有人参与到决策里边去,信息公开透明,同时用一种结构化的数据作出决策。

 

 

可信权重是这套算法里边的基础,目的是帮助人们表达自己的意见,同时帮助人们从更高的层次去观察整体。简单说就是开了一个上帝视角,你有没有机会脱离所有的干扰观察到整个全局,这样是不是就能作出最好的决策。当我们玩一些战略游戏的时候我们总是觉得如果能从上帝视角看整个决策的时候就能作出最好的决策,而不是一团黑雾,走到哪里就能散出黑雾。

 

 

 

 

 “我希望有一种模式,利用群体的智慧让最好的想法总能够脱颖而出”。这句话是我跟桥水的话衍生出来有没有机会呢?不知道,但是我们尝试着做一下。

 

各位可以看一下这段视频,这段并不是做出来的,是用的一些模型、算法自然而然跑出来的。这张模型叫做博弈模型,每个点就是一只鸟,并不是控制好让每个鸟或者每个群体应该怎么飞,并不是如此。

 

这段模型在80年代的时候有一些经典的学术代表提出来的这套理论,是一套研究人的行为的群体智慧的模型。根据这套模型,其实人的智慧或者我们整个社会群体的行为看上去很纷繁复杂,实际上背后的逻辑并不复杂。这些东西在我们看出来似乎不能够了解到底怎么飞,但是实际上每个群体都有基本的规律。

 

每个模型都要保持一定的距离,以至于不要撞到一起。第二,一致性,每个个体之间都往一个方向飞。第三,凝聚力控制。

 

 这样一种模型有没有机会应用到区块链社区里边体现出群体智慧呢?

 

群体智慧是一种无中心集中控制的组织结构,基于区块链的特区里边原本有本质性的常态。其实我说的这些东西只不过是一种解释和描述,并不是说我发明了这些东西,可以说我发现了这些,或者说把一些模型和我所观察到的生态结合到一起。如果各位参与到区块链的社区和其他的社区运作里边是不是也符合刚才描述的状态,比如说周围的人投资于某一个项目的时候是不是跟着他投,同时又保持一定的距离,同时又会跟着他的方向做。这样的投资决策是不是我们发现的状态了。

 

 

区块链重构Bodis Model规则。利用Boids  Model规则,保持一致性协议、分离协议、凝聚力协议,区块链结构下这三点可以描述成每个节点之间是不是透明,每个节点之间具有独立的特征,区块链每个结点都非常清晰,总能够以此作为一种采纳下一个行为的依据。

 

由此我们就得出分布式下的新组织结构。这里提到The Dao,这个项目是去年五六月份的项目,TheDao的创新是提出了去中心化的自治组织,智能合约管理信任,处理意见不一致是DAO分离带来无限递归问题。不得不采用分离的方式来解决整个组织里边意见的不一致,因为对DAO这个项目来说做法是先组织一大批人在一起,然后再让大家分裂,因为不得不分裂。

 

 

我们认为这种模式是有些问题的,但是也有创新之处。我们在看组织净化的时候我们发现其实DAO提出关于组织的理论是整个人类去中心化其中一部分。这边是来自于青色组织或者组织净化提出来的过程,最里边是狼性组织,再往上是人人平等的概念,最外面是青色组织。青色组织是自我组织,所谓自组织。

 

 我们去看我们想要的,或者我们现在已经面临这样的社会里边各种各样的组织机构的确有各种各样的形式,我们从自我的感知去认知的时候会发现可能在现在,在区块链的时代里边已经有一种更好的组织架构体现出来的。这种组织架构就和这里描述的青色组织非常一致。

 

 

青色组织的社区技术—“合弄制”。这并不是我提出来的,很出名,合弄制是美国一位研究整个历史,整个组织架构进程提出来的概念。合弄制里边会说到底什么样的组织架构是更好的,是我们刚刚说的一层一层往下的,还是每个都平等的,抑或是组织结构,并不是层级,也不是完全平等,是一个圈。

 

 采用合弄制,或者和青色组织靠拢,再加上自我的观察是不是提出分布式权利的建立,是一种可靠的解决方案呢?假设说有这样的组织架构,首先有提案人提出一个提案,然后来到了一个问题的阶段,有一个组织里边有一个人作一个提案人,其他人会参与到提案人的问题,也就是所谓的共识圈。比如说我在一个群里边发起一个提案,我提出了这样的提案,我想做这个事情,很多人会给我反馈,反馈之后我应该做还是不做,我应该改还是不改其实都无所谓,重要的是我和这个群体,和这个圈子产生了一个交互,我让这个圈子里边的人参与到我的这个提案,我这个圈子里边来了,成为圈子里的一部分。

 

 我可以收集他们的意见,进行修补、改进和澄清,这是没什么问题的。包括第五阶段也会有一些反面的意见,没有什么问题,就应该是这样。最后做决策是大家一起做决策吗?不是的,最后还是提案者整合这些东西,最后得出一个决策。也就是说在这套决策机制里边提案者可以是传统理念,最后提出提案的人也会把这个提案做到底,不管到底有多少人认同或者不认同,我们发现在这个决策里边和整个圈子的人沟通和交互的过程是非常重要的。一方面可以把提案磨炼的更好,但更重要的是可以让这些人参与到我这个提案的意见、圈子,以及共同形成社区的过程里去,这才是更重要的。

 

首先这是一个美式的做法,在于我提出了提案,我会把这件事做到底,这是一个个人主义,但是又不如此。因为我在产生这样一个圈子的时候我需要和圈子里边的人产生充分的交互,这些圈子里的人就是我的社区,这就是继续往下走的动作。

    

这件事情可以认为是既形成了一个坚定的,能够挖掘个人意志的,能够把我的提案和能动性充分挖掘出来的一种方案,同时又形成了一个社区,让一些社区支持我的方式方法的人站在我的周围,同时让他们的意见帮助我更好地提升我的提案。这就是合支弄制的本质。所谓圈子会围绕一个核心,围绕这个核心会不断地把这个圈子作大。

 

    

 

 

合弄制没有解决的问题是什么?没有解决的问题是组织内的经济激励,以及确权的帐本。我们去想象一下合弄制的状态,一开始能够做下去,但是当它做到后面的时候由于这个组织内部没有一种经济的激励,我们可以想见跟着这些人,走到后面发现我好像没有切实的好处,这种好处我们认为是一种经济激励的好处,兴趣容易慢慢的衰退、衰减,是一种信仰,可能衰减的更加慢一点,但是也会衰减。但是作为一种补偿或者长期的平衡,我们引入一种经济激励的话能够让圈子的稳定状态更加的长久。

    

需要有一个经济激励的时候就需要有一个确权帐本,而这两点都是区块链能够带来的。我们发现这张图上面分布式权利和确权帐本是不同的。这样一套组织架构我们认为是DAO2.0,既有分布式权利,又有组织内的经济激励,又有确权帐本。

 

    

我们提一下群体智慧的决策系统。首先我如何知道我的选择是对的?我是集权管理,我是老大,我是独裁者,我如何知道我这个方向到底对还是不对,因为没有人能够反对我,这是很困扰的一件事情,站在最上面的人往往拿不到最下面人的意见。这就是集权国家治理遇到的很严重的问题。

    

第二,是否有绝对最优的方案。我们每个人都是平等的,请问在每个人都是平等的过程里边到底哪一个方案才是最正确的,那就到分裂,这是绝对民主所面临的问题,导致一盘散沙。

    

社区信任建立的过程里边我们提出了从认可到认同,最后提出贡献的三步走。首先是建立信心,然后维持关系,最后是拓展事业。你在做这些事情的过程里边首先是一种认同,然后是通过一种区块链的经济激励来让这个社区长期的凝聚到一起,把这个圈子一起做大。

    

我们会发现建立这个圈子的时候我们是一种三层的建立机制,第一层是SupporterCircle,第二层是共识圈,第三层是执行圈。这三层里边所需要付出的职责,得到的回报都是不一样的。

    

到2.0的组织架构里边就是推一种信任渐进圈的机制。最外层是支持圈,第二层是共识圈,第三层是践行圈。最外层的人只看不做,第二层是共识圈,来到共识圈你需要对社区有一种共识,你需要和社区的核心意识形成达成一种共识。第三,践行圈,你需要做,做的过程需要进行交互。我们需要保障这个共识。所以需要有一种共识的稳定,以及挖掘这种共识,并不会走向集权过程的一种保障。它是有生命周期的。

    

早期贡献的意义是什么?成员的准入协议、联合共识协议、激励机制。同时我们要减弱群体思维的影响,所以引入挑战圈。我们都知道乌合之众这个概念,乌合之众就是所谓的群体思维起到了负面的作用,大家都不思考了,跟着一个引路人走了,后面跳到山崖里去了。挑战圈的人只做件事情就是不断地挑战,不断地挑战。

    

最后再说一下NEO,NEO是智能经济,我们提出了一个理念,数字资产+数字身份+智能合约。是世界领先的区块链社区。

    

这是我新做的项目,叫Xtoken,是基于刚才一套群体智慧的理念做出来的项目,目标是一起发现价值。这就是即将在未来的几个月里边做的项目。也请各位多多关注。谢谢!

 

转载于:https://www.cnblogs.com/xujiahui/p/8466058.html

#include <dirent.h> #include <sys/statfs.h> #include "common.h" #include "storage.h" #include "iot_info.h" #include "device.h" #ifdef HEALTH_INFO_SUPPORT #include "health_info.h" #endif void storage_session_free(STORAGE_SESSION *storage_session); void storage_server_timeout_handle_immediately(STORAGE_SERVER *storage_server); /*---------------------------------------------------------*/ void storage_delete_posted_session(STORAGE_SERVER *storage_server, char *dev_id, int storage_type) { STORAGE_SESSION *session, *nsession; if (!storage_server || !dev_id) { DBG_ERR("arg NULL\n"); return; } if (STORAGE_STREAMING == storage_type) { list_for_each_entry_safe(session, nsession, &storage_server->stream_session, list) { if (!strncmp(session->dev_id, dev_id, LEN_MAX_ID) && !session->uploading_flag && (!session->stream_send_to_cloud_flag || session->delete_flag)) { storage_session_free(session); } } } else if (STORAGE_SNAPSHOT == storage_type) { list_for_each_entry_safe(session, nsession, &storage_server->snapshot_session, list) { DBG_DBG("%s %s %d %d %d\n", dev_id,session->dev_id, session->uploading_flag, session->snapshot_send_to_cloud_flag, session->delete_flag); if (!strncmp(session->dev_id, dev_id, LEN_MAX_ID) && !session->uploading_flag && (!session->snapshot_send_to_cloud_flag || session->delete_flag)) { storage_session_free(session); } } } } static void storage_session_telemetry_failure_count(STORAGE_SESSION *storage_session) { DEV_INFO *dev_info; MEDIACENTER_CTX *mediacenter_ctx; if (!storage_session) { DBG_ERR("arg NULL\n"); return; } mediacenter_ctx = storage_session->storage_server->worker_ctx->top_ctx; dev_info = get_dev_info(mediacenter_ctx, storage_session->dev_id, storage_session->dev_ip); if (dev_info && storage_session->post_success && (dev_info->telemetry.relay_clips_failure_count > 0)) { dev_info->telemetry.relay_clips_failure_count--; #ifdef HEALTH_INFO_SUPPORT dev_update_health_info(dev_info->dev_id); #endif } return; } void storage_delete_sessions_of_device(MEDIACENTER_CTX *mediacenter_ctx, char *dev_id) { WORKER_CTX *worker_ctx = NULL; STORAGE_SERVER *storage_server = NULL; STORAGE_SESSION *session = NULL, *nsession = NULL; if (!mediacenter_ctx || !dev_id) { DBG_ERR("arg NULL\n"); return; } if (list_empty(&mediacenter_ctx->worker_list)) { DBG_ERR("worker list empty\n"); return; } worker_ctx = (WORKER_CTX *)list_first_entry(&mediacenter_ctx->worker_list, STORAGE_SESSION, list); if (worker_ctx) { storage_server = worker_ctx->storage_server; if (storage_server) { list_for_each_entry_safe(session, nsession, &storage_server->stream_session, list) { if (!strncmp(session->dev_id, dev_id, LEN_MAX_ID)) { storage_session_free(session); } } list_for_each_entry_safe(session, nsession, &storage_server->snapshot_session, list) { if (!strncmp(session->dev_id, dev_id, LEN_MAX_ID)) { storage_session_free(session); } } } } } int storage_session_get_push_status(STORAGE_SESSION *session) { DEV_INFO *dev_info; MEDIACENTER_CTX *mediacenter_ctx; if (!session) { DBG_ERR("arg NULL\n"); return -1; } mediacenter_ctx = session->storage_server->worker_ctx->top_ctx; dev_info = get_dev_info(mediacenter_ctx, session->dev_id, session->dev_ip); if (dev_info) { if (STORAGE_STREAMING == session->storage_type) { return dev_info->alarm_stream_push_status; } else { return dev_info->alarm_snapshot_push_status; } } return -1; } void storage_session_set_push_status(STORAGE_SESSION *session, int status) { DEV_INFO *dev_info; MEDIACENTER_CTX *mediacenter_ctx; if (!session) { DBG_ERR("arg NULL\n"); return; } mediacenter_ctx = session->storage_server->worker_ctx->top_ctx; dev_info = get_dev_info(mediacenter_ctx, session->dev_id, session->dev_ip); if (dev_info) { DBG_INFO("It is alarm re-upload below\n"); if (STORAGE_STREAMING == session->storage_type) { dev_set_alarm_stream_status(dev_info, status); } else { dev_set_alarm_snapshot_status(dev_info, status); } } return; } void storage_session_reupload_timeout(struct uloop_timeout* tmo) { //MYDEBUG("STORAGE SESSION REUPLOAD CHECK TIMEOUT\n"); unsigned long long sent; STORAGE_SESSION *storage_session = container_of(tmo, STORAGE_SESSION, reupload_tmo); if (!storage_session) { DBG_ERR("arg NULL\n"); return; } uloop_timeout_set(&storage_session->reupload_tmo, storage_session->reupload_idle_time); if (storage_session->sock) { sent = storage_session->reupload_total_load - storage_session->sock->write_buf_length; if (sent <= storage_session->reupload_last_sent) { DBG_ERR("STORAGE SESSION REUPLOAD TIMEOUT\n"); tpsocket_free(storage_session->sock); } else { storage_session->reupload_last_sent = sent; storage_session->reupload_last_load = storage_session->reupload_total_load; } } } bool storage_session_write_list_force(STORAGE_SESSION *storage_session, struct list_head *head, bool force) { struct tpsocket_buf *buf; unsigned long long to_send = 0; if (!storage_session || !head) { DBG_ERR("arg NULL\n"); return false; } /* data length */ list_for_each_entry(buf, head, list) { to_send += tpbuf_data_len(buf); } /* move bufs to sock */ if (false == tpsocket_write_list_force(storage_session->sock, head, force)) { DBG_ERR("tpsocket_write_list error\n"); return false; } /* calculate total load */ storage_session->reupload_total_load += to_send; return true; } void storage_session_stream_add_tail(STORAGE_SESSION *storage_session) { struct list_head head; if (!storage_session) { DBG_ERR("arg NULL\n"); return; } if (storage_session->chunked) { INIT_LIST_HEAD(&head); tpsocket_chunk_encode(&head);//generate chunk end '0\r\n' tpsocket_write_list_force(storage_session->sock, &head, true); } tpsocket_write_list_force(storage_session->sock, &head, true); } void storage_session_snapshot_add_tail(STORAGE_SESSION *storage_session) { struct tpsocket_buf *mybuf = NULL; struct list_head head; if (!storage_session) { DBG_ERR("arg NULL\n"); return; } INIT_LIST_HEAD(&head); mybuf = tpbuf_snprintf(LEN_MAX_128, "\r\n"); if (mybuf) { list_add_tail(&mybuf->list, &head); } if (storage_session->region_buf_len) { mybuf = tpbuf_snprintf(LEN_MAX_128*4, "%s", storage_session->region_buf); if (mybuf) { list_add_tail(&mybuf->list, &head); } } mybuf = tpbuf_snprintf(LEN_MAX_128, "--%s--", SNAPSHOT_FILE_BOUNDARY);//generate snapshot multi-part end if (mybuf) { list_add_tail(&mybuf->list, &head); } if (storage_session->chunked) { tpsocket_chunk_encode(&head); } tpsocket_write_list_force(storage_session->sock, &head, true); } int storage_session_set_region(STORAGE_SESSION *storage_session, TRIGGER_REGION *region) { if (!storage_session || !region) { DBG_ERR("arg NULL\n"); return -1; } if (0 == region->x1 && 0 == region->y1 && 0 == region->x2 && 0 == region->y2) { } else { storage_session->region_buf_len = snprintf(storage_session->region_buf, LEN_MAX_128*4, "--%s\r\n" "Content-Disposition: form-data;name=\"region\"\r\n\r\n" "{\"top\": %d, \"bottom\": %d, \"left\": %d, \"right\": %d}\r\n", SNAPSHOT_FILE_BOUNDARY, region->y1, region->y2, region->x1, region->x2); MYDEBUG("%s", storage_session->region_buf); } return 0; } int storage_session_set_event_timestamp(STORAGE_SESSION *storage_session, time_t timestamp) { if (!storage_session) { DBG_ERR("arg NULL\n"); return -1; } //storage_session->event_timestamp = timestamp - common_get_time_diff(); storage_session->event_timestamp = timestamp; return 0; } int storage_session_set_event_type(STORAGE_SESSION *storage_session, enum EVENT_TYPE event_type) { if (!storage_session) { DBG_ERR("arg NULL\n"); return -1; } storage_session->event_type = event_type; return 0; } void storage_update_list(STORAGE_SERVER *storage_server, int flag) { STORAGE_SESSION *storage_session; char list_path[LEN_MAX_PATH] = {0}; FILE *fd; if (!storage_server) { DBG_ERR("arg NULL\n"); return; } if (STORAGE_STREAMING == flag) { memset(list_path, 0, LEN_MAX_PATH); snprintf(list_path, LEN_MAX_PATH, "%s/%s", STORAGE_ROOT_PATH, STORAGE_STREAMING_LIST); fd = fopen(list_path, "w"); if (fd) { list_for_each_entry(storage_session, &storage_server->stream_session, list) { fwrite(storage_session, sizeof(*storage_session), 1, fd); } fclose(fd); } } if (STORAGE_SNAPSHOT == flag) { memset(list_path, 0, LEN_MAX_PATH); snprintf(list_path, LEN_MAX_PATH, "%s/%s", STORAGE_ROOT_PATH, STORAGE_SNAPSHOT_LIST); fd = fopen(list_path, "w"); if (fd) { list_for_each_entry(storage_session, &storage_server->snapshot_session, list) { fwrite(storage_session, sizeof(*storage_session), 1, fd); } fclose(fd); } } } void storage_session_add_session(STORAGE_SESSION *storage_session) { STORAGE_SERVER *storage_server; STORAGE_SESSION *session, *nsession; if (!storage_session) { MYDEBUG("arg NULL\n"); return; } storage_server = storage_session->storage_server; if (STORAGE_STREAMING == storage_session->storage_type) { MYDEBUG("storage stream session = %p\n", storage_session); if (false == storage_session->storage_success) { storage_session_free(storage_session); return; } list_for_each_entry_safe(session, nsession, &storage_server->stream_session, list) { if (!strncmp(session->dev_id, storage_session->dev_id, LEN_MAX_ID) && !session->uploading_flag) { if (!strncmp(session->stream_path, storage_session->stream_path, LEN_MAX_PATH)) memset(session->stream_path, 0, LEN_MAX_PATH);//if old session's path is the same as new, do not delete file storage_session_free(session); } } if (get_dev_info(storage_server->worker_ctx->top_ctx, storage_session->dev_id, NULL)) { list_add_tail(&storage_session->list, &storage_server->stream_session); storage_update_list(storage_server, STORAGE_STREAMING); } else { //if device not exist, free session storage_session_free(storage_session); } } else { MYDEBUG("storage snapshot session = %p\n", storage_session); if (false == storage_session->storage_success) { storage_session_free(storage_session); return; } list_for_each_entry_safe(session, nsession, &storage_server->snapshot_session, list) { if (!strncmp(session->dev_id, storage_session->dev_id, LEN_MAX_ID) && !session->uploading_flag) { if (!strncmp(session->snapshot_path, storage_session->snapshot_path, LEN_MAX_PATH)) memset(session->snapshot_path, 0, LEN_MAX_PATH); storage_session_free(session); } } if (get_dev_info(storage_server->worker_ctx->top_ctx, storage_session->dev_id, NULL)) { list_add_tail(&storage_session->list, &storage_server->snapshot_session); storage_update_list(storage_server, STORAGE_SNAPSHOT); } else { //if device not exist, free session storage_session_free(storage_session); } } return; } bool storage_session_add_fixed_header(STORAGE_SESSION *storage_session) { struct tpsocket_buf *buf; if (!storage_session) { DBG_ERR("arg NULL\n"); return false; } /* add */ buf = tpbuf_snprintf(512, "%x\r\n" "--data-boundary--\r\n" "Content-Length: 218\r\n" "Content-Type: text/http-header\r\n" "\r\n" "HTTP/1.0 200 OK\r\n" "Content-Type: multipart/x-mixed-replace;boundary=data-boundary--\r\n" "X-Encrypt-Type: PLAIN\r\n" "Pragma: no-cache\r\n" "Cache-Control: no-cache\r\n" "Content-Length: -1\r\n" "Transfer-Encoding: chunked\r\n" "Connection: close\r\n\r\n" "\r\n", 292); if (buf) { list_add_tail(&buf->list, &storage_session->headers); } return true; } bool storage_stream_consumer_post(STORAGE_SESSION *storage_session) { struct list_head head; struct tpsocket_buf *buf; DEV_INFO *dev_info; char *request = NULL; if (!storage_session) { DBG_ERR("arg NULL\n"); return false; } dev_info = get_dev_info(storage_session->storage_server->worker_ctx->top_ctx, storage_session->dev_id, NULL); if (!dev_info) { DBG_ERR("get dev info failed\n"); return false; } INIT_LIST_HEAD(&head); char url[256] = {0}; request = strstr(dev_info->relay_storage_url, "/v1"); snprintf(url, 256, "%sactivity?deviceid=%s" "&eventid=%s" "&type=mixed" "&resolution=HD", request, storage_session->dev_id, storage_session->event_id); //MYDEBUG("storage token:%s\n", storage_session->xtoken); //MYDEBUG("storage cookie:%s\n", storage_session->cookie); buf = tpbuf_snprintf(1024, "POST %s HTTP/1.1\r\n" "User-Agent: %s/%s %s\r\n" // MODEL/FW_VERSION PROGRAM_NAME/PROGRAM_VERSION "Host: %s\r\n" "Transfer-Encoding: chunked\r\n" //http chunked transfer encoding "Content-Type: multipart/x-mixed-replace;boundary=%s\r\n" "X-token: %s\r\n" // X-token "X-Audio-Codec: %s\r\n" "Cookie: %s\r\n" "\r\n", url, dev_info->dev_model?dev_info->dev_model:"*", dev_info->sw_ver?dev_info->sw_ver:"*", THIS_PROCESS, storage_session->host, "data-boundary--", storage_session->xtoken, dev_info->audio_fmt, storage_session->cookie); list_add_tail(&buf->list, &head); /* header */ list_splice_tail_init(&storage_session->headers, &head); /* ready to write */ debug_show_tpbuf_list(&head, "STORAGE STREAM POST:", 3); if (false == storage_session_write_list_force(storage_session, &head, false)) { DBG_ERR("storage_session_write_list_force error\n"); tpsocket_free_buf(&head, NULL, 0); return false; } return true; } bool storage_snapshot_consumer_post(STORAGE_SESSION *storage_session) { struct list_head head; struct tpsocket_buf *buf; DEV_INFO *dev_info; char *request = NULL; if (!storage_session) { DBG_ERR("arg NULL\n"); return false; } INIT_LIST_HEAD(&head); dev_info = get_dev_info(storage_session->storage_server->worker_ctx->top_ctx, storage_session->dev_id, NULL); if (!dev_info) { DBG_ERR("get dev info failed\n"); return false; } /* add POST header */ char url[256] = {0}; int sub_header_len; char sub_header[512] = {0}; request = strstr(dev_info->storage_url, "/v1"); snprintf(url, 256, "%sdevice/%s/activity/%s/snapshot", request, storage_session->dev_id, storage_session->event_id); sub_header_len = snprintf(sub_header, 512, "--%s\r\n" "Content-Disposition: form-data;name=\"snapshot\";filename=\"example.jpg\"\r\n" "Content-Type: image/jpeg\r\n\r\n", SNAPSHOT_FILE_BOUNDARY); int content_len = sub_header_len+storage_session->content_len+strlen("\r\n")+storage_session->region_buf_len+ strlen("--")+strlen(SNAPSHOT_FILE_BOUNDARY)+strlen("--"); buf = tpbuf_snprintf(1024, "POST %s HTTP/1.1\r\n" "User-Agent: %s/%s %s\r\n" // MODEL/FW_VERSION PROGRAM_NAME/PROGRAM_VERSION "X-token: %s\r\n" // X-token "Host: %s\r\n" "Content-Length: %d\r\n" "Content-Type: multipart/form-data;boundary=%s\r\n" "\r\n" "%s",//sub_header url, dev_info->dev_model?dev_info->dev_model:"*", dev_info->sw_ver?dev_info->sw_ver:"*", THIS_PROCESS, storage_session->xtoken, storage_session->host, content_len, SNAPSHOT_FILE_BOUNDARY, sub_header); list_add_tail(&buf->list, &head); debug_show_tpbuf_list(&head, "STORAGE SNAPSHOT POST:", 3); if (false == storage_session_write_list_force(storage_session, &head, false)) { DBG_ERR("storage_session_write_list_force error\n"); tpsocket_free_buf(&head, NULL, 0); return false; } return true; } bool storage_file_produce(STORAGE_SESSION *storage_session, struct list_head *buf) { if (!storage_session) { DBG_ERR("arg NULL\n"); return false; } if (!storage_session->sock) { DBG_ERR("%p arg NULL, file = %p\n", storage_session, storage_session->file); return false; } if (tpsocket_writable(storage_session->sock)) { if (!list_empty(buf) && storage_session->chunked && !tpsocket_chunk_encode(buf)) { tpsocket_free_buf(buf, "PRODUCE", 0); return false; } storage_session_write_list_force(storage_session, buf, true); } return true; } bool storage_read_file(struct tpsocket_handler*handler, struct list_head*buf, int event) { STORAGE_SESSION *storage_session = handler->priv; //MYDEBUG("storage read file event = %s, sock = %p, consumer = %p\n", tpsocket_event_name(event), sock, storage_session); switch(event) { case TPSOCKET_EVENT_CONNECTED: break; case TPSOCKET_EVENT_REQ_HEAD: break; case TPSOCKET_EVENT_RSP_HEAD: break; case TPSOCKET_EVENT_UPGRADE: break; case TPSOCKET_EVENT_SUB_HEAD: case TPSOCKET_EVENT_STREAM: case TPSOCKET_EVENT_MESSAGE: if (false == storage_file_produce(storage_session, buf)) { return false; } goto out; //break; case TPSOCKET_EVENT_RESET: break; case TPSOCKET_EVENT_CLOSED: if (storage_session) { storage_session->file = NULL; if (storage_session->sock) { if (!list_empty(buf)) { if (storage_session->chunked) { tpsocket_chunk_encode(buf); tpsocket_write_list_force(storage_session->sock, buf, true); } } if (STORAGE_STREAMING == storage_session->storage_type) { storage_session_stream_add_tail(storage_session); } else { storage_session_snapshot_add_tail(storage_session); } } } break; case TPSOCKET_EVENT_ERROR: default: break; } tpsocket_free_buf(buf, tpsocket_event_name(event), 0); out: return true; } bool storage_start_read(STORAGE_SESSION *storage_session) { if (!storage_session) { DBG_ERR("arg NULL\n"); return false; } struct tpsocket_handler client = { .cb = storage_read_file, .flowcontrol = 10*1024, .priv = storage_session, .read_buf_max = STORAGE_MEMORY_CACHE_LIMIT, }; if (STORAGE_STREAMING == storage_session->storage_type) { storage_session->file = tpsocket_new(NULL, "stream", NULL, storage_session->stream_path, TPSOCKET_TYPE_FILE, &client); if (!storage_session->file) { DBG_ERR("open %s failed\n", storage_session->stream_path); return false; } } else { storage_session->file = tpsocket_new(NULL, "read", NULL, storage_session->snapshot_path, TPSOCKET_TYPE_FILE, &client); if (!storage_session->file) { DBG_ERR("open %s failed\n", storage_session->snapshot_path); return false; } } DBG_DBG("session = %p, sock = %p, open file = %p\n", storage_session, storage_session->sock, storage_session->file); return true; } bool tpsocket_event_storage_stream_to_cloud(struct tpsocket_handler *handler, struct list_head *buf, int event) { //debug_show_tpbuf_list(buf, "TEST", 3); struct tpsocket_fd *sock = container_of(handler, struct tpsocket_fd, handler); STORAGE_SESSION *storage_session = handler->priv; int err_code = 0; if (event != TPSOCKET_EVENT_WRITABLE) { MYDEBUG("storage stream to cloud event = %s, sock = %p, session = %p\n", tpsocket_event_name(event), sock, storage_session); if (storage_session) { DBG_DBG("Device Id = %s, session sock = %p\n", storage_session->dev_id, storage_session->sock); } } switch(event) { case TPSOCKET_EVENT_LISTEN: break; case TPSOCKET_EVENT_SHUTDOWN: break; case TPSOCKET_EVENT_ACCEPT: break; case TPSOCKET_EVENT_CONNECTED: MYDEBUG("STORAGESTREAM2CLOUD: %s:%s Connected\n", sock->addr, sock->port); if (false == storage_stream_consumer_post(storage_session)) return false; if (false == storage_start_read(storage_session)) { DBG_ERR("read stream file failed\n"); return false; } break; case TPSOCKET_EVENT_REQ_HEAD: break; case TPSOCKET_EVENT_RSP_HEAD: break; case TPSOCKET_EVENT_SUB_HEAD: break; case TPSOCKET_EVENT_UPGRADE: break; case TPSOCKET_EVENT_WRITABLE: if (storage_session->file && !list_empty(&(storage_session->file->parser.buf))) { if (storage_session->chunked && !tpsocket_chunk_encode(&(storage_session->file->parser.buf))) { tpsocket_free_buf(&(storage_session->file->parser.buf), tpsocket_event_name(event), 0); break; } storage_session_write_list_force(storage_session, &(storage_session->file->parser.buf), true); storage_session->file->read_buf_length = 0; } break; case TPSOCKET_EVENT_STREAM: return true; case TPSOCKET_EVENT_MESSAGE: err_code = common_check_error_code(buf); if (!err_code) { DBG_DBG("storage stream push success\n"); storage_session->post_success = 1; #ifndef TAPO_CAMERA } else if (err_code == -98400) { //If the recording/relay server finds out that the device is not eligible for cloud storage, // it returns an error code of -98400 upon which the firmware updates the cloud storage feature of that device. DEV_INFO *dev_info = get_dev_info(storage_session->storage_server->worker_ctx->top_ctx, storage_session->dev_id, NULL); if (dev_info) { dev_info->cloud_storage = false; } // mark as success here to avoid retrying. storage_session->post_success = 1; #endif } tpsocket_free_buf(buf, tpsocket_event_name(event), 0); common_tpsocket_close_immediately(sock); break; case TPSOCKET_EVENT_RESET: break; case TPSOCKET_EVENT_KEEPALIVE: common_tpsocket_close_immediately(sock); break; case TPSOCKET_EVENT_CLOSED: if (storage_session) { tpsocket_unbind(sock, &storage_session->sock); uloop_timeout_cancel(&storage_session->reupload_tmo); tpsocket_free2(&storage_session->file, storage_session); if (storage_session->post_success) { storage_session->stream_send_to_cloud_flag = 0; } storage_session->uploading_flag = 0; storage_session_set_push_status(storage_session, 0); /* handle timeout immediately */ storage_server_timeout_handle_immediately(storage_session->storage_server); } break; case TPSOCKET_EVENT_ERROR: default: break; } tpsocket_free_buf(buf, tpsocket_event_name(event), 0); return true; } bool tpsocket_event_storage_snapshot_to_cloud(struct tpsocket_handler *handler, struct list_head *buf, int event) { //debug_show_tpbuf_list(buf, "TEST", 3); struct tpsocket_fd *sock = container_of(handler, struct tpsocket_fd, handler); STORAGE_SESSION *storage_session = handler->priv; if (event != TPSOCKET_EVENT_WRITABLE) { MYDEBUG("storage snapshot to cloud event = %s, sock = %p, session = %p\n", tpsocket_event_name(event), sock, storage_session); if (storage_session) { DBG_DBG("Device Id = %s, session sock = %p\n", storage_session->dev_id, storage_session->sock); } } switch(event) { case TPSOCKET_EVENT_LISTEN: break; case TPSOCKET_EVENT_SHUTDOWN: break; case TPSOCKET_EVENT_ACCEPT: break; case TPSOCKET_EVENT_CONNECTED: MYDEBUG("STORAGESNAPSHOT2CLOUD: %s:%s Connected\n", sock->addr, sock->port); if (false == storage_snapshot_consumer_post(storage_session)) return false; if (false == storage_start_read(storage_session)) { DBG_ERR("read snapshot file failed\n"); return false; } break; case TPSOCKET_EVENT_REQ_HEAD: break; case TPSOCKET_EVENT_RSP_HEAD: break; case TPSOCKET_EVENT_SUB_HEAD: break; case TPSOCKET_EVENT_UPGRADE: break; case TPSOCKET_EVENT_WRITABLE: if (storage_session->file && !list_empty(&(storage_session->file->parser.buf))) { if (storage_session->chunked && !tpsocket_chunk_encode(&(storage_session->file->parser.buf))) { tpsocket_free_buf(&(storage_session->file->parser.buf), tpsocket_event_name(event), 0); break; } storage_session_write_list_force(storage_session, &(storage_session->file->parser.buf), true); storage_session->file->read_buf_length = 0; } break; case TPSOCKET_EVENT_STREAM: return true; case TPSOCKET_EVENT_MESSAGE: if (!common_check_error_code(buf)) { DBG_DBG("storage snapshot push success\n"); storage_session->post_success = 1; } tpsocket_free_buf(buf, tpsocket_event_name(event), 0); common_tpsocket_close_immediately(sock); break; case TPSOCKET_EVENT_RESET: break; case TPSOCKET_EVENT_KEEPALIVE: common_tpsocket_close_immediately(sock); break; case TPSOCKET_EVENT_CLOSED: if (storage_session) { tpsocket_unbind(sock, &storage_session->sock); uloop_timeout_cancel(&storage_session->reupload_tmo); tpsocket_free2(&storage_session->file, storage_session); if (storage_session->post_success) { storage_session->snapshot_send_to_cloud_flag = 0; //storage_session_free(storage_session); } else { storage_session->retry_count++; //list_add_tail(&storage_session->list, &storage_session->storage_server->snapshot_session); } storage_session->uploading_flag = 0; storage_session_set_push_status(storage_session, 0); /* handle timeout immediately */ storage_server_timeout_handle_immediately(storage_session->storage_server); } break; case TPSOCKET_EVENT_ERROR: default: break; } tpsocket_free_buf(buf, tpsocket_event_name(event), 0); return true; } bool storage_stream_upload(STORAGE_SESSION *storage_session) { DEV_INFO *dev_info = NULL; STORAGE_SERVER *storage_server; if (!storage_session) { DBG_ERR("arg NULL\n"); return false; } if (storage_session->sock) { DBG_DBG("sock exist\n"); return true; } storage_server = storage_session->storage_server; uloop_timeout_set(&storage_session->reupload_tmo, storage_session->reupload_idle_time); dev_info = get_dev_info(storage_server->worker_ctx->top_ctx, storage_session->dev_id, NULL); if (!dev_info) { DBG_ERR("Not find device\n"); return false; } /* create socket */ struct tpsocket_handler storage_stream_to_cloud = { .cb = tpsocket_event_storage_stream_to_cloud, .cer = MEDIACENTER_CERTFILE, .write_buf_max = STORAGE_MEMORY_CACHE_LIMIT, }; storage_session->retry_count++; if ((storage_session->sock = tpsocket_from_url(dev_info->relay_storage_url, &storage_stream_to_cloud))) { //if (storage_session->sock = tpsocket_from_url("http://192.168.137.103:8010", &storage_stream_to_cloud)) { DBG_DBG("Create session sock = %p, session = %p\n", storage_session->sock, storage_session); storage_session->sock->handler.priv = storage_session; storage_session->uploading_flag = 1; storage_session_set_push_status(storage_session, 1); } return true; } void storage_set_iot_info(void *arg, const char *host, const char *cookie, const char *token, const char *event_id) { STORAGE_SESSION *storage_session; #ifdef TAPO_CAMERA if (!arg || !host || !token || !event_id) { #else if (!arg || !host || !cookie || !token || !event_id) { #endif DBG_ERR("arg NULL\n"); return; } storage_session = (STORAGE_SESSION*)arg; snprintf(storage_session->host, LEN_MAX_HOST, "%s", host); snprintf(storage_session->cookie, LEN_MAX_COOKIE, "%s", cookie); snprintf(storage_session->xtoken, LEN_MAX_TOKEN, "%s", token); snprintf(storage_session->event_id, LEN_MAX_EVENT_ID, "%s", event_id); MYDEBUG("host: %s\n", storage_session->host); //MYDEBUG("cookie: %s\n", storage_session->cookie); //MYDEBUG("xtoken: %s\n", storage_session->xtoken); MYDEBUG("event_id: %s\n", storage_session->event_id); return; } void storage_get_iot_info_cb(void *priv, void *iot_info, char *host, char *cookie, char *token, char *event_id, int err_code) { STORAGE_SESSION *storage_session; if (!priv) { DBG_ERR("arg NULL\n"); return; } storage_session = (STORAGE_SESSION*)priv; if (err_code) { DBG_ERR("storage get iot info failed\n"); goto out; } if (!host || !cookie || !token || !event_id) { DBG_ERR("storage get iot info param error\n"); goto out; } storage_set_iot_info(storage_session, host, cookie, token, event_id); MYDEBUG("Have got iot info, reupload now\n"); if (STORAGE_STREAMING == storage_session->storage_type) { storage_stream_upload(storage_session); return; } out: storage_session->retry_count++; return; } int storage_stream_reupload(STORAGE_SESSION *storage_session) { IOT_INFO *iot_info; if (!storage_session) { DBG_ERR("arg NULL\n"); return -1; } if (storage_session->event_id[0]) { MYDEBUG("Have got iot info, reupload now\n"); if (false == storage_stream_upload(storage_session)) { storage_session->stream_send_to_cloud_flag = 0; } } else { //find event id iot_info = iot_info_find(storage_session->storage_server->worker_ctx->iot_info_server, storage_session->dev_id , storage_session->count_id); if (iot_info) { if (iot_info->ready) { #ifdef TAPO_CAMERA storage_get_iot_info_cb((void*)storage_session, NULL, iot_info->host, iot_info->cookie, iot_info->xtoken, iot_info->alarm_id, 0); #else storage_get_iot_info_cb((void*)storage_session, NULL, iot_info->host, iot_info->cookie, iot_info->token, iot_info->event_id, 0); #endif if (IOT_INFO_CONSUMER_NUM == iot_info->consumer_used_count) { iot_info_free(iot_info); } } else { storage_session->retry_count++; } } else { storage_session->stream_send_to_cloud_flag = 0; } } return 0; } void storage_force_delete_session(STORAGE_SERVER *storage_server) { struct statfs diskInfo; STORAGE_SESSION *stream_session, *nstream_session; int force = 0; if (!storage_server) { DBG_ERR("arg NULL\n"); return; } retry: if (force++ < 2 && !statfs(STORAGE_ROOT_PATH, &diskInfo)) { int free_size = (diskInfo.f_bsize * diskInfo.f_bfree) >> 20; if (free_size < 20) { MYDEBUG("FREE MEMORY %u MB!\n", free_size); if (free_size < 8) { list_for_each_entry_safe(stream_session, nstream_session, &storage_server->stream_session, list) { if (!stream_session->iot_consumer) { if ((force == 1 && stream_session->delete_flag) || (force == 2 && !stream_session->uploading_flag)) { storage_session_free(stream_session); } } } goto retry; } } } } void storage_check_list_to_upload(STORAGE_SERVER *storage_server) { STORAGE_SESSION *stream_session, *nstream_session; int ret = 0; if (!storage_server) { DBG_ERR("arg NULL\n"); return; } list_for_each_entry_safe(stream_session, nstream_session, &storage_server->stream_session, list) { if (!stream_session->iot_consumer) {//make sure alarm push finished //MYDEBUG("find storage stream session = %p\n", stream_session); if (stream_session->uploading_flag) { MYDEBUG("stream session %p uploading push busy ...\n", stream_session); break; } ret = storage_session_get_push_status(stream_session); if (1 == ret) { MYDEBUG("this stream session %p is ready, but other session of the same device push busy ...\n", stream_session); break; } else if (-1 == ret) { MYDEBUG("device not found\n"); break; } if (stream_session->stream_send_to_cloud_flag && (stream_session->retry_count < 3)) { MYDEBUG("stream session %p do upload!\n", stream_session); storage_stream_reupload(stream_session); } else { //DBG_DBG("stream session %p can be deleted\n", stream_session); stream_session->delete_flag = true; } } else { MYDEBUG("waitting iot session %p closed\n", stream_session->iot_consumer); } } } void storage_server_timeout_handle_immediately(STORAGE_SERVER *storage_server) { MYDEBUG("storage_server_timeout_handle_immediately ...\n"); if (!storage_server) { DBG_ERR("arg NULL\n"); return; } uloop_timeout_set(&storage_server->tmo, 100); } void storage_server_timeout_cb(struct uloop_timeout *tmo) { //MYDEBUG("storage server timeout cb ...\n"); STORAGE_SERVER *storage_server = container_of(tmo, STORAGE_SERVER, tmo); if (!storage_server) { DBG_ERR("storage_server NULL\n"); return; } storage_force_delete_session(storage_server); storage_check_list_to_upload(storage_server); uloop_timeout_set(tmo, 10*1000); } void storage_session_set_send_flag(void *arg, int flag) { STORAGE_SESSION *storage_session = NULL; if (!arg) { DBG_ERR("arg NULL\n"); return; } storage_session = (STORAGE_SESSION*)arg; if (STORAGE_STREAMING == storage_session->storage_type) { storage_session->stream_send_to_cloud_flag = flag; } else { storage_session->snapshot_send_to_cloud_flag = flag; } return; } void storage_session_set_iot_consumer_null(void *arg) { STORAGE_SESSION *storage_session; if (!arg) { DBG_ERR("arg NULL\n"); return; } storage_session = (STORAGE_SESSION*)arg; storage_session->iot_consumer = NULL; storage_update_list(storage_session->storage_server, storage_session->storage_type); return; } int storage_delete_empty_dir(char *filedir) { int count = 0; DIR *dir = NULL; struct dirent *Dirent = NULL; //char cmd[128] = {0}; if(!filedir){ DBG_ERR("arg NULL\n"); return -1; } dir = opendir(filedir); if(dir == NULL){ DBG_NOTICE("opendir:%s error:%s\n", filedir, strerror(errno)); return -1; } while((Dirent = readdir(dir)) != NULL){ if(0 != strncmp(Dirent->d_name, ".", sizeof(".")) && 0 != strncmp(Dirent->d_name, "..", sizeof(".."))) { count++; } } closedir(dir); if(!count){ //memset(cmd, '\0', 128); //snprintf(cmd, 128, "rm -rf %s", filedir); //if (system(cmd) < 0) { if (tpcom_system("rm", ("-rf"), ("%s", filedir)) < 0) { DBG_ERR("remove:%s error:%s\n", filedir, strerror(errno)); return -1; } DBG_DBG("remove dir:%s ok\n", filedir); } else { DBG_DBG("has %d file in %s\n", count, filedir); } return 0; } void storage_delete_file(char *file) { char *pos; char dev_id[LEN_MAX_ID] = {0}; char count_id[LEN_MAX_ID] = {0}; char dir[LEN_MAX_PATH] = {0}; if (!file) { DBG_ERR("arg NULL\n"); return; } if (!file[0]) { DBG_DBG("No file to delete\n"); return; } pos = file+strlen(STORAGE_ROOT_PATH); DBG_DBG("pos = %s\n", pos); sscanf(pos, "/%[^/]/%[^/]/%*s", dev_id, count_id); DBG_DBG("dev_id = %s, count_id = %s\n", dev_id, count_id); /* delete file */ remove(file); /* delete count id dir */ memset(dir, 0, LEN_MAX_PATH); snprintf(dir, LEN_MAX_PATH, "%s/%s/%s", STORAGE_ROOT_PATH, dev_id, count_id); storage_delete_empty_dir(dir); /* delete device dir */ memset(dir, 0, LEN_MAX_PATH); snprintf(dir, LEN_MAX_PATH, "%s/%s", STORAGE_ROOT_PATH, dev_id); storage_delete_empty_dir(dir); } void storage_move_file(char *file) { char *pos; char cmd[512] = {0}; char dev_id[LEN_MAX_ID] = {0}; char count_id[LEN_MAX_ID] = {0}; char dir[LEN_MAX_PATH] = {0}; char count_path[LEN_MAX_PATH] = {0}; if (!file || !file[0]) { DBG_ERR("arg NULL\n"); return; } pos = file+strlen(STORAGE_ROOT_PATH); MYDEBUG("pos = %s\n", pos); sscanf(pos, "/%[^/]/%[^/]/%*s", dev_id, count_id); MYDEBUG("dev_id = %s, count_id = %s\n", dev_id, count_id); /* remove file */ snprintf(count_path, LEN_MAX_PATH, "%s/%s/%s/", NFS_ROOT, dev_id, count_id); if (access(count_path, F_OK)) { //snprintf(cmd, 512, "mkdir -p %s", count_path); //system(cmd); tpcom_system("mkdir", ("-p"), ("%s", count_path)); } memset(cmd, 0, 512); snprintf(cmd, 512, "mv %s %s", file, count_path); DEBUG("cmd: %s\n", cmd); //system(cmd); tpcom_system("mv", (file), ("%s", count_path)); if (!access(file, F_OK)) { DBG_ERR("%s failed\n", cmd); remove(file); } /* delete count id dir */ memset(dir, 0, LEN_MAX_PATH); snprintf(dir, LEN_MAX_PATH, "%s/%s/%s", STORAGE_ROOT_PATH, dev_id, count_id); storage_delete_empty_dir(dir); /* delete device dir */ memset(dir, 0, LEN_MAX_PATH); snprintf(dir, LEN_MAX_PATH, "%s/%s", STORAGE_ROOT_PATH, dev_id); storage_delete_empty_dir(dir); } void storage_session_free(STORAGE_SESSION *storage_session) { MYDEBUG("STORAGE SESSION FREE %p\n", storage_session); if (!storage_session) { return; } list_del(&storage_session->list); uloop_timeout_cancel(&storage_session->reupload_tmo); tpsocket_free_buf(&storage_session->headers, NULL, 0); if (storage_session->sock) { storage_session_set_push_status(storage_session, 0); } tpsocket_free2(&storage_session->sock, storage_session); tpsocket_free2(&storage_session->file, storage_session); if (storage_session->iot_consumer) { alarm_consumer_set_storage_session_null(storage_session->iot_consumer); } /* telemetry */ storage_session_telemetry_failure_count(storage_session); /* delete file */ if (STORAGE_STREAMING == storage_session->storage_type) { if (STORAGE_DEBUG && !access("/nfsroot", F_OK)) { storage_move_file(storage_session->stream_path); } else { storage_delete_file(storage_session->stream_path); } } else { if (STORAGE_DEBUG && !access("/nfsroot", F_OK)) { storage_move_file(storage_session->snapshot_path); } else { storage_delete_file(storage_session->snapshot_path); } } free(storage_session); } bool tpsocket_event_consumer_to_storage(struct tpsocket_handler *handler, struct list_head *buf, int event) { struct tpsocket_fd *sock = container_of(handler, struct tpsocket_fd, handler); STORAGE_SESSION *storage_session = (STORAGE_SESSION*)handler->priv; if (STORAGE_SNAPSHOT == storage_session->storage_type) { MYDEBUG("storage snpashot cb: sock=%p, event=%d\n", sock, event); } switch(event) { case TPSOCKET_EVENT_CONNECTED: break; case TPSOCKET_EVENT_REQ_HEAD: break; case TPSOCKET_EVENT_RSP_HEAD: break; case TPSOCKET_EVENT_SUB_HEAD: break; case TPSOCKET_EVENT_UPGRADE: break; case TPSOCKET_EVENT_MESSAGE: break; case TPSOCKET_EVENT_STREAM: break; case TPSOCKET_EVENT_RESET: break; case TPSOCKET_EVENT_CLOSED: if (storage_session) { storage_session->file = NULL; list_del_init(&storage_session->list); storage_session_add_session(storage_session); } break; case TPSOCKET_EVENT_ERROR: MYDEBUG("tpsocket_event_consumer_to_storage ERROR %p\n", storage_session); storage_session->storage_success = false; default: break; } tpsocket_free_buf(buf, tpsocket_event_name(event), 0); return true; } STORAGE_SESSION *storage_session_new(char *path, STORAGE_SERVER *storage_server) { STORAGE_SESSION *storage_session; if (!path || ! storage_server) { DBG_ERR("arg NULL\n"); return NULL; } storage_session = (STORAGE_SESSION*)malloc(sizeof(*storage_session)); if (!storage_session) { DBG_ERR("malloc failed\n"); return NULL; } memset(storage_session, 0, sizeof(*storage_session)); INIT_LIST_HEAD(&storage_session->list); INIT_LIST_HEAD(&storage_session->headers); storage_session->storage_server = storage_server; storage_session->reupload_idle_time = 10000; storage_session->reupload_tmo.cb = storage_session_reupload_timeout; storage_session->storage_success = true; storage_session->delete_flag = false; /* create socket */ struct tpsocket_handler client = { .cb = tpsocket_event_consumer_to_storage, .flowcontrol = 500000, .priv = storage_session }; storage_session->file = tpsocket_new(NULL, "write", NULL, path, TPSOCKET_TYPE_FILE, &client); if (!(storage_session->file)) { DBG_ERR("tpsocket_new failed\n"); free(storage_session); return NULL; } //consumer->chunked = 0; //consumer->store = 1; return storage_session; } int storage_find_file_in_list(char *dev_id, char *timestamp_str, struct list_head *head) { STORAGE_SESSION *session; int timestamp; if (!dev_id || !timestamp_str || !head) { DBG_ERR("arg NULL\n"); return -1; } timestamp = atoi(timestamp_str); list_for_each_entry(session, head, list) { if (!strncmp(session->dev_id, dev_id, LEN_MAX_ID) && (session->event_timestamp == timestamp)) { return 0; } } return -1; } void storage_check_file(STORAGE_SERVER *storage_server) { if (!storage_server) { DBG_ERR("arg NULL\n"); return; } DIR *dir = NULL, *dev_dir = NULL; struct dirent *root_dirent = NULL, *dev_dirent = NULL; char dev_id[LEN_MAX_ID] = {0}, timestamp[LEN_MAX_ID] = {0}; char dev_path[LEN_MAX_PATH] = {0}, event_path[LEN_MAX_PATH] = {0}, file_path[LEN_MAX_PATH] = {0}; dir = opendir(STORAGE_ROOT_PATH); if(dir == NULL){ DBG_NOTICE("opendir:%s error:%s\n", STORAGE_ROOT_PATH, strerror(errno)); return; } while((root_dirent = readdir(dir)) != NULL){ /* dev */ if(0 == strncmp(root_dirent->d_name, ".", sizeof(".")) || 0 == strncmp(root_dirent->d_name, "..", sizeof(".."))) { continue; } memset(dev_path, 0, LEN_MAX_PATH); snprintf(dev_path, LEN_MAX_PATH, "%s/%s", STORAGE_ROOT_PATH, root_dirent->d_name); dev_dir = opendir(dev_path); if(dev_dir == NULL){ DBG_NOTICE("opendir:%s error:%s\n", dev_path, strerror(errno)); continue; } memset(dev_id, 0, LEN_MAX_ID); snprintf(dev_id, LEN_MAX_ID, "%s", root_dirent->d_name); while ((dev_dirent = readdir(dev_dir)) != NULL) { /* event */ if(0 == strncmp(dev_dirent->d_name, ".", sizeof(".")) || 0 == strncmp(dev_dirent->d_name, "..", sizeof(".."))) { continue; } memset(timestamp, 0, LEN_MAX_ID); snprintf(timestamp, LEN_MAX_ID, "%s", dev_dirent->d_name); memset(event_path, 0, LEN_MAX_PATH); snprintf(event_path, LEN_MAX_PATH, "%s/%s/%s", STORAGE_ROOT_PATH, dev_id, timestamp); memset(file_path, 0, LEN_MAX_PATH); snprintf(file_path, LEN_MAX_PATH, "%s/%s/%s/%s", STORAGE_ROOT_PATH, dev_id, timestamp, STORAGE_STREAMING_FILE); if (!access(file_path, F_OK)) { DBG_NOTICE("streaming file: %s\n", file_path); if (storage_find_file_in_list(dev_id, timestamp, &storage_server->stream_session)) { storage_delete_file(file_path); } } memset(file_path, 0, LEN_MAX_PATH); snprintf(file_path, LEN_MAX_PATH, "%s/%s/%s/%s", STORAGE_ROOT_PATH, dev_id, timestamp, STORAGE_SNAPSHOT_FILE); if (!access(file_path, F_OK)) { DBG_NOTICE("snapshot file: %s\n", file_path); if (storage_find_file_in_list(dev_id, timestamp, &storage_server->snapshot_session)) { storage_delete_file(file_path); } } storage_delete_empty_dir(event_path); } closedir(dev_dir); } closedir(dir); return; } void storage_get_iot_info(STORAGE_SESSION *session, struct list_head *iot_info_head) { IOT_INFO *iot_info; if (!session || !iot_info_head) { DBG_ERR("arg NULL\n"); return; } list_for_each_entry(iot_info, iot_info_head, list) { if (!strncmp(session->dev_id, iot_info->dev_id, LEN_MAX_ID) && (session->count_id == iot_info->count_id)) { #ifdef TAPO_CAMERA storage_set_iot_info(session, iot_info->host, iot_info->cookie, iot_info->xtoken, iot_info->alarm_id); #else storage_set_iot_info(session, iot_info->host, iot_info->cookie, iot_info->token, iot_info->event_id); #endif break; } } } bool storage_check_version() { FILE *version_fd; char version_path[LEN_MAX_PATH] = {0}, version[4] = {0}; //char cmd[128] = {0}; snprintf(version_path, LEN_MAX_PATH, "%s/%s", STORAGE_ROOT_PATH, STORAGE_VERSION_FILE); if (access(version_path, F_OK)) { DBG_NOTICE("storage version file not exist\n"); goto failed; } if (!(version_fd = fopen(version_path, "r"))) { DBG_NOTICE("open storage version file failed\n"); goto failed; } if (1 != fread(version, 4, 1, version_fd)) { DBG_NOTICE("read storage version file error\n"); fclose(version_fd); goto failed; } fclose(version_fd); if (strncmp(version, STORAGE_VERSION, 4)) { DBG_NOTICE("old version\n"); goto failed; } return true; failed: /* delete storage */ //snprintf(cmd, 128, "rm -rf %s/*", STORAGE_ROOT_PATH); //system(cmd); tpcom_system("rm", ("-rf"), ("%s", STORAGE_ROOT_PATH)); tpcom_system("mkdir", ("-p"), ("%s", STORAGE_ROOT_PATH)); /* create version */ snprintf(version, 4, "%s", STORAGE_VERSION); version_fd = fopen(version_path, "w"); if (version_fd) { fwrite(version, 4, 1, version_fd); fclose(version_fd); } return false; } void storage_file_list_init(STORAGE_SERVER *storage_server) { FILE *iot_info_fd, *stream_fd, *snapshot_fd; char iot_info_path[LEN_MAX_PATH] = {0}, stream_path[LEN_MAX_PATH] = {0}, snapshot_path[LEN_MAX_PATH] = {0}; struct list_head list; IOT_INFO *iot_info, *niot_info; if (!storage_server) { DBG_ERR("arg NULL\n"); return; } /* check version */ if (false == storage_check_version()) { return; } /* iot info */ INIT_LIST_HEAD(&list); snprintf(iot_info_path, LEN_MAX_PATH, "%s/%s", STORAGE_ROOT_PATH, STORAGE_IOT_INFO_LIST); iot_info_fd = fopen(iot_info_path, "r"); if (iot_info_fd) { while (1) { iot_info = malloc(sizeof(IOT_INFO)); if (fread(iot_info, sizeof(IOT_INFO), 1, iot_info_fd) <= 0) { free(iot_info); break; } list_add_tail(&iot_info->list, &list); } fclose(iot_info_fd); } /* stream session */ snprintf(stream_path, LEN_MAX_PATH, "%s/%s", STORAGE_ROOT_PATH, STORAGE_STREAMING_LIST); stream_fd = fopen(stream_path, "r"); if (stream_fd) { while (1) { STORAGE_SESSION *stream_session = malloc(sizeof(STORAGE_SESSION)); if (fread(stream_session, sizeof(STORAGE_SESSION), 1, stream_fd) <= 0) { free(stream_session); break; } DBG_NOTICE("dev_id: %s, count_id: %d\n", stream_session->dev_id, stream_session->count_id); INIT_LIST_HEAD(&stream_session->list); INIT_LIST_HEAD(&stream_session->headers); stream_session->sock = NULL; stream_session->file = NULL; stream_session->uploading_flag = 0; stream_session->producer = NULL; stream_session->iot_consumer = NULL; stream_session->storage_server = storage_server; storage_session_add_fixed_header(stream_session); memset(&stream_session->reupload_tmo, 0, sizeof(struct uloop_timeout)); list_add_tail(&stream_session->list, &storage_server->stream_session); storage_get_iot_info(stream_session, &list); } fclose(stream_fd); } /* snapshot session */ snprintf(snapshot_path, LEN_MAX_PATH, "%s/%s", STORAGE_ROOT_PATH, STORAGE_SNAPSHOT_LIST); snapshot_fd = fopen(snapshot_path, "r"); if (snapshot_fd) { while (1) { STORAGE_SESSION *snapshot_session = malloc(sizeof(STORAGE_SESSION)); if (fread(snapshot_session, sizeof(STORAGE_SESSION), 1, snapshot_fd) <= 0) { free(snapshot_session); break; } DBG_NOTICE("dev_id: %s, count_id: %d\n", snapshot_session->dev_id, snapshot_session->count_id); INIT_LIST_HEAD(&snapshot_session->list); INIT_LIST_HEAD(&snapshot_session->headers); snapshot_session->sock = NULL; snapshot_session->file = NULL; snapshot_session->uploading_flag = 0; snapshot_session->producer = NULL; snapshot_session->iot_consumer = NULL; snapshot_session->storage_server = storage_server; memset(&snapshot_session->reupload_tmo, 0, sizeof(struct uloop_timeout)); list_add_tail(&snapshot_session->list, &storage_server->snapshot_session); storage_get_iot_info(snapshot_session, &list); } fclose(snapshot_fd); } /* free & delete */ list_for_each_entry_safe(iot_info, niot_info, &list, list) { free(iot_info); } /* check file */ storage_check_file(storage_server); return; } void storage_server_free(STORAGE_SERVER *storage_server) { STORAGE_SESSION *storage_session, *nstorage_session; if (!storage_server) { return; } uloop_timeout_cancel(&storage_server->tmo); list_for_each_entry_safe(storage_session, nstorage_session, &storage_server->stream_session, list) { storage_session_free(storage_session); } list_for_each_entry_safe(storage_session, nstorage_session, &storage_server->snapshot_session, list) { storage_session_free(storage_session); } if (storage_server->worker_ctx) { storage_server->worker_ctx->storage_server = NULL; } list_del_init(&storage_server->list); free(storage_server); } int storage_server_init(WORKER_CTX *worker_ctx) { STORAGE_SERVER *storage_server; if (!worker_ctx) { DBG_ERR("worker ctx NULL\n"); return -1; } /* create */ storage_server = (STORAGE_SERVER*)malloc(sizeof(*storage_server)); if (!storage_server) { DBG_ERR("malloc failed\n"); return -1; } memset(storage_server, 0, sizeof(*storage_server)); storage_server->worker_ctx = worker_ctx; INIT_LIST_HEAD(&storage_server->list); INIT_LIST_HEAD(&storage_server->stream_session); INIT_LIST_HEAD(&storage_server->snapshot_session); //timer to re-upload storage_server->tmo.cb = storage_server_timeout_cb; uloop_timeout_set(&storage_server->tmo, 10*1000); /* set */ worker_ctx->storage_server = storage_server; /* file list init */ storage_file_list_init(storage_server); return 0; }
09-23
/****************************************************************************** * * Copyright (c) 2015 TP-Link Systems Inc. * All rights reserved. * * FILE NAME : storage.c * VERSION : 1.x * DESCRIPTION: * * AUTHOR : WangLin * * CREATE DATE: 2018/05/25 * * HISTORY : * 01 2018/05/25 WangLin create * ******************************************************************************/ #include <dirent.h> #include <sys/statfs.h> #include "common.h" #include "storage.h" #include "iot_info.h" #include "device.h" #ifdef HEALTH_INFO_SUPPORT #include "health_info.h" #endif void storage_session_free(STORAGE_SESSION *storage_session); void storage_server_timeout_handle_immediately(STORAGE_SERVER *storage_server); /*---------------------------------------------------------*/ void storage_delete_posted_session(STORAGE_SERVER *storage_server, char *dev_id, int storage_type) { STORAGE_SESSION *session, *nsession; if (!storage_server || !dev_id) { DBG_ERR("arg NULL\n"); return; } if (STORAGE_STREAMING == storage_type) { list_for_each_entry_safe(session, nsession, &storage_server->stream_session, list) { if (!strncmp(session->dev_id, dev_id, LEN_MAX_ID) && !session->uploading_flag && (!session->stream_send_to_cloud_flag || session->delete_flag)) { storage_session_free(session); } } } else if (STORAGE_SNAPSHOT == storage_type) { list_for_each_entry_safe(session, nsession, &storage_server->snapshot_session, list) { DBG_DBG("%s %s %d %d %d\n", dev_id,session->dev_id, session->uploading_flag, session->snapshot_send_to_cloud_flag, session->delete_flag); if (!strncmp(session->dev_id, dev_id, LEN_MAX_ID) && !session->uploading_flag && (!session->snapshot_send_to_cloud_flag || session->delete_flag)) { storage_session_free(session); } } } } static void storage_session_telemetry_failure_count(STORAGE_SESSION *storage_session) { DEV_INFO *dev_info; MEDIACENTER_CTX *mediacenter_ctx; if (!storage_session) { DBG_ERR("arg NULL\n"); return; } mediacenter_ctx = storage_session->storage_server->worker_ctx->top_ctx; dev_info = get_dev_info(mediacenter_ctx, storage_session->dev_id, storage_session->dev_ip); if (dev_info && storage_session->post_success && (dev_info->telemetry.relay_clips_failure_count > 0)) { dev_info->telemetry.relay_clips_failure_count--; #ifdef HEALTH_INFO_SUPPORT dev_update_health_info(dev_info->dev_id); #endif } return; } void storage_delete_sessions_of_device(MEDIACENTER_CTX *mediacenter_ctx, char *dev_id) { WORKER_CTX *worker_ctx = NULL; STORAGE_SERVER *storage_server = NULL; STORAGE_SESSION *session = NULL, *nsession = NULL; if (!mediacenter_ctx || !dev_id) { DBG_ERR("arg NULL\n"); return; } if (list_empty(&mediacenter_ctx->worker_list)) { DBG_ERR("worker list empty\n"); return; } worker_ctx = (WORKER_CTX *)list_first_entry(&mediacenter_ctx->worker_list, STORAGE_SESSION, list); if (worker_ctx) { storage_server = worker_ctx->storage_server; if (storage_server) { list_for_each_entry_safe(session, nsession, &storage_server->stream_session, list) { if (!strncmp(session->dev_id, dev_id, LEN_MAX_ID)) { storage_session_free(session); } } list_for_each_entry_safe(session, nsession, &storage_server->snapshot_session, list) { if (!strncmp(session->dev_id, dev_id, LEN_MAX_ID)) { storage_session_free(session); } } } } } int storage_session_get_push_status(STORAGE_SESSION *session) { DEV_INFO *dev_info; MEDIACENTER_CTX *mediacenter_ctx; if (!session) { DBG_ERR("arg NULL\n"); return -1; } mediacenter_ctx = session->storage_server->worker_ctx->top_ctx; dev_info = get_dev_info(mediacenter_ctx, session->dev_id, session->dev_ip); if (dev_info) { if (STORAGE_STREAMING == session->storage_type) { return dev_info->alarm_stream_push_status; } else { return dev_info->alarm_snapshot_push_status; } } return -1; } void storage_session_set_push_status(STORAGE_SESSION *session, int status) { DEV_INFO *dev_info; MEDIACENTER_CTX *mediacenter_ctx; if (!session) { DBG_ERR("arg NULL\n"); return; } mediacenter_ctx = session->storage_server->worker_ctx->top_ctx; dev_info = get_dev_info(mediacenter_ctx, session->dev_id, session->dev_ip); if (dev_info) { DBG_INFO("It is alarm re-upload below\n"); if (STORAGE_STREAMING == session->storage_type) { dev_set_alarm_stream_status(dev_info, status); } else { dev_set_alarm_snapshot_status(dev_info, status); } } return; } void storage_session_reupload_timeout(struct uloop_timeout* tmo) { //MYDEBUG("STORAGE SESSION REUPLOAD CHECK TIMEOUT\n"); unsigned long long sent; STORAGE_SESSION *storage_session = container_of(tmo, STORAGE_SESSION, reupload_tmo); if (!storage_session) { DBG_ERR("arg NULL\n"); return; } uloop_timeout_set(&storage_session->reupload_tmo, storage_session->reupload_idle_time); if (storage_session->sock) { sent = storage_session->reupload_total_load - storage_session->sock->write_buf_length; if (sent <= storage_session->reupload_last_sent) { DBG_ERR("STORAGE SESSION REUPLOAD TIMEOUT\n"); tpsocket_free(storage_session->sock); } else { storage_session->reupload_last_sent = sent; storage_session->reupload_last_load = storage_session->reupload_total_load; } } } bool storage_session_write_list_force(STORAGE_SESSION *storage_session, struct list_head *head, bool force) { struct tpsocket_buf *buf; unsigned long long to_send = 0; if (!storage_session || !head) { DBG_ERR("arg NULL\n"); return false; } /* data length */ list_for_each_entry(buf, head, list) { to_send += tpbuf_data_len(buf); } /* move bufs to sock */ if (false == tpsocket_write_list_force(storage_session->sock, head, force)) { DBG_ERR("tpsocket_write_list error\n"); return false; } /* calculate total load */ storage_session->reupload_total_load += to_send; return true; } void storage_session_stream_add_tail(STORAGE_SESSION *storage_session) { struct list_head head; if (!storage_session) { DBG_ERR("arg NULL\n"); return; } if (storage_session->chunked) { INIT_LIST_HEAD(&head); tpsocket_chunk_encode(&head);//generate chunk end '0\r\n' tpsocket_write_list_force(storage_session->sock, &head, true); } tpsocket_write_list_force(storage_session->sock, &head, true); } void storage_session_snapshot_add_tail(STORAGE_SESSION *storage_session) { struct tpsocket_buf *mybuf = NULL; struct list_head head; if (!storage_session) { DBG_ERR("arg NULL\n"); return; } INIT_LIST_HEAD(&head); mybuf = tpbuf_snprintf(LEN_MAX_128, "\r\n"); if (mybuf) { list_add_tail(&mybuf->list, &head); } if (storage_session->region_buf_len) { mybuf = tpbuf_snprintf(LEN_MAX_128*4, "%s", storage_session->region_buf); if (mybuf) { list_add_tail(&mybuf->list, &head); } } mybuf = tpbuf_snprintf(LEN_MAX_128, "--%s--", SNAPSHOT_FILE_BOUNDARY);//generate snapshot multi-part end if (mybuf) { list_add_tail(&mybuf->list, &head); } if (storage_session->chunked) { tpsocket_chunk_encode(&head); } tpsocket_write_list_force(storage_session->sock, &head, true); } int storage_session_set_region(STORAGE_SESSION *storage_session, TRIGGER_REGION *region) { if (!storage_session || !region) { DBG_ERR("arg NULL\n"); return -1; } if (0 == region->x1 && 0 == region->y1 && 0 == region->x2 && 0 == region->y2) { } else { storage_session->region_buf_len = snprintf(storage_session->region_buf, LEN_MAX_128*4, "--%s\r\n" "Content-Disposition: form-data;name=\"region\"\r\n\r\n" "{\"top\": %d, \"bottom\": %d, \"left\": %d, \"right\": %d}\r\n", SNAPSHOT_FILE_BOUNDARY, region->y1, region->y2, region->x1, region->x2); MYDEBUG("%s", storage_session->region_buf); } return 0; } int storage_session_set_event_timestamp(STORAGE_SESSION *storage_session, time_t timestamp) { if (!storage_session) { DBG_ERR("arg NULL\n"); return -1; } //storage_session->event_timestamp = timestamp - common_get_time_diff(); storage_session->event_timestamp = timestamp; return 0; } int storage_session_set_event_type(STORAGE_SESSION *storage_session, enum EVENT_TYPE event_type) { if (!storage_session) { DBG_ERR("arg NULL\n"); return -1; } storage_session->event_type = event_type; return 0; } void storage_update_list(STORAGE_SERVER *storage_server, int flag) { STORAGE_SESSION *storage_session; char list_path[LEN_MAX_PATH] = {0}; FILE *fd; if (!storage_server) { DBG_ERR("arg NULL\n"); return; } if (STORAGE_STREAMING == flag) { memset(list_path, 0, LEN_MAX_PATH); snprintf(list_path, LEN_MAX_PATH, "%s/%s", STORAGE_ROOT_PATH, STORAGE_STREAMING_LIST); fd = fopen(list_path, "w"); if (fd) { list_for_each_entry(storage_session, &storage_server->stream_session, list) { fwrite(storage_session, sizeof(*storage_session), 1, fd); } fclose(fd); } } if (STORAGE_SNAPSHOT == flag) { memset(list_path, 0, LEN_MAX_PATH); snprintf(list_path, LEN_MAX_PATH, "%s/%s", STORAGE_ROOT_PATH, STORAGE_SNAPSHOT_LIST); fd = fopen(list_path, "w"); if (fd) { list_for_each_entry(storage_session, &storage_server->snapshot_session, list) { fwrite(storage_session, sizeof(*storage_session), 1, fd); } fclose(fd); } } } void storage_session_add_session(STORAGE_SESSION *storage_session) { STORAGE_SERVER *storage_server; STORAGE_SESSION *session, *nsession; if (!storage_session) { MYDEBUG("arg NULL\n"); return; } storage_server = storage_session->storage_server; if (STORAGE_STREAMING == storage_session->storage_type) { MYDEBUG("storage stream session = %p\n", storage_session); if (false == storage_session->storage_success) { storage_session_free(storage_session); return; } list_for_each_entry_safe(session, nsession, &storage_server->stream_session, list) { if (!strncmp(session->dev_id, storage_session->dev_id, LEN_MAX_ID) && !session->uploading_flag) { if (!strncmp(session->stream_path, storage_session->stream_path, LEN_MAX_PATH)) memset(session->stream_path, 0, LEN_MAX_PATH);//if old session's path is the same as new, do not delete file storage_session_free(session); } } if (get_dev_info(storage_server->worker_ctx->top_ctx, storage_session->dev_id, NULL)) { list_add_tail(&storage_session->list, &storage_server->stream_session); storage_update_list(storage_server, STORAGE_STREAMING); } else { //if device not exist, free session storage_session_free(storage_session); } } else { MYDEBUG("storage snapshot session = %p\n", storage_session); if (false == storage_session->storage_success) { storage_session_free(storage_session); return; } list_for_each_entry_safe(session, nsession, &storage_server->snapshot_session, list) { if (!strncmp(session->dev_id, storage_session->dev_id, LEN_MAX_ID) && !session->uploading_flag) { if (!strncmp(session->snapshot_path, storage_session->snapshot_path, LEN_MAX_PATH)) memset(session->snapshot_path, 0, LEN_MAX_PATH); storage_session_free(session); } } if (get_dev_info(storage_server->worker_ctx->top_ctx, storage_session->dev_id, NULL)) { list_add_tail(&storage_session->list, &storage_server->snapshot_session); storage_update_list(storage_server, STORAGE_SNAPSHOT); } else { //if device not exist, free session storage_session_free(storage_session); } } return; } bool storage_session_add_fixed_header(STORAGE_SESSION *storage_session) { struct tpsocket_buf *buf; if (!storage_session) { DBG_ERR("arg NULL\n"); return false; } /* add */ buf = tpbuf_snprintf(512, "%x\r\n" "--data-boundary--\r\n" "Content-Length: 218\r\n" "Content-Type: text/http-header\r\n" "\r\n" "HTTP/1.0 200 OK\r\n" "Content-Type: multipart/x-mixed-replace;boundary=data-boundary--\r\n" "X-Encrypt-Type: PLAIN\r\n" "Pragma: no-cache\r\n" "Cache-Control: no-cache\r\n" "Content-Length: -1\r\n" "Transfer-Encoding: chunked\r\n" "Connection: close\r\n\r\n" "\r\n", 292); if (buf) { list_add_tail(&buf->list, &storage_session->headers); } return true; } bool storage_stream_consumer_post(STORAGE_SESSION *storage_session) { struct list_head head; struct tpsocket_buf *buf; DEV_INFO *dev_info; char *request = NULL; if (!storage_session) { DBG_ERR("arg NULL\n"); return false; } dev_info = get_dev_info(storage_session->storage_server->worker_ctx->top_ctx, storage_session->dev_id, NULL); if (!dev_info) { DBG_ERR("get dev info failed\n"); return false; } INIT_LIST_HEAD(&head); char url[256] = {0}; request = strstr(dev_info->relay_storage_url, "/v1"); snprintf(url, 256, "%sactivity?deviceid=%s" "&eventid=%s" "&type=mixed" "&resolution=HD", request, storage_session->dev_id, storage_session->event_id); //MYDEBUG("storage token:%s\n", storage_session->xtoken); //MYDEBUG("storage cookie:%s\n", storage_session->cookie); buf = tpbuf_snprintf(1024, "POST %s HTTP/1.1\r\n" "User-Agent: %s/%s %s\r\n" // MODEL/FW_VERSION PROGRAM_NAME/PROGRAM_VERSION "Host: %s\r\n" "Transfer-Encoding: chunked\r\n" //http chunked transfer encoding "Content-Type: multipart/x-mixed-replace;boundary=%s\r\n" "X-token: %s\r\n" // X-token "X-Audio-Codec: %s\r\n" "Cookie: %s\r\n" "\r\n", url, dev_info->dev_model?dev_info->dev_model:"*", dev_info->sw_ver?dev_info->sw_ver:"*", THIS_PROCESS, storage_session->host, "data-boundary--", storage_session->xtoken, dev_info->audio_fmt, storage_session->cookie); list_add_tail(&buf->list, &head); /* header */ list_splice_tail_init(&storage_session->headers, &head); /* ready to write */ debug_show_tpbuf_list(&head, "STORAGE STREAM POST:", 3); if (false == storage_session_write_list_force(storage_session, &head, false)) { DBG_ERR("storage_session_write_list_force error\n"); tpsocket_free_buf(&head, NULL, 0); return false; } return true; } bool storage_snapshot_consumer_post(STORAGE_SESSION *storage_session) { struct list_head head; struct tpsocket_buf *buf; DEV_INFO *dev_info; char *request = NULL; if (!storage_session) { DBG_ERR("arg NULL\n"); return false; } INIT_LIST_HEAD(&head); dev_info = get_dev_info(storage_session->storage_server->worker_ctx->top_ctx, storage_session->dev_id, NULL); if (!dev_info) { DBG_ERR("get dev info failed\n"); return false; } /* add POST header */ char url[256] = {0}; int sub_header_len; char sub_header[512] = {0}; request = strstr(dev_info->storage_url, "/v1"); snprintf(url, 256, "%sdevice/%s/activity/%s/snapshot", request, storage_session->dev_id, storage_session->event_id); sub_header_len = snprintf(sub_header, 512, "--%s\r\n" "Content-Disposition: form-data;name=\"snapshot\";filename=\"example.jpg\"\r\n" "Content-Type: image/jpeg\r\n\r\n", SNAPSHOT_FILE_BOUNDARY); int content_len = sub_header_len+storage_session->content_len+strlen("\r\n")+storage_session->region_buf_len+ strlen("--")+strlen(SNAPSHOT_FILE_BOUNDARY)+strlen("--"); buf = tpbuf_snprintf(1024, "POST %s HTTP/1.1\r\n" "User-Agent: %s/%s %s\r\n" // MODEL/FW_VERSION PROGRAM_NAME/PROGRAM_VERSION "X-token: %s\r\n" // X-token "Host: %s\r\n" "Content-Length: %d\r\n" "Content-Type: multipart/form-data;boundary=%s\r\n" "\r\n" "%s",//sub_header url, dev_info->dev_model?dev_info->dev_model:"*", dev_info->sw_ver?dev_info->sw_ver:"*", THIS_PROCESS, storage_session->xtoken, storage_session->host, content_len, SNAPSHOT_FILE_BOUNDARY, sub_header); list_add_tail(&buf->list, &head); debug_show_tpbuf_list(&head, "STORAGE SNAPSHOT POST:", 3); if (false == storage_session_write_list_force(storage_session, &head, false)) { DBG_ERR("storage_session_write_list_force error\n"); tpsocket_free_buf(&head, NULL, 0); return false; } return true; } bool storage_file_produce(STORAGE_SESSION *storage_session, struct list_head *buf) { if (!storage_session) { DBG_ERR("arg NULL\n"); return false; } if (!storage_session->sock) { DBG_ERR("%p arg NULL, file = %p\n", storage_session, storage_session->file); return false; } if (tpsocket_writable(storage_session->sock)) { if (!list_empty(buf) && storage_session->chunked && !tpsocket_chunk_encode(buf)) { tpsocket_free_buf(buf, "PRODUCE", 0); return false; } storage_session_write_list_force(storage_session, buf, true); } return true; } bool storage_read_file(struct tpsocket_handler*handler, struct list_head*buf, int event) { STORAGE_SESSION *storage_session = handler->priv; //MYDEBUG("storage read file event = %s, sock = %p, consumer = %p\n", tpsocket_event_name(event), sock, storage_session); switch(event) { case TPSOCKET_EVENT_CONNECTED: break; case TPSOCKET_EVENT_REQ_HEAD: break; case TPSOCKET_EVENT_RSP_HEAD: break; case TPSOCKET_EVENT_UPGRADE: break; case TPSOCKET_EVENT_SUB_HEAD: case TPSOCKET_EVENT_STREAM: case TPSOCKET_EVENT_MESSAGE: if (false == storage_file_produce(storage_session, buf)) { return false; } goto out; //break; case TPSOCKET_EVENT_RESET: break; case TPSOCKET_EVENT_CLOSED: if (storage_session) { storage_session->file = NULL; if (storage_session->sock) { if (!list_empty(buf)) { if (storage_session->chunked) { tpsocket_chunk_encode(buf); tpsocket_write_list_force(storage_session->sock, buf, true); } } if (STORAGE_STREAMING == storage_session->storage_type) { storage_session_stream_add_tail(storage_session); } else { storage_session_snapshot_add_tail(storage_session); } } } break; case TPSOCKET_EVENT_ERROR: default: break; } tpsocket_free_buf(buf, tpsocket_event_name(event), 0); out: return true; } bool storage_start_read(STORAGE_SESSION *storage_session) { if (!storage_session) { DBG_ERR("arg NULL\n"); return false; } struct tpsocket_handler client = { .cb = storage_read_file, .flowcontrol = 10*1024, .priv = storage_session, .read_buf_max = STORAGE_MEMORY_CACHE_LIMIT, }; if (STORAGE_STREAMING == storage_session->storage_type) { storage_session->file = tpsocket_new(NULL, "stream", NULL, storage_session->stream_path, TPSOCKET_TYPE_FILE, &client); if (!storage_session->file) { DBG_ERR("open %s failed\n", storage_session->stream_path); return false; } } else { storage_session->file = tpsocket_new(NULL, "read", NULL, storage_session->snapshot_path, TPSOCKET_TYPE_FILE, &client); if (!storage_session->file) { DBG_ERR("open %s failed\n", storage_session->snapshot_path); return false; } } DBG_DBG("session = %p, sock = %p, open file = %p\n", storage_session, storage_session->sock, storage_session->file); return true; } bool tpsocket_event_storage_stream_to_cloud(struct tpsocket_handler *handler, struct list_head *buf, int event) { //debug_show_tpbuf_list(buf, "TEST", 3); struct tpsocket_fd *sock = container_of(handler, struct tpsocket_fd, handler); STORAGE_SESSION *storage_session = handler->priv; int err_code = 0; if (event != TPSOCKET_EVENT_WRITABLE) { MYDEBUG("storage stream to cloud event = %s, sock = %p, session = %p\n", tpsocket_event_name(event), sock, storage_session); if (storage_session) { DBG_DBG("Device Id = %s, session sock = %p\n", storage_session->dev_id, storage_session->sock); } } switch(event) { case TPSOCKET_EVENT_LISTEN: break; case TPSOCKET_EVENT_SHUTDOWN: break; case TPSOCKET_EVENT_ACCEPT: break; case TPSOCKET_EVENT_CONNECTED: MYDEBUG("STORAGESTREAM2CLOUD: %s:%s Connected\n", sock->addr, sock->port); if (false == storage_stream_consumer_post(storage_session)) return false; if (false == storage_start_read(storage_session)) { DBG_ERR("read stream file failed\n"); return false; } break; case TPSOCKET_EVENT_REQ_HEAD: break; case TPSOCKET_EVENT_RSP_HEAD: break; case TPSOCKET_EVENT_SUB_HEAD: break; case TPSOCKET_EVENT_UPGRADE: break; case TPSOCKET_EVENT_WRITABLE: if (storage_session->file && !list_empty(&(storage_session->file->parser.buf))) { if (storage_session->chunked && !tpsocket_chunk_encode(&(storage_session->file->parser.buf))) { tpsocket_free_buf(&(storage_session->file->parser.buf), tpsocket_event_name(event), 0); break; } storage_session_write_list_force(storage_session, &(storage_session->file->parser.buf), true); storage_session->file->read_buf_length = 0; } break; case TPSOCKET_EVENT_STREAM: return true; case TPSOCKET_EVENT_MESSAGE: err_code = common_check_error_code(buf); if (!err_code) { DBG_DBG("storage stream push success\n"); storage_session->post_success = 1; #ifndef TAPO_CAMERA } else if (err_code == -98400) { //If the recording/relay server finds out that the device is not eligible for cloud storage, // it returns an error code of -98400 upon which the firmware updates the cloud storage feature of that device. DEV_INFO *dev_info = get_dev_info(storage_session->storage_server->worker_ctx->top_ctx, storage_session->dev_id, NULL); if (dev_info) { dev_info->cloud_storage = false; } // mark as success here to avoid retrying. storage_session->post_success = 1; #endif } tpsocket_free_buf(buf, tpsocket_event_name(event), 0); common_tpsocket_close_immediately(sock); break; case TPSOCKET_EVENT_RESET: break; case TPSOCKET_EVENT_KEEPALIVE: common_tpsocket_close_immediately(sock); break; case TPSOCKET_EVENT_CLOSED: if (storage_session) { tpsocket_unbind(sock, &storage_session->sock); uloop_timeout_cancel(&storage_session->reupload_tmo); tpsocket_free2(&storage_session->file, storage_session); if (storage_session->post_success) { storage_session->stream_send_to_cloud_flag = 0; } storage_session->uploading_flag = 0; storage_session_set_push_status(storage_session, 0); /* handle timeout immediately */ storage_server_timeout_handle_immediately(storage_session->storage_server); } break; case TPSOCKET_EVENT_ERROR: default: break; } tpsocket_free_buf(buf, tpsocket_event_name(event), 0); return true; } bool tpsocket_event_storage_snapshot_to_cloud(struct tpsocket_handler *handler, struct list_head *buf, int event) { //debug_show_tpbuf_list(buf, "TEST", 3); struct tpsocket_fd *sock = container_of(handler, struct tpsocket_fd, handler); STORAGE_SESSION *storage_session = handler->priv; if (event != TPSOCKET_EVENT_WRITABLE) { MYDEBUG("storage snapshot to cloud event = %s, sock = %p, session = %p\n", tpsocket_event_name(event), sock, storage_session); if (storage_session) { DBG_DBG("Device Id = %s, session sock = %p\n", storage_session->dev_id, storage_session->sock); } } switch(event) { case TPSOCKET_EVENT_LISTEN: break; case TPSOCKET_EVENT_SHUTDOWN: break; case TPSOCKET_EVENT_ACCEPT: break; case TPSOCKET_EVENT_CONNECTED: MYDEBUG("STORAGESNAPSHOT2CLOUD: %s:%s Connected\n", sock->addr, sock->port); if (false == storage_snapshot_consumer_post(storage_session)) return false; if (false == storage_start_read(storage_session)) { DBG_ERR("read snapshot file failed\n"); return false; } break; case TPSOCKET_EVENT_REQ_HEAD: break; case TPSOCKET_EVENT_RSP_HEAD: break; case TPSOCKET_EVENT_SUB_HEAD: break; case TPSOCKET_EVENT_UPGRADE: break; case TPSOCKET_EVENT_WRITABLE: if (storage_session->file && !list_empty(&(storage_session->file->parser.buf))) { if (storage_session->chunked && !tpsocket_chunk_encode(&(storage_session->file->parser.buf))) { tpsocket_free_buf(&(storage_session->file->parser.buf), tpsocket_event_name(event), 0); break; } storage_session_write_list_force(storage_session, &(storage_session->file->parser.buf), true); storage_session->file->read_buf_length = 0; } break; case TPSOCKET_EVENT_STREAM: return true; case TPSOCKET_EVENT_MESSAGE: if (!common_check_error_code(buf)) { DBG_DBG("storage snapshot push success\n"); storage_session->post_success = 1; } tpsocket_free_buf(buf, tpsocket_event_name(event), 0); common_tpsocket_close_immediately(sock); break; case TPSOCKET_EVENT_RESET: break; case TPSOCKET_EVENT_KEEPALIVE: common_tpsocket_close_immediately(sock); break; case TPSOCKET_EVENT_CLOSED: if (storage_session) { tpsocket_unbind(sock, &storage_session->sock); uloop_timeout_cancel(&storage_session->reupload_tmo); tpsocket_free2(&storage_session->file, storage_session); if (storage_session->post_success) { storage_session->snapshot_send_to_cloud_flag = 0; //storage_session_free(storage_session); } else { storage_session->retry_count++; //list_add_tail(&storage_session->list, &storage_session->storage_server->snapshot_session); } storage_session->uploading_flag = 0; storage_session_set_push_status(storage_session, 0); /* handle timeout immediately */ storage_server_timeout_handle_immediately(storage_session->storage_server); } break; case TPSOCKET_EVENT_ERROR: default: break; } tpsocket_free_buf(buf, tpsocket_event_name(event), 0); return true; } bool storage_stream_upload(STORAGE_SESSION *storage_session) { DEV_INFO *dev_info = NULL; STORAGE_SERVER *storage_server; if (!storage_session) { DBG_ERR("arg NULL\n"); return false; } if (storage_session->sock) { DBG_DBG("sock exist\n"); return true; } storage_server = storage_session->storage_server; uloop_timeout_set(&storage_session->reupload_tmo, storage_session->reupload_idle_time); dev_info = get_dev_info(storage_server->worker_ctx->top_ctx, storage_session->dev_id, NULL); if (!dev_info) { DBG_ERR("Not find device\n"); return false; } /* create socket */ struct tpsocket_handler storage_stream_to_cloud = { .cb = tpsocket_event_storage_stream_to_cloud, .cer = MEDIACENTER_CERTFILE, .write_buf_max = STORAGE_MEMORY_CACHE_LIMIT, }; storage_session->retry_count++; if ((storage_session->sock = tpsocket_from_url(dev_info->relay_storage_url, &storage_stream_to_cloud))) { //if (storage_session->sock = tpsocket_from_url("http://192.168.137.103:8010", &storage_stream_to_cloud)) { DBG_DBG("Create session sock = %p, session = %p\n", storage_session->sock, storage_session); storage_session->sock->handler.priv = storage_session; storage_session->uploading_flag = 1; storage_session_set_push_status(storage_session, 1); } return true; } void storage_set_iot_info(void *arg, const char *host, const char *cookie, const char *token, const char *event_id) { STORAGE_SESSION *storage_session; #ifdef TAPO_CAMERA if (!arg || !host || !token || !event_id) { #else if (!arg || !host || !cookie || !token || !event_id) { #endif DBG_ERR("arg NULL\n"); return; } storage_session = (STORAGE_SESSION*)arg; snprintf(storage_session->host, LEN_MAX_HOST, "%s", host); snprintf(storage_session->cookie, LEN_MAX_COOKIE, "%s", cookie); snprintf(storage_session->xtoken, LEN_MAX_TOKEN, "%s", token); snprintf(storage_session->event_id, LEN_MAX_EVENT_ID, "%s", event_id); MYDEBUG("host: %s\n", storage_session->host); //MYDEBUG("cookie: %s\n", storage_session->cookie); //MYDEBUG("xtoken: %s\n", storage_session->xtoken); MYDEBUG("event_id: %s\n", storage_session->event_id); return; } void storage_get_iot_info_cb(void *priv, void *iot_info, char *host, char *cookie, char *token, char *event_id, int err_code) { STORAGE_SESSION *storage_session; if (!priv) { DBG_ERR("arg NULL\n"); return; } storage_session = (STORAGE_SESSION*)priv; if (err_code) { DBG_ERR("storage get iot info failed\n"); goto out; } if (!host || !cookie || !token || !event_id) { DBG_ERR("storage get iot info param error\n"); goto out; } storage_set_iot_info(storage_session, host, cookie, token, event_id); MYDEBUG("Have got iot info, reupload now\n"); if (STORAGE_STREAMING == storage_session->storage_type) { storage_stream_upload(storage_session); return; } out: storage_session->retry_count++; return; } int storage_stream_reupload(STORAGE_SESSION *storage_session) { IOT_INFO *iot_info; if (!storage_session) { DBG_ERR("arg NULL\n"); return -1; } if (storage_session->event_id[0]) { MYDEBUG("Have got iot info, reupload now\n"); if (false == storage_stream_upload(storage_session)) { storage_session->stream_send_to_cloud_flag = 0; } } else { //find event id iot_info = iot_info_find(storage_session->storage_server->worker_ctx->iot_info_server, storage_session->dev_id , storage_session->count_id); if (iot_info) { if (iot_info->ready) { #ifdef TAPO_CAMERA storage_get_iot_info_cb((void*)storage_session, NULL, iot_info->host, iot_info->cookie, iot_info->xtoken, iot_info->alarm_id, 0); #else storage_get_iot_info_cb((void*)storage_session, NULL, iot_info->host, iot_info->cookie, iot_info->token, iot_info->event_id, 0); #endif if (IOT_INFO_CONSUMER_NUM == iot_info->consumer_used_count) { iot_info_free(iot_info); } } else { storage_session->retry_count++; } } else { storage_session->stream_send_to_cloud_flag = 0; } } return 0; } void storage_force_delete_session(STORAGE_SERVER *storage_server) { struct statfs diskInfo; STORAGE_SESSION *stream_session, *nstream_session; int force = 0; if (!storage_server) { DBG_ERR("arg NULL\n"); return; } retry: if (force++ < 2 && !statfs(STORAGE_ROOT_PATH, &diskInfo)) { int free_size = (diskInfo.f_bsize * diskInfo.f_bfree) >> 20; if (free_size < 20) { MYDEBUG("FREE MEMORY %u MB!\n", free_size); if (free_size < 8) { list_for_each_entry_safe(stream_session, nstream_session, &storage_server->stream_session, list) { if (!stream_session->iot_consumer) { if ((force == 1 && stream_session->delete_flag) || (force == 2 && !stream_session->uploading_flag)) { storage_session_free(stream_session); } } } goto retry; } } } } void storage_check_list_to_upload(STORAGE_SERVER *storage_server) { STORAGE_SESSION *stream_session, *nstream_session; int ret = 0; if (!storage_server) { DBG_ERR("arg NULL\n"); return; } list_for_each_entry_safe(stream_session, nstream_session, &storage_server->stream_session, list) { if (!stream_session->iot_consumer) {//make sure alarm push finished //MYDEBUG("find storage stream session = %p\n", stream_session); if (stream_session->uploading_flag) { MYDEBUG("stream session %p uploading push busy ...\n", stream_session); break; } ret = storage_session_get_push_status(stream_session); if (1 == ret) { MYDEBUG("this stream session %p is ready, but other session of the same device push busy ...\n", stream_session); break; } else if (-1 == ret) { MYDEBUG("device not found\n"); break; } if (stream_session->stream_send_to_cloud_flag && (stream_session->retry_count < 3)) { MYDEBUG("stream session %p do upload!\n", stream_session); storage_stream_reupload(stream_session); } else { //DBG_DBG("stream session %p can be deleted\n", stream_session); stream_session->delete_flag = true; } } else { MYDEBUG("waitting iot session %p closed\n", stream_session->iot_consumer); } } } void storage_server_timeout_handle_immediately(STORAGE_SERVER *storage_server) { MYDEBUG("storage_server_timeout_handle_immediately ...\n"); if (!storage_server) { DBG_ERR("arg NULL\n"); return; } uloop_timeout_set(&storage_server->tmo, 100); } void storage_server_timeout_cb(struct uloop_timeout *tmo) { //MYDEBUG("storage server timeout cb ...\n"); STORAGE_SERVER *storage_server = container_of(tmo, STORAGE_SERVER, tmo); if (!storage_server) { DBG_ERR("storage_server NULL\n"); return; } storage_force_delete_session(storage_server); storage_check_list_to_upload(storage_server); uloop_timeout_set(tmo, 10*1000); } void storage_session_set_send_flag(void *arg, int flag) { STORAGE_SESSION *storage_session = NULL; if (!arg) { DBG_ERR("arg NULL\n"); return; } storage_session = (STORAGE_SESSION*)arg; if (STORAGE_STREAMING == storage_session->storage_type) { storage_session->stream_send_to_cloud_flag = flag; } else { storage_session->snapshot_send_to_cloud_flag = flag; } return; } void storage_session_set_iot_consumer_null(void *arg) { STORAGE_SESSION *storage_session; if (!arg) { DBG_ERR("arg NULL\n"); return; } storage_session = (STORAGE_SESSION*)arg; storage_session->iot_consumer = NULL; storage_update_list(storage_session->storage_server, storage_session->storage_type); return; } int storage_delete_empty_dir(char *filedir) { int count = 0; DIR *dir = NULL; struct dirent *Dirent = NULL; //char cmd[128] = {0}; if(!filedir){ DBG_ERR("arg NULL\n"); return -1; } dir = opendir(filedir); if(dir == NULL){ DBG_NOTICE("opendir:%s error:%s\n", filedir, strerror(errno)); return -1; } while((Dirent = readdir(dir)) != NULL){ if(0 != strncmp(Dirent->d_name, ".", sizeof(".")) && 0 != strncmp(Dirent->d_name, "..", sizeof(".."))) { count++; } } closedir(dir); if(!count){ //memset(cmd, '\0', 128); //snprintf(cmd, 128, "rm -rf %s", filedir); //if (system(cmd) < 0) { if (tpcom_system("rm", ("-rf"), ("%s", filedir)) < 0) { DBG_ERR("remove:%s error:%s\n", filedir, strerror(errno)); return -1; } DBG_DBG("remove dir:%s ok\n", filedir); } else { DBG_DBG("has %d file in %s\n", count, filedir); } return 0; } void storage_delete_file(char *file) { char *pos; char dev_id[LEN_MAX_ID] = {0}; char count_id[LEN_MAX_ID] = {0}; char dir[LEN_MAX_PATH] = {0}; if (!file) { DBG_ERR("arg NULL\n"); return; } if (!file[0]) { DBG_DBG("No file to delete\n"); return; } pos = file+strlen(STORAGE_ROOT_PATH); DBG_DBG("pos = %s\n", pos); sscanf(pos, "/%[^/]/%[^/]/%*s", dev_id, count_id); DBG_DBG("dev_id = %s, count_id = %s\n", dev_id, count_id); /* delete file */ remove(file); /* delete count id dir */ memset(dir, 0, LEN_MAX_PATH); snprintf(dir, LEN_MAX_PATH, "%s/%s/%s", STORAGE_ROOT_PATH, dev_id, count_id); storage_delete_empty_dir(dir); /* delete device dir */ memset(dir, 0, LEN_MAX_PATH); snprintf(dir, LEN_MAX_PATH, "%s/%s", STORAGE_ROOT_PATH, dev_id); storage_delete_empty_dir(dir); } void storage_move_file(char *file) { char *pos; char cmd[512] = {0}; char dev_id[LEN_MAX_ID] = {0}; char count_id[LEN_MAX_ID] = {0}; char dir[LEN_MAX_PATH] = {0}; char count_path[LEN_MAX_PATH] = {0}; if (!file || !file[0]) { DBG_ERR("arg NULL\n"); return; } pos = file+strlen(STORAGE_ROOT_PATH); MYDEBUG("pos = %s\n", pos); sscanf(pos, "/%[^/]/%[^/]/%*s", dev_id, count_id); MYDEBUG("dev_id = %s, count_id = %s\n", dev_id, count_id); /* remove file */ snprintf(count_path, LEN_MAX_PATH, "%s/%s/%s/", NFS_ROOT, dev_id, count_id); if (access(count_path, F_OK)) { //snprintf(cmd, 512, "mkdir -p %s", count_path); //system(cmd); tpcom_system("mkdir", ("-p"), ("%s", count_path)); } memset(cmd, 0, 512); snprintf(cmd, 512, "mv %s %s", file, count_path); DEBUG("cmd: %s\n", cmd); //system(cmd); tpcom_system("mv", (file), ("%s", count_path)); if (!access(file, F_OK)) { DBG_ERR("%s failed\n", cmd); remove(file); } /* delete count id dir */ memset(dir, 0, LEN_MAX_PATH); snprintf(dir, LEN_MAX_PATH, "%s/%s/%s", STORAGE_ROOT_PATH, dev_id, count_id); storage_delete_empty_dir(dir); /* delete device dir */ memset(dir, 0, LEN_MAX_PATH); snprintf(dir, LEN_MAX_PATH, "%s/%s", STORAGE_ROOT_PATH, dev_id); storage_delete_empty_dir(dir); } void storage_session_free(STORAGE_SESSION *storage_session) { MYDEBUG("STORAGE SESSION FREE %p\n", storage_session); if (!storage_session) { return; } list_del(&storage_session->list); uloop_timeout_cancel(&storage_session->reupload_tmo); tpsocket_free_buf(&storage_session->headers, NULL, 0); if (storage_session->sock) { storage_session_set_push_status(storage_session, 0); } tpsocket_free2(&storage_session->sock, storage_session); tpsocket_free2(&storage_session->file, storage_session); if (storage_session->iot_consumer) { alarm_consumer_set_storage_session_null(storage_session->iot_consumer); } /* telemetry */ storage_session_telemetry_failure_count(storage_session); /* delete file */ if (STORAGE_STREAMING == storage_session->storage_type) { if (STORAGE_DEBUG && !access("/nfsroot", F_OK)) { storage_move_file(storage_session->stream_path); } else { storage_delete_file(storage_session->stream_path); } } else { if (STORAGE_DEBUG && !access("/nfsroot", F_OK)) { storage_move_file(storage_session->snapshot_path); } else { storage_delete_file(storage_session->snapshot_path); } } free(storage_session); } bool tpsocket_event_consumer_to_storage(struct tpsocket_handler *handler, struct list_head *buf, int event) { struct tpsocket_fd *sock = container_of(handler, struct tpsocket_fd, handler); STORAGE_SESSION *storage_session = (STORAGE_SESSION*)handler->priv; if (STORAGE_SNAPSHOT == storage_session->storage_type) { MYDEBUG("storage snpashot cb: sock=%p, event=%d\n", sock, event); } switch(event) { case TPSOCKET_EVENT_CONNECTED: break; case TPSOCKET_EVENT_REQ_HEAD: break; case TPSOCKET_EVENT_RSP_HEAD: break; case TPSOCKET_EVENT_SUB_HEAD: break; case TPSOCKET_EVENT_UPGRADE: break; case TPSOCKET_EVENT_MESSAGE: break; case TPSOCKET_EVENT_STREAM: break; case TPSOCKET_EVENT_RESET: break; case TPSOCKET_EVENT_CLOSED: if (storage_session) { storage_session->file = NULL; list_del_init(&storage_session->list); storage_session_add_session(storage_session); } break; case TPSOCKET_EVENT_ERROR: MYDEBUG("tpsocket_event_consumer_to_storage ERROR %p\n", storage_session); storage_session->storage_success = false; default: break; } tpsocket_free_buf(buf, tpsocket_event_name(event), 0); return true; } STORAGE_SESSION *storage_session_new(char *path, STORAGE_SERVER *storage_server) { STORAGE_SESSION *storage_session; if (!path || ! storage_server) { DBG_ERR("arg NULL\n"); return NULL; } storage_session = (STORAGE_SESSION*)malloc(sizeof(*storage_session)); if (!storage_session) { DBG_ERR("malloc failed\n"); return NULL; } memset(storage_session, 0, sizeof(*storage_session)); INIT_LIST_HEAD(&storage_session->list); INIT_LIST_HEAD(&storage_session->headers); storage_session->storage_server = storage_server; storage_session->reupload_idle_time = 10000; storage_session->reupload_tmo.cb = storage_session_reupload_timeout; storage_session->storage_success = true; storage_session->delete_flag = false; /* create socket */ struct tpsocket_handler client = { .cb = tpsocket_event_consumer_to_storage, .flowcontrol = 500000, .priv = storage_session }; storage_session->file = tpsocket_new(NULL, "write", NULL, path, TPSOCKET_TYPE_FILE, &client); if (!(storage_session->file)) { DBG_ERR("tpsocket_new failed\n"); free(storage_session); return NULL; } //consumer->chunked = 0; //consumer->store = 1; return storage_session; } int storage_find_file_in_list(char *dev_id, char *timestamp_str, struct list_head *head) { STORAGE_SESSION *session; int timestamp; if (!dev_id || !timestamp_str || !head) { DBG_ERR("arg NULL\n"); return -1; } timestamp = atoi(timestamp_str); list_for_each_entry(session, head, list) { if (!strncmp(session->dev_id, dev_id, LEN_MAX_ID) && (session->event_timestamp == timestamp)) { return 0; } } return -1; } void storage_check_file(STORAGE_SERVER *storage_server) { if (!storage_server) { DBG_ERR("arg NULL\n"); return; } DIR *dir = NULL, *dev_dir = NULL; struct dirent *root_dirent = NULL, *dev_dirent = NULL; char dev_id[LEN_MAX_ID] = {0}, timestamp[LEN_MAX_ID] = {0}; char dev_path[LEN_MAX_PATH] = {0}, event_path[LEN_MAX_PATH] = {0}, file_path[LEN_MAX_PATH] = {0}; dir = opendir(STORAGE_ROOT_PATH); if(dir == NULL){ DBG_NOTICE("opendir:%s error:%s\n", STORAGE_ROOT_PATH, strerror(errno)); return; } while((root_dirent = readdir(dir)) != NULL){ /* dev */ if(0 == strncmp(root_dirent->d_name, ".", sizeof(".")) || 0 == strncmp(root_dirent->d_name, "..", sizeof(".."))) { continue; } memset(dev_path, 0, LEN_MAX_PATH); snprintf(dev_path, LEN_MAX_PATH, "%s/%s", STORAGE_ROOT_PATH, root_dirent->d_name); dev_dir = opendir(dev_path); if(dev_dir == NULL){ DBG_NOTICE("opendir:%s error:%s\n", dev_path, strerror(errno)); continue; } memset(dev_id, 0, LEN_MAX_ID); snprintf(dev_id, LEN_MAX_ID, "%s", root_dirent->d_name); while ((dev_dirent = readdir(dev_dir)) != NULL) { /* event */ if(0 == strncmp(dev_dirent->d_name, ".", sizeof(".")) || 0 == strncmp(dev_dirent->d_name, "..", sizeof(".."))) { continue; } memset(timestamp, 0, LEN_MAX_ID); snprintf(timestamp, LEN_MAX_ID, "%s", dev_dirent->d_name); memset(event_path, 0, LEN_MAX_PATH); snprintf(event_path, LEN_MAX_PATH, "%s/%s/%s", STORAGE_ROOT_PATH, dev_id, timestamp); memset(file_path, 0, LEN_MAX_PATH); snprintf(file_path, LEN_MAX_PATH, "%s/%s/%s/%s", STORAGE_ROOT_PATH, dev_id, timestamp, STORAGE_STREAMING_FILE); if (!access(file_path, F_OK)) { DBG_NOTICE("streaming file: %s\n", file_path); if (storage_find_file_in_list(dev_id, timestamp, &storage_server->stream_session)) { storage_delete_file(file_path); } } memset(file_path, 0, LEN_MAX_PATH); snprintf(file_path, LEN_MAX_PATH, "%s/%s/%s/%s", STORAGE_ROOT_PATH, dev_id, timestamp, STORAGE_SNAPSHOT_FILE); if (!access(file_path, F_OK)) { DBG_NOTICE("snapshot file: %s\n", file_path); if (storage_find_file_in_list(dev_id, timestamp, &storage_server->snapshot_session)) { storage_delete_file(file_path); } } storage_delete_empty_dir(event_path); } closedir(dev_dir); } closedir(dir); return; } void storage_get_iot_info(STORAGE_SESSION *session, struct list_head *iot_info_head) { IOT_INFO *iot_info; if (!session || !iot_info_head) { DBG_ERR("arg NULL\n"); return; } list_for_each_entry(iot_info, iot_info_head, list) { if (!strncmp(session->dev_id, iot_info->dev_id, LEN_MAX_ID) && (session->count_id == iot_info->count_id)) { #ifdef TAPO_CAMERA storage_set_iot_info(session, iot_info->host, iot_info->cookie, iot_info->xtoken, iot_info->alarm_id); #else storage_set_iot_info(session, iot_info->host, iot_info->cookie, iot_info->token, iot_info->event_id); #endif break; } } } bool storage_check_version() { FILE *version_fd; char version_path[LEN_MAX_PATH] = {0}, version[4] = {0}; //char cmd[128] = {0}; snprintf(version_path, LEN_MAX_PATH, "%s/%s", STORAGE_ROOT_PATH, STORAGE_VERSION_FILE); if (access(version_path, F_OK)) { DBG_NOTICE("storage version file not exist\n"); goto failed; } if (!(version_fd = fopen(version_path, "r"))) { DBG_NOTICE("open storage version file failed\n"); goto failed; } if (1 != fread(version, 4, 1, version_fd)) { DBG_NOTICE("read storage version file error\n"); fclose(version_fd); goto failed; } fclose(version_fd); if (strncmp(version, STORAGE_VERSION, 4)) { DBG_NOTICE("old version\n"); goto failed; } return true; failed: /* delete storage */ //snprintf(cmd, 128, "rm -rf %s/*", STORAGE_ROOT_PATH); //system(cmd); tpcom_system("rm", ("-rf"), ("%s", STORAGE_ROOT_PATH)); tpcom_system("mkdir", ("-p"), ("%s", STORAGE_ROOT_PATH)); /* create version */ snprintf(version, 4, "%s", STORAGE_VERSION); version_fd = fopen(version_path, "w"); if (version_fd) { fwrite(version, 4, 1, version_fd); fclose(version_fd); } return false; } void storage_file_list_init(STORAGE_SERVER *storage_server) { FILE *iot_info_fd, *stream_fd, *snapshot_fd; char iot_info_path[LEN_MAX_PATH] = {0}, stream_path[LEN_MAX_PATH] = {0}, snapshot_path[LEN_MAX_PATH] = {0}; struct list_head list; IOT_INFO *iot_info, *niot_info; if (!storage_server) { DBG_ERR("arg NULL\n"); return; } /* check version */ if (false == storage_check_version()) { return; } /* iot info */ INIT_LIST_HEAD(&list); snprintf(iot_info_path, LEN_MAX_PATH, "%s/%s", STORAGE_ROOT_PATH, STORAGE_IOT_INFO_LIST); iot_info_fd = fopen(iot_info_path, "r"); if (iot_info_fd) { while (1) { iot_info = malloc(sizeof(IOT_INFO)); if (fread(iot_info, sizeof(IOT_INFO), 1, iot_info_fd) <= 0) { free(iot_info); break; } list_add_tail(&iot_info->list, &list); } fclose(iot_info_fd); } /* stream session */ snprintf(stream_path, LEN_MAX_PATH, "%s/%s", STORAGE_ROOT_PATH, STORAGE_STREAMING_LIST); stream_fd = fopen(stream_path, "r"); if (stream_fd) { while (1) { STORAGE_SESSION *stream_session = malloc(sizeof(STORAGE_SESSION)); if (fread(stream_session, sizeof(STORAGE_SESSION), 1, stream_fd) <= 0) { free(stream_session); break; } DBG_NOTICE("dev_id: %s, count_id: %d\n", stream_session->dev_id, stream_session->count_id); INIT_LIST_HEAD(&stream_session->list); INIT_LIST_HEAD(&stream_session->headers); stream_session->sock = NULL; stream_session->file = NULL; stream_session->uploading_flag = 0; stream_session->producer = NULL; stream_session->iot_consumer = NULL; stream_session->storage_server = storage_server; storage_session_add_fixed_header(stream_session); memset(&stream_session->reupload_tmo, 0, sizeof(struct uloop_timeout)); list_add_tail(&stream_session->list, &storage_server->stream_session); storage_get_iot_info(stream_session, &list); } fclose(stream_fd); } /* snapshot session */ snprintf(snapshot_path, LEN_MAX_PATH, "%s/%s", STORAGE_ROOT_PATH, STORAGE_SNAPSHOT_LIST); snapshot_fd = fopen(snapshot_path, "r"); if (snapshot_fd) { while (1) { STORAGE_SESSION *snapshot_session = malloc(sizeof(STORAGE_SESSION)); if (fread(snapshot_session, sizeof(STORAGE_SESSION), 1, snapshot_fd) <= 0) { free(snapshot_session); break; } DBG_NOTICE("dev_id: %s, count_id: %d\n", snapshot_session->dev_id, snapshot_session->count_id); INIT_LIST_HEAD(&snapshot_session->list); INIT_LIST_HEAD(&snapshot_session->headers); snapshot_session->sock = NULL; snapshot_session->file = NULL; snapshot_session->uploading_flag = 0; snapshot_session->producer = NULL; snapshot_session->iot_consumer = NULL; snapshot_session->storage_server = storage_server; memset(&snapshot_session->reupload_tmo, 0, sizeof(struct uloop_timeout)); list_add_tail(&snapshot_session->list, &storage_server->snapshot_session); storage_get_iot_info(snapshot_session, &list); } fclose(snapshot_fd); } /* free & delete */ list_for_each_entry_safe(iot_info, niot_info, &list, list) { free(iot_info); } /* check file */ storage_check_file(storage_server); return; } void storage_server_free(STORAGE_SERVER *storage_server) { STORAGE_SESSION *storage_session, *nstorage_session; if (!storage_server) { return; } uloop_timeout_cancel(&storage_server->tmo); list_for_each_entry_safe(storage_session, nstorage_session, &storage_server->stream_session, list) { storage_session_free(storage_session); } list_for_each_entry_safe(storage_session, nstorage_session, &storage_server->snapshot_session, list) { storage_session_free(storage_session); } if (storage_server->worker_ctx) { storage_server->worker_ctx->storage_server = NULL; } list_del_init(&storage_server->list); free(storage_server); } int storage_server_init(WORKER_CTX *worker_ctx) { STORAGE_SERVER *storage_server; if (!worker_ctx) { DBG_ERR("worker ctx NULL\n"); return -1; } /* create */ storage_server = (STORAGE_SERVER*)malloc(sizeof(*storage_server)); if (!storage_server) { DBG_ERR("malloc failed\n"); return -1; } memset(storage_server, 0, sizeof(*storage_server)); storage_server->worker_ctx = worker_ctx; INIT_LIST_HEAD(&storage_server->list); INIT_LIST_HEAD(&storage_server->stream_session); INIT_LIST_HEAD(&storage_server->snapshot_session); //timer to re-upload storage_server->tmo.cb = storage_server_timeout_cb; uloop_timeout_set(&storage_server->tmo, 10*1000); /* set */ worker_ctx->storage_server = storage_server; /* file list init */ storage_file_list_init(storage_server); return 0; }
09-19
LMHMambaOut The process of LMHMambaOut consists of a lightweight multi-head gate CNN (LMHGCNN), layer normalization, and a linear layer. It adopts residual connection, as shown in Figure 5a, and the specific process can be described as follows: ˆX = Linear(X ′ ) (7) X ′ = LMHGCNN(LN Xi xy , j  ) + Xi xy , j (8) where Xi xy , j denotes the input feature vector and LN denotes layer normalization. Linear represents the linear layer, and LMHGCNN represents the LMHGCNN operation. Since it is necessary to extract the global spatial information from multiple scales, we optimized the gate CNN block of the MambaOut core module and proposed a lightweight multi-head gate CNN(LMHGCNN), as shown in Figure 5b. LMHGCNN is the LMHMambaOut core module. Compared with the core module (GCNN) of MambaOut, standard convolution was substituted with LS2CM. To make up for the fact that GCNN cannot extract global features at multiple scales, LMHGCN adopts Remote Sens. 2025, 17, 1864 8 of 27 LS2CM parallel connections of different convolutional kernels to establish multi-scale global context relationships quickly and effectively. The LMHGCNN can be described as follows: LMHGCNN = Xtoken K σ(XLN) (9) where Xtoken represents the features after multi-scale fusion vector, σ denotes the Gelu activation function, and XLN denotes the output of the input Xi xy , j after passing through layer normalization. The activation function represents the activation spatial features and assigns different weights to different spatial feature positions to represent spatial context dependency. The multi-scale LS2CM is used to activate the rich spectral semantic information, which is expressed as follows: Xtoken = concat(LS2CM1, LS2CM2, LS2CM3) (10) where LS2CM1, LS2CM2, and LS2CM3 represent depth-wise convolution kernel sizes of 1 × 1, 3 × 3, and 5 × 5, respectively这是原文,看看有没有需要修改的
07-02
评论
成就一亿技术人!
拼手气红包6.0元
还能输入1000个字符  | 博主筛选后可见
 
红包 添加红包
表情包 插入表情
 条评论被折叠 查看
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值