联邦学习fate笔记小结

20230408
fate arbiror 监督方
fate有一个exchge节点
20211006

https://gitee.com/jfdwd/FedRec
fate推荐
faterec

job_config和task_config不是同一个东西
task_config包含job_config里面的一些参数

20210930

导包错误
这里的导包错误是因为把miniconda变成了source_root
取消之后就好了
no module named requests 错误是没有在服务器上面把项目所在地的上一层目录设置为环境变量
在pycharm中设置为content_root的意思就是把这个路径加入系统环境变量
一个项目中只能有一个content_root 可以是多个source_folders

ModuleNotFoundError
: No module named "arch": 将PYTHONPATH设置为fate_flow目录的父目录。

20210922

lmdb.Error: Attempt to operate on closed/deleted/dropped object.
把分区数设小一点 内存不够 最好把之前上传的数据物理删掉之后再重新
上传

https://github.com/FederatedAI/FedRec/blob/master/fate_flow/README_zh.md
运行流程fateflow

https://fate.readthedocs.io/en/latest/_build_temp/examples/dsl/v1/README.html?highlight=download
下载预测结果

conf文件两个地方其实是同一个东西
test 的地方只是给个案例给人参照看的

component homo_lr_o
module： homoLR

20210917

Parameter object is the only way to pass user-define runtime parameters to the developing module, so every module has it’s own parameter object. In order to define a usable parameter object, three steps will be needed.

The purpose to define a setting conf is that fate_flow module extract this file to get the information of how to start program of the module.

upstream components(DataIO for example)

https://fate.readthedocs.io/en/latest/_build_temp/doc/develop_guide.html
开发指南

flow data upload -c /fate-root/fate/examples/dsl/v1/homo_logistic_regression/upload_data_guest.json

上传数据的时候用完整的绝对路径

具体执行代码的模块
task_executor.py

-j 202109170938469173010 -n hetero_lr_0 -t 202109170938469173010_hetero_lr_0 -v 0 -r host -p 10000 -c /fate-root/fate/jobs/202109170938469173010/host/10000/hetero_lr_0/202109170938469173010_hetero_lr_0/0/task_parameters.json --run_ip 127.0.0.1 --job_server 127.0.0.1:9380


-j 202109170938469173010 -n hetero_lr_0 -t 202109170938469173010_hetero_lr_0 -v 0 -r guest -p 9999 -c /fate-root/fate/jobs/202109170938469173010/guest/9999/hetero_lr_0/202109170938469173010_hetero_lr_0/0/task_parameters.json --run_ip 127.0.0.1 --job_server 127.0.0.1:9380


模型部分 task_executor的参数

-j 202109170938469173010 -n dataio_0 -t 202109170938469173010_dataio_0 -v 0 -r guest -p 9999 -c /fate-root/fate/jobs/202109170938469173010/guest/9999/dataio_0/202109170938469173010_dataio_0/0/task_parameters.json --run_ip 127.0.0.1 --job_server 127.0.0.1:9380

 -j 202109170938469173010 -n dataio_0 -t 202109170938469173010_dataio_0 -v 0 -r host -p 10000 -c /fate-root/fate/jobs/202109170938469173010/host/10000/dataio_0/202109170938469173010_dataio_0/0/task_parameters.json --run_ip 127.0.0.1 --job_server 127.0.0.1:9380

dataio task_executor的参数

只需修改对应部分的参数就行了

20210915

需要进入细节研究的是我们的每个任务模块(fate中称为component)，比如（intersect,featurebinning,hetero_lr等），所以我们若果想debug，必须找到fate-flow-server执行每个模块的启动代码（或者说执行入口）

与具体每个component的执行入口最相关的两个文件 task_scheduler.py 和task_executor.py

看文件名我们就很清楚知道，第一个是task（这里起名为task，是需要和job区分，job主要是指我们整体的作业，而task是根据job配置文件拆分后得到的每个子任务）调度相关代码，第二个是task执行相关的代码

https://zhuanlan.zhihu.com/p/105798559
https://zhuanlan.zhihu.com/p/342079952?ivk_sa=1024320u
联邦学习开源框架FATE研究–使用PyCharm开发和调试

C:\Users\hz\AppData\Local\JetBrains\PyCharm2020.1\remote_sources\4510329\-329753495\pipeline

pipeline 运行时所需要的文件所在位置

{"data":{"board_url":"http://127.0.0.1:8080/index.html#/dashboard?job_id=202109151539074672960&role=guest&party_id=10000","job_dsl_path":"/fate-root/fate/jobs/202109151539074672960/job_dsl.json","job_id":"202109151539074672960","job_runtime_conf_on_party_path":"/fate-root/fate/jobs/202109151539074672960/guest/job_runtime_on_party_conf.json","job_runtime_conf_path":"/fate-root/fate/jobs/202109151539074672960/job_runtime_conf.json","logs_directory":"/fate-root/fate/logs/202109151539074672960","model_info":{"model_id":"arbiter-9999#guest-10000#host-9999#model","model_version":"202109151539074672960"},"pipeline_dsl_path":"/fate-root/fate/jobs/202109151539074672960/pipeline_dsl.json","train_runtime_conf_path":"/fate-root/fate/jobs/202109151539074672960/train_runtime_conf.json"},"jobId":"202109151539074672960","retcode":0,"retmsg":"success"}

运行过程中各种文件信息的位置

D:\project_pycharm\fate-test\venv\lib\python3.6\site-packages\flow_sdk\client\base.py
这个文件里面的 response 里面

DSL:就是根据pipeline流程自动生成的一个流程配置文件
components就等于整个DSL 文件

通过model_info 来检索模型
model_id和model_version

submit_conf
initiator
role
job_parameters
componets_parameters

1   File "./fate/python/fate_flow/operation/task_executor.py", line 154, in run_task
12     run_object.run(component_parameters_on_party, task_run_args)
13   File "./fate/python/federatedml/model_base.py", line 98, in run
14     this_data_output = func(*real_param)
15   File "./fate/python/federatedml/util/data_transform.py", line 885, in fit
16     data_inst = self.transformer.read_data(data_inst, "fit")
17   File "./fate/python/federatedml/util/data_transform.py", line 120, in read_data
18     abnormal_detection.empty_table_detection(input_data)
19   File "./fate/python/federatedml/util/abnormal_detection.py", line 29, in empty_table_detection
20     raise ValueError(f"Count of data_instance is 0: {data_instances}")
21 ValueError: Count of data_instance is 0: <fate_arch.computing.standalone._table.Table object at 0x7fa07f54d828>

没有事先上传所需的数据表

20210908

dsl 定义了整个模型的运行流程

fate 启动流程
bash init.sh init 一定要执行

https://fate.readthedocs.io/en/latest/search.html?q=PSI&check_keywords=yes&area=default#
搜索各个专业名词

https://fate.readthedocs.io/en/latest/_build_temp/python/fate_client/pipeline/component/README.html?highlight=component#component-list

components lis 组件列表

fate flow是训练相关，serving做预测

打不开 fateboard 就执行一下
bash init.sh init ？

FATE-Flow Client 命令行使用说明

https://fate.readthedocs.io/en/latest/_build_temp/python/fate_client/flow_client/README_zh.html?highlight=model%20version

No such file or directory federate
训练,预测,部署的 guest,host id 不一致导致的

没上传数据

单机fate运行步骤

1. 启动init_env.sh
2. 启动fate_flow_server
3. 跑具体的代码

ImportError: cannot import name 'DispatcherMiddleware'

#  path to data
#  default fate installation path
DATA_BASE = "/data/projects/fate"

Toy_example Test
cd standalone_fate_master_${version}
source bin/init_env.sh
python ./examples/toy_example/run_toy_example.py 10000 10000 0

(venv) [root@localhost standalone_fate_master_1.6.1]# 

启动环境之后 才能各种路径识别

20210902

venv 目录在联邦自己的路径下
最主要是启动这个server.py 文件 启动服务器

https://gitee.com/WeBank/FATE/blob/master/doc/develop_guide_zh.rst
fate中文

https://blog.youkuaiyun.com/WenDong1997/article/details/106743620/
联邦学习框架FATE实践（训练/测试步骤及参数说明）