编译加速 remote cache

最新推荐文章于 2024-05-19 14:36:36 发布

weixin_30832983

最新推荐文章于 2024-05-19 14:36:36 发布

阅读量434

点赞数

CC 4.0 BY-SA版权

文章标签：运维 golang python

原文链接：http://www.cnblogs.com/cjyp/p/10622545.html

本文介绍了一种名为remotecache的远程编译缓存技术，该技术通过将编译输入的SHA码作为key与编译输出建立对应关系，存储在远程缓存中，从而加速编译过程。当进行编译时，若能找到匹配的key则直接获取对应的value，否则在本地进行编译并考虑是否更新远程缓存。此外还讨论了distcc和BuildGrid等分布式编译方案。

remote cache
大致原理就是根据编译输入的sha码与编译输出建立（key,value）对，存储在远端cache．编译过程如果能匹配到key，则获取GET到value.
如果hit不到key，则客户端在本地编译，并控制是否更新远端缓存库．

https://github.com/buchgr/bazel-remote/
(https://golang.org/pkg/net/http/)   KEY: bazel-remote/server/http.go
.......
docker run -d -v `pwd`/bazel_cache/:/data -p 8080:8080 buchgr/bazel-remote-cache --max_size=20

build:remote --experimental_strict_action_env --remote_http_cache=http://192.168.1.102:8080
build:remote --experimental_execution_log_file=/tmp/exec1.log
build:remote --build_event_text_file=/tmp/bep.txt
build:remote --remote_upload_local_results=false
build:remote_cache --remote_upload_local_results=true

[3,541 / 3,565] 12 actions, 7 running
...lation_world:simulation_world_service; 0s remote-cache
.../backend/sim_control:sim_control; 0s remote-cache
...lation_world:simulation_world_service; 0s remote-cache
Compiling .../dreamview/backend/main.cc; 0s remote-cache
...lation_world:simulation_world_updater; 0s remote-cache
...lation_world:simulation_world_updater; 0s remote-cache
.../handlers:websocket_handler_test; 0s remote-cache
[-----] .../backend/handlers:image_handler_cpplint ...

...

INFO: Elapsed time: 68.628s, Critical Path: 8.32s
INFO: 1212 processes: 1212 remote cache hit.
INFO: Build completed successfully, 3700 total actions
modules/perception/proto/perception_lane.proto: warning: Import modules/common/proto/error_code.proto but not used.
1============================
[ OK ] Build passed!
[INFO] Took 234 seconds
============================

===================================================================================================
distribute compilation

Some conclusions:
Buildfarm: Now it maybe still exists much problems. The work plan of project Buildfarm https://github.com/bazelbuild/bazel/projects/1.
Buildbarn: The implement of Buildfarm in go. So see Buildfarm.
BuildGrid: This project is an open source proj by python. It seems may work for us, and not try yet.
distcc: This proj may be mature, but cannot use in bazel directly．And the bazel team suggests that remote execution has made huge leaps in the past year and a half and should be ready for you to try as a replacement of distcc.

In a word, try distcc or BuildGrid, and wait for Buildfarm.

在所有您想用来编译的机器上安装 distcc
启动每台机器上的 distcc 新进程
用变量名导出 DISTCC_HOSTS 环境变量
启动 distcc 监视器（这样就可以观察正在发生什么事！）
使用 CC=distcc ./configure ，而不是使用 ./configure 来进行配置
使用 make -j n ，而不是使用 make 或 make -j 2 来进行编译，其中，n 是 DISTCC_HOSTS中机器数目的两到三倍

转载于:https://www.cnblogs.com/cjyp/p/10622545.html