轻量级的软件开发方法学，如XP和近期流行的敏捷软件开发，都注重代码的持续集成和自动测试。开发人员以非常短的迭代周期工作，以便能够经常性的交付可以工作的软件。在这样的软件方法学中，代码持续集成和自动测试极其重要，以至于被形象地称为项目开发的心跳，心跳停止之时通常是项目死亡之时。这个至关重要的心跳，通常由Nightly Build系统来实现。
实现Nightly Build途径之一是利用已有的工具。
“对于Java开发项目，支持Night Build的软件很多，开源的选择有AntHill OS和CruiseControl，后者是大名鼎鼎的ThoughtWorks的作品，配置起来比较麻烦。而AntHill是一个Java Web应用，拥有友好的定制界面，是2003年Jolt大奖的Productivity Award获得者，比CruiseControl更容易入手。”
这方面的工具中我推荐Buildbot。BuildBot基于Python，配置简洁明了。BuildBot是分布式结构，master监控若干slaves（监视状态，下发命令，搜集结果），各个slave按照指示执行update/compile/test并返回结果。Python和其他许多项目利用它做持续构建。例如Python的持续构建报告在http://www.python.org/dev/buildbot/；Wireshark的构建报告http://buildbot.wireshark.org/trunk/。
以上工具都只是持续构建的框架，版本构建的每个步骤（编写Ant脚本、Python脚本或者其他）仍然需要自己实现。
途径之二就是完全自己动手：脚本+crond服务。脚本可以选择Python/Perl/TCL/Shell等。
按照我的理解：持续构建分为如下几个部分：
(1)调度策略：定时构建，还是每次提交就构建？分布式构建还是集中式构建？
BuildBot支持定时构建/每次提交构建等多种，支持分布式构建。而自己动手则仅支持定时构建，仅支持集中式构建。
(2)版本构建步骤：一个版本是由多个组件构成的。这些组件之间的依赖关系如何？版本特定的操作有哪些（如更新Changlog，计算Build number等）？
BuildBot不关心组件依赖关系和版本特定操作；AntHill之类的工具直接支持这些概念。而自己动手则需要自己处理这些东西。
(3)组件构建步骤：从何处获取最新版本源码？如何编译？如何放置到版本文件中去？
一个组件通常由一个Makefile（传统C/C++项目），Ant脚本（Java项目）或Scons脚本（我偏爱的，Make替代品）来管理，所以构建步骤非常简单。所有的工具都将单个组件的构建任务交给用户编写脚本片段去做。
我偏爱自己编写脚本做Nightly Build，因为它简单、灵活、够用。我不认为大项目非得用个Nightly Build工具。大项目只是组件多一些，但每个组件的构建步骤仍然是极其简单的一个make命令搞定的事情！如果可移植性不是很重要，就不需要分布式构建。如果组件之间的依赖关系很复杂，只能说明项目规划差劲。
中小型项目需要Nightly Build吗？项目在开发阶段没有做Nightly Build，现在进入维护阶段了，还需要费劲实现这个吗？按照我的经验，回答都是肯定的。我维护过一个中等规模的针对嵌入式Linux的项目IPCam，大的组件只有两个，其他还有配置文件、初始化脚本等。这个项目的版本文件内容包括两部分：发布说明和目标板上用的Linux根文件系统。这个项目针对不同的目标平台分别构建版本文件。这个项目的版本文件构建步骤比较简单，就是从各个SVN服务器上取出最新版本，编译或者删除.svn临时目录后，替换掉上一版本文件中对应的组件，更新releasenotes，最后打包。就这么简单的步骤，我和同事们往往花15分钟时间小心翼翼地做，但仍然不时地出错，例如忘记了删除.svn临时目录、为某目标平台编译组件时忘记了修改Makefile中相应的选项、不同目标平台的组件弄混了、忘记在发布说明中添加新版本的说明...在我花两三天时间为这个项目编写500多行的Python脚本之后，整个项目组在制作版本文件方面花费的心思从此降为0。
其实实现NightlyBuild机制并不难。编写一个脚本，将其作为一个crond任务于每晚运行即可。在众多脚本语言中，我偏爱Python。Python语法简明并且许多事情都能做得更好。Perl和Shell稍微复杂点的语法就难以理解了（理解是维护的前提），远不如Python好用。
以下介绍我在上述项目中的Nightly Build脚本（涉及项目细节的，为保密起见作了适当修改）。它实现了以下功能：
(1)每晚构建（作为一个crond任务即可）。
(2)无论构建成功与否都有邮件通知。
(3)没有修改的组件不作编译和更新动作。
(4)记录了构建轨迹，即每次构建出的文件版本号与各个SVN库的修订号的对应关系。方便下次构建，也方便事后的问题跟踪。
(5)把自上次构建以来所有的SVN提交动作的log作为本次构建的changelog，放入发布说明中。如果是构建的版本文件需要正式发布（如发布给客户），则changelog需要人工调整一下。
需要说明的是，它并不是一个框架或者库。要修改成框架或者库要花心思，但它目前对我来说够用了。所以如果其他项目要移植这个脚本的话，需要改动一些东西：(1)为每个组件定义一个类实现其特有的构建步骤、更新版本文件中相应组件的操作。(2)修改版本构建步骤，也就是Project::makeRamdisk()方法。(3)修改配置文件build.conf以反映特定的项目-SVN库-组件的3层关系。另外事先要安装svn客户端的Python绑定(http://pysvn.tigris.org/) 。对于CVS库，可用http://pycvs.sourceforge.net/提供的pycvs。
在下面的脚本中，有3个组件Comp1/Comp2/Comp3。Comp1和Comp2位于同一SVN库中，且都需要编译；Comp3位于另一SVN库中，且不需要编译（例如配置文件、网页之类的）。
文件autobuild.py内容如下：
#! /usr/bin/env python
"""Autobuild: the main tool of nightly build mechanism adopted by my IPCam team.
Requirements:
1. You should has the privilege to do as follow: mount and unmount at shell via sudo, delete and write in ramdisk rootfs when updating it.
2. The config file (normally './build.conf', but can specified by option '-f') is good.
3. Require some common tools on GUN/Linux: Python, GNU environment(make/tar/gzip...), sudo.
4. Require ipcam cross platform compile tool (arm-linux-gcc) installed.
5. Require scons (http://www.scons.org/) installed.
6. Require pysvn (http://pysvn.tigris.org/) installed.
Main features:
1. Project, repositories and components are organized into a 3-levels tree.
The config file 'build.conf' records the tree.
2. A component may have several directories according to target board type.
3. Assure consistency of the componnets's revision in the same repository.
4. Ignore unchanged components when buiding or updating rootfs for the whole project.
5. Stop building immediatly and send email to the core members when building
any component faild.
6. Record repositories' revisions and ramdisks' versions in 'build.conf' and update them
whenever a new version made out.
7. Compute next build number automaticly.
8. Send mail to the whole team when a new version made out. The content of the mail:
ramdisks' version, repositories' revisions and the logs since the last version."""
import sys
import os
import os.path
import re
import pysvn
import time
import commands
import shutil
import smtplib
from optparse import OptionParser
def isRoot():
euid = os.geteuid()
return(euid==0)
def sendmails(config, bldSuccess, content):
#if build succeed, notify the whole group, otherwise notify the core members
group = config["group"].split(",")
core = config["core"].split(",")
group_addrs = []
for member in group:
group_addrs.append(member+'@'+config["addr_domain"])
core_addrs = []
for member in core:
core_addrs.append(member+'@'+config["addr_domain"])
fromaddr = "Autobuild<autobuild@"+config["addr_domain"]+">"
toaddrs = []
subject = ""
if(bldSuccess==True):
toaddrs = group_addrs
subject = "Autobuild succeed!"
else:
toaddrs = core_addrs
subject = "Autobuild failed!"
print "send mail to: %s"%str(toaddrs)
print "subject: %s"%subject
print "content:/n%s"%content
msg = ("From: %s/r/nTo: %s/r/nSubject: %s/r/n/r/n"% (fromaddr, ", ".join(toaddrs), subject))
msg += content
server = smtplib.SMTP(config["smtp_server"])
server.sendmail(fromaddr, toaddrs, msg)
server.quit()
def nextBuild(cur_build):
ind_bld = cur_build.rindex(".")
bld_num = int(cur_build[ind_bld+1:])
next = cur_build[0:ind_bld+1]+str(bld_num+1)
#print "current: %s, next: %s"%(cur_build,next)
return next
def compBuildNum(build1,build2):
bld1_list = build1.split(".")
bld2_list = build2.split(".")
len1 = len(bld1_list)
len2 = len(bld2_list)
min_len = min(len1,len2)
for i in range(min_len):
num1 = int(bld1_list[i])
num2 = int(bld2_list[i])
if(num1>num2):
return 1
elif(num1<num2):
return -1
if(len1==len2):
return 0
elif(len1>len2):
return 1
else:
return -1
def readCfg(cfgFilePath):
cfgFile = open(cfgFilePath)
cfgDict = dict()
prog = re.compile(r"(?P<item>[a-zA-Z0-9_]+)[ /t]+(?P<value>/S+)")
while(1):
line = cfgFile.readline()
if(len(line)==0): #The end of file
break
m = prog.match(line)
if(m==None):
continue
cfgDict[m.group("item")]=m.group("value")
cfgFile.close()
return cfgDict
class SvnComponent:
def __init__(self,name,path):
self.name=name
self.cur_revision=0
self.head_revision=0
self.paths=path.split(",")
self.logs=dict()
self.work_dir=os.path.abspath(".")
self.contentChanged=True
def updateToHead(self):
print "svn cleanup, update and log %s..."%self.name
client = pysvn.Client()
end_ver = pysvn.Revision(pysvn.opt_revision_kind.number, self.cur_revision)
self.logs.clear()
for compPath in self.paths:
client.cleanup(compPath)
updatedTo = client.update(compPath)
self.head_revision=updatedTo[0].number
log_messages = client.log(compPath, revision_end=end_ver)
for log_entry in log_messages:
#print log_entry.revision.number, log_entry.message
if(log_entry.revision.number!=self.cur_revision): #remove log of cur_verision
self.logs[log_entry.revision.number] = log_entry.message
if(len(self.logs)>0):
self.contentChanged=True
else:
self.contentChanged=False
def build(self):
raise NotImplementedError
def updateRootfs(self,rootfs_path,ipcamType):
raise NotImplementedError
class SvnRepo:
def __init__(self,repoName,revision):
self.name=repoName
self.cur_revision=revision
self.components=[]
self.head_revision=0
self.logs=dict()
def appendComponent(self,compName,path):
comp=None
#TODO: 组件名称与类的对应关系。项目细节，请酌情调整
if(compName=="comp1"):
comp=Comp1(path)
elif(compName=="comp2"):
comp=Comp2(path)
elif(compName=="comp3"):
comp=Comp3(path)
else:
print "The component named '%s' isn't supported!"%compName
return -1
if(comp==None):
print "Error! comp is null!"
return -1
comp.cur_revision=self.cur_revision
self.components.append(comp)
return 0
def updateToHead(self):
for comp in self.components:
comp.updateToHead()
return self.__verifyConsistency()
def __verifyConsistency(self):
self.head_revision=0
self.logs.clear()
for comp in self.components:
if(self.head_revision==0):
self.head_revision=comp.head_revision
if(self.head_revision!=comp.head_revision):
return -1
self.logs.update(comp.logs)
return 0
def build(self):
for comp in self.components:
if(comp.contentChanged==False):
continue
iRet=comp.build()
if(iRet<0):
return -1
return 0
def updateRootfs(self,rootfsPath,ipcamType):
for comp in self.components:
if(comp.contentChanged==False):
continue
comp.updateRootfs(rootfsPath,ipcamType)
class Project:
def __init__(self,name,confFile):
self.name=name
self.repos=[]
self.notes=""
self.logsCnt=0
self.nextBldNums=dict()
self.confFile=os.path.abspath(confFile)
self.config=readCfg(confFile)
repoNameList=self.config["repositories"].split(",")
for repoName in repoNameList:
strCur=self.config[repoName+"_revision"]
repo=SvnRepo(repoName,int(strCur))
compNameList=self.config[repoName].split(",")
for compName in compNameList:
repo.appendComponent(compName,self.config[compName])
self.repos.append(repo)
def updateToHead(self):
for repo in self.repos:
iRet=repo.updateToHead()
if(iRet<0):
return -1
self.logsCnt,self.notes = self.__getLogs()
return 0
def build(self):
for repo in self.repos:
iRet=repo.build()
if(iRet<0):
return -1
return 0
def __updateRootfs(self,rootfsPath,ipcamType):
for repo in self.repos:
repo.updateRootfs(rootfsPath,ipcamType)
def __getLogs(self):
notes=""
index=1
for repo in self.repos:
ver_list = repo.logs.keys()
if(len(ver_list)==0):
continue
ver_list.sort()
for ver in ver_list:
message = repo.logs[ver]
if(message[-1]=="/n"):
note = "%d.(%s%d) "%(index,repo.name,ver) + message
else:
note = "%d.(%s%d) "%(index,repo.name,ver) + message + "/n"
notes += note
index +=1
return index-1,notes
# Assume: webs, webcam, lrweb are ready in saveDir
def makeRamdisk(self, ipcamType):
#TODO: 所有组件build完成后调用本方法制作Linux根文件系统。项目细节，请酌情实现。
return 0
def smokeTest(self):
print "smoke test..."
#TODO: unit test here
return 0
class Comp1(SvnComponent):
def __init__(self,path):
SvnComponent.__init__(self,"comp1",path)
def __compileComp1(self, ipcamType, saveDir):
print "compile comp1 for %s..."%ipcamType
target = os.path.abspath(os.path.join(saveDir,"comp1_%s"%ipcamType))
clearCmd = "scons -c"
compileCmd = "scons"
#TODO: 不同目标板可能编译选项不一样。项目细节，请酌情实现。
try:
os.remove(target) #Attenrion: shutil.rmtree() isn't suitble for file
except OSError, e: # [Errno 2] No such file or directory
pass
print "enter "+self.paths[0]
os.chdir(self.paths[0])
print clearCmd
clearResult = commands.getoutput(clearCmd)
print clearResult
print compileCmd
(status,output) = commands.getstatusoutput(compileCmd)
print output
if (status != 0):
return -1
#TODO: 本处假设编译出的executable名字为comp1。请酌情修改。
shutil.copyfile("comp1", target)
return 0
def build(self):
#TODO: 本处假设目标板型号有两个，分别为ipcamType1和ipcamType2。请酌情修改。
iRet1=self.__compileComp1("ipcamType1",self.work_dir)
iRet2=self.__compileComp1("ipcamType2",self.work_dir)
if(iRet1<0 or iRet2<0):
return -1
return 0
def updateRootfs(self,rootfsPath,ipcamType):
print "update webcam in rootfs..."
comp1 = os.path.join(self.work_dir,"comp1_%s"%ipcamType)
#TODO: 本处假设comp1被安装到根文件系统的/etc/comp1。请酌情修改。
target = os.path.abspath(os.path.join(rootfsPath,"etc/comp1"))
shutil.copyfile(webcam,target)
class Comp2(SvnComponent):
def __init__(self,path):
SvnComponent.__init__(self,"comp2",path)
def __compileComp2(self, ipcamType, saveDir):
print "compile comp2 for %s..."%ipcamType
target = os.path.abspath(os.path.join(saveDir,"comp2_%s"%ipcamType))
clearCmd = "make clean"
compileCmd = "make all; make strip"
try:
os.remove(target) #Attenrion: shutil.rmtree() isn't suitble for file
except OSError, e: # [Errno 2] No such file or directory
pass
print "enter "+self.paths[0]
os.chdir(self.paths[0])
#TODO: 不同目标板可能编译选项不一样。项目细节，请酌情实现。
print clearCmd
clearResult = commands.getoutput(clearCmd)
print clearResult
print compileCmd
(status,output) = commands.getstatusoutput(compileCmd)
print output
if (status != 0):
return -1
#TODO: 本处假设编译出的executable名字为comp2。请酌情修改。
shutil.copyfile("comp2", target)
return 0
def __prepareWebpages(self,saveDir):
target = os.path.abspath(os.path.join(saveDir,"webpages"))
source = os.path.abspath(os.path.join(self.paths[0],"webpages"))
print "prepare web pages..."
shutil.rmtree(target, True) # [Errno 2] No such file or directory
shutil.copytree(source, target)
rmSvnDirsCmd = "find . -name .svn | xargs rm -fr"
print "enter "+target
os.chdir(target)
(status,output) = commands.getstatusoutput(rmSvnDirsCmd)
print output
def build(self):
#TODO: 本处假设目标板型号有两个，分别为ipcamType1和ipcamType2。请酌情修改。
iRet1=self.__compileComp2("ipcamType1",self.work_dir)
iRet2=self.__compileComp2("ipcamType2",self.work_dir)
if(iRet1!=0 or iRet2!=0):
return -1
self.__prepareWebpages(self.work_dir)
return 0
def updateRootfs(self,rootfsPath,ipcamType):
print "update comp2 in rootfs..."
comp2 = os.path.abspath(os.path.join(self.work_dir,"comp2_%s"%ipcamType))
webpages = os.path.abspath(os.path.join(self.work_dir,"webpages"))
#TODO: 本处假设comp2和webpages分别被安装到根文件系统的/etc/comp2和/var/www/html。请酌情修改。
target_comp2 = os.path.join(rootfsPath,"etc/comp2")
target_webpages = os.path.join(rootfsPath,"var/www/html")
shutil.copyfile(webs,target_comp2)
shutil.rmtree(target_webpages,True)
shutil.copytree(webpages, target_webpages)
class Comp3(SvnComponent):
def __init__(self,path):
#Python require that super class's __init__ must be called!
SvnComponent.__init__(self,"cfgfiles",path)
def __prepareComp3(self, ipcamType):
target = os.path.abspath(os.path.join(self.work_dir,"comp3_%s"%ipcamType))
source = ""
#TODO: 本处假设目标板型号有两个，分别为ipcamType1和ipcamType2。
#还假设comp3针对不同的目标板使用不同svn工作拷贝。请酌情修改。
if(ipcamType=="ipcamType1"):
source=self.paths[0]
else:
source=self.paths[1]
print "prepare comp3 for %s..."%ipcamType
shutil.rmtree(target, True) # [Errno 2] No such file or directory
shutil.copytree(source, target)
rmSvnDirsCmd = "find . -name .svn | xargs rm -fr"
print "enter "+target
os.chdir(target)
(status,output) = commands.getstatusoutput(rmSvnDirsCmd)
print output
def build(self):
self.__prepareComp3("ipcamType1")
self.__prepareComp3("ipcamType2")
return 0
def updateRootfs(self,rootfsPath,ipcamType):
print "update comp3 in rootfs..."
cfgfiles = os.path.abspath(os.path.join(self.work_dir,"comp3_%s"%ipcamType))
#TODO: 本处假设comp3被安装到根文件系统的/etc/conf。请酌情修改。
target = os.path.join(rootfsPath, "etc/conf")
shutil.rmtree(target,True)
shutil.copytree(cfgfiles,target)
def main():
print >>sys.stdout, __doc__
parser = OptionParser()
parser.add_option("-q", "--quiet", dest="sendmail", action="store_false",
default=True, help="don't send mails when build finished.")
parser.add_option("-f", "--file", dest="cfgfile", default="build.conf",
help="use another config file rather than './build.conf'")
(options, args) = parser.parse_args()
cfgFileFullPath = os.path.abspath(options.cfgfile) #Becareful to use relative path
print "use config file "+cfgFileFullPath
project = Project("IPCam",cfgFileFullPath)
if(options.sendmail==False):
project.config["send_mail"] = "no"
#Svn update components and fetch logs
iRet = project.updateToHead()
if(iRet<0):
print "components in the same repository are with different head revisions!"
sys.exit(-1)
if(project.logsCnt==0):
print "the whole project doesn't changed since last build!"
for repo in project.repos:
print repo.name+":(HEAD)"+str(repo.head_revision)+"<=>"+project.config["%s_revision"%repo.name]
sys.exit(0)
#Build components
iRet=project.build()
if(iRet!=0 and project.config["send_mail"].lower()=="yes"):
sendmails(project.config, False, "Compile failed!")
#Make ramdisk, iterate all types of ipcam
#TODO: 本处假设目标板型号有两个，分别为ipcamType1和ipcamType2。
for ipcamType in ["ipcamType1","ipcamType2"]:
project.makeRamdisk(ipcamType)
#Update build.conf
project.updateCfgfile()
#Smoke test
res = project.smokeTest()
if(res!=0):
msg = "smoke test failed!/r/n"
if(project.config["send_mail"].lower()=="yes"):
sendmails(project.config, False, msg + project.notes)
sys.exit(2)
#At last, notify the team about new versions
if(project.config["send_mail"].lower()=="yes"):
#TODO: 本处假设目标板型号有两个，分别为ipcamType1和ipcamType2。
msg = "note about new build (ipcamType1: %s, ipcamType2: %s): /r/n"%(project.nextBldNums["ipcamType1"],project.nextBldNums["ipcamType2"])
sendmails(project.config, True, msg+project.notes)
print "congratulations!"
if __name__ == '__main__':
main()
文件build.conf是以上脚本使用的配置文件。每次构建版本文件成功之后，都会在最后记录这次构建出的版本号以及各个SVN库的修订号。内容如下（请酌情修改）：
#This is the config file for autobuild.py.
#The regexp of a config line is r"(?P<item>[a-zA-Z0-9_]+)[ /t]+(?P<value>/S+)"
#If a item occurs multi times, the effective one is the latest one.
#Don't use any redundant space character in value part of a config line.
#Don't use relative anywhere, because current directory may change during running!
#Don't write comments at the end of a config line.
#repositories is a comma separated list
repositories HQ,SH
HQ comp1,comp2
#components' absoulte path
comp1 /home/kenny/svn_HQ/ipcam/comp1
comp2 /home/kenny/svn_HQ/ipcam/comp2
SH comp3
#Attention: a comma separated list, the first is for ipcamType1, the second is for ipcamType2
comp3 /home/kenny/svn_SH/ipcam/trunk/ipcamType1/conf,/home/kenny/svn_SH/ipcam/trunk/ipcamType2/conf
ipcamType1_ramdisk /home/kenny/ramdisk_ipcamType1
ipcamType2_ramdisk /home/kenny/ramdisk_ipcamType2
#Email addr prefix list is a comma separated list.
group frank_wang,kenny,alex,steason
core frank_wang,kenny
addr_domain example.com
#The SMTP server must be configed without authentication
smtp_server localhost
send_mail yes
HQ_revision 2340
SH_revision 228
ipcamType1_version 1.1.14.23
ipcamType2_version 1.1.13.24
HQ_revision 2357
SH_revision 244
ipcamType1_version 1.1.14.24
ipcamType2_version 1.1.13.25
实现Nightly Build途径之一是利用已有的工具。
“对于Java开发项目，支持Night Build的软件很多，开源的选择有AntHill OS和CruiseControl，后者是大名鼎鼎的ThoughtWorks的作品，配置起来比较麻烦。而AntHill是一个Java Web应用，拥有友好的定制界面，是2003年Jolt大奖的Productivity Award获得者，比CruiseControl更容易入手。”
这方面的工具中我推荐Buildbot。BuildBot基于Python，配置简洁明了。BuildBot是分布式结构，master监控若干slaves（监视状态，下发命令，搜集结果），各个slave按照指示执行update/compile/test并返回结果。Python和其他许多项目利用它做持续构建。例如Python的持续构建报告在http://www.python.org/dev/buildbot/；Wireshark的构建报告http://buildbot.wireshark.org/trunk/。
以上工具都只是持续构建的框架，版本构建的每个步骤（编写Ant脚本、Python脚本或者其他）仍然需要自己实现。
途径之二就是完全自己动手：脚本+crond服务。脚本可以选择Python/Perl/TCL/Shell等。
按照我的理解：持续构建分为如下几个部分：
(1)调度策略：定时构建，还是每次提交就构建？分布式构建还是集中式构建？
BuildBot支持定时构建/每次提交构建等多种，支持分布式构建。而自己动手则仅支持定时构建，仅支持集中式构建。
(2)版本构建步骤：一个版本是由多个组件构成的。这些组件之间的依赖关系如何？版本特定的操作有哪些（如更新Changlog，计算Build number等）？
BuildBot不关心组件依赖关系和版本特定操作；AntHill之类的工具直接支持这些概念。而自己动手则需要自己处理这些东西。
(3)组件构建步骤：从何处获取最新版本源码？如何编译？如何放置到版本文件中去？
一个组件通常由一个Makefile（传统C/C++项目），Ant脚本（Java项目）或Scons脚本（我偏爱的，Make替代品）来管理，所以构建步骤非常简单。所有的工具都将单个组件的构建任务交给用户编写脚本片段去做。
我偏爱自己编写脚本做Nightly Build，因为它简单、灵活、够用。我不认为大项目非得用个Nightly Build工具。大项目只是组件多一些，但每个组件的构建步骤仍然是极其简单的一个make命令搞定的事情！如果可移植性不是很重要，就不需要分布式构建。如果组件之间的依赖关系很复杂，只能说明项目规划差劲。
中小型项目需要Nightly Build吗？项目在开发阶段没有做Nightly Build，现在进入维护阶段了，还需要费劲实现这个吗？按照我的经验，回答都是肯定的。我维护过一个中等规模的针对嵌入式Linux的项目IPCam，大的组件只有两个，其他还有配置文件、初始化脚本等。这个项目的版本文件内容包括两部分：发布说明和目标板上用的Linux根文件系统。这个项目针对不同的目标平台分别构建版本文件。这个项目的版本文件构建步骤比较简单，就是从各个SVN服务器上取出最新版本，编译或者删除.svn临时目录后，替换掉上一版本文件中对应的组件，更新releasenotes，最后打包。就这么简单的步骤，我和同事们往往花15分钟时间小心翼翼地做，但仍然不时地出错，例如忘记了删除.svn临时目录、为某目标平台编译组件时忘记了修改Makefile中相应的选项、不同目标平台的组件弄混了、忘记在发布说明中添加新版本的说明...在我花两三天时间为这个项目编写500多行的Python脚本之后，整个项目组在制作版本文件方面花费的心思从此降为0。
其实实现NightlyBuild机制并不难。编写一个脚本，将其作为一个crond任务于每晚运行即可。在众多脚本语言中，我偏爱Python。Python语法简明并且许多事情都能做得更好。Perl和Shell稍微复杂点的语法就难以理解了（理解是维护的前提），远不如Python好用。
以下介绍我在上述项目中的Nightly Build脚本（涉及项目细节的，为保密起见作了适当修改）。它实现了以下功能：
(1)每晚构建（作为一个crond任务即可）。
(2)无论构建成功与否都有邮件通知。
(3)没有修改的组件不作编译和更新动作。
(4)记录了构建轨迹，即每次构建出的文件版本号与各个SVN库的修订号的对应关系。方便下次构建，也方便事后的问题跟踪。
(5)把自上次构建以来所有的SVN提交动作的log作为本次构建的changelog，放入发布说明中。如果是构建的版本文件需要正式发布（如发布给客户），则changelog需要人工调整一下。
需要说明的是，它并不是一个框架或者库。要修改成框架或者库要花心思，但它目前对我来说够用了。所以如果其他项目要移植这个脚本的话，需要改动一些东西：(1)为每个组件定义一个类实现其特有的构建步骤、更新版本文件中相应组件的操作。(2)修改版本构建步骤，也就是Project::makeRamdisk()方法。(3)修改配置文件build.conf以反映特定的项目-SVN库-组件的3层关系。另外事先要安装svn客户端的Python绑定(http://pysvn.tigris.org/) 。对于CVS库，可用http://pycvs.sourceforge.net/提供的pycvs。
在下面的脚本中，有3个组件Comp1/Comp2/Comp3。Comp1和Comp2位于同一SVN库中，且都需要编译；Comp3位于另一SVN库中，且不需要编译（例如配置文件、网页之类的）。
文件autobuild.py内容如下：
#! /usr/bin/env python
"""Autobuild: the main tool of nightly build mechanism adopted by my IPCam team.
Requirements:
1. You should has the privilege to do as follow: mount and unmount at shell via sudo, delete and write in ramdisk rootfs when updating it.
2. The config file (normally './build.conf', but can specified by option '-f') is good.
3. Require some common tools on GUN/Linux: Python, GNU environment(make/tar/gzip...), sudo.
4. Require ipcam cross platform compile tool (arm-linux-gcc) installed.
5. Require scons (http://www.scons.org/) installed.
6. Require pysvn (http://pysvn.tigris.org/) installed.
Main features:
1. Project, repositories and components are organized into a 3-levels tree.
The config file 'build.conf' records the tree.
2. A component may have several directories according to target board type.
3. Assure consistency of the componnets's revision in the same repository.
4. Ignore unchanged components when buiding or updating rootfs for the whole project.
5. Stop building immediatly and send email to the core members when building
any component faild.
6. Record repositories' revisions and ramdisks' versions in 'build.conf' and update them
whenever a new version made out.
7. Compute next build number automaticly.
8. Send mail to the whole team when a new version made out. The content of the mail:
ramdisks' version, repositories' revisions and the logs since the last version."""
import sys
import os
import os.path
import re
import pysvn
import time
import commands
import shutil
import smtplib
from optparse import OptionParser
def isRoot():
euid = os.geteuid()
return(euid==0)
def sendmails(config, bldSuccess, content):
#if build succeed, notify the whole group, otherwise notify the core members
group = config["group"].split(",")
core = config["core"].split(",")
group_addrs = []
for member in group:
group_addrs.append(member+'@'+config["addr_domain"])
core_addrs = []
for member in core:
core_addrs.append(member+'@'+config["addr_domain"])
fromaddr = "Autobuild<autobuild@"+config["addr_domain"]+">"
toaddrs = []
subject = ""
if(bldSuccess==True):
toaddrs = group_addrs
subject = "Autobuild succeed!"
else:
toaddrs = core_addrs
subject = "Autobuild failed!"
print "send mail to: %s"%str(toaddrs)
print "subject: %s"%subject
print "content:/n%s"%content
msg = ("From: %s/r/nTo: %s/r/nSubject: %s/r/n/r/n"% (fromaddr, ", ".join(toaddrs), subject))
msg += content
server = smtplib.SMTP(config["smtp_server"])
server.sendmail(fromaddr, toaddrs, msg)
server.quit()
def nextBuild(cur_build):
ind_bld = cur_build.rindex(".")
bld_num = int(cur_build[ind_bld+1:])
next = cur_build[0:ind_bld+1]+str(bld_num+1)
#print "current: %s, next: %s"%(cur_build,next)
return next
def compBuildNum(build1,build2):
bld1_list = build1.split(".")
bld2_list = build2.split(".")
len1 = len(bld1_list)
len2 = len(bld2_list)
min_len = min(len1,len2)
for i in range(min_len):
num1 = int(bld1_list[i])
num2 = int(bld2_list[i])
if(num1>num2):
return 1
elif(num1<num2):
return -1
if(len1==len2):
return 0
elif(len1>len2):
return 1
else:
return -1
def readCfg(cfgFilePath):
cfgFile = open(cfgFilePath)
cfgDict = dict()
prog = re.compile(r"(?P<item>[a-zA-Z0-9_]+)[ /t]+(?P<value>/S+)")
while(1):
line = cfgFile.readline()
if(len(line)==0): #The end of file
break
m = prog.match(line)
if(m==None):
continue
cfgDict[m.group("item")]=m.group("value")
cfgFile.close()
return cfgDict
class SvnComponent:
def __init__(self,name,path):
self.name=name
self.cur_revision=0
self.head_revision=0
self.paths=path.split(",")
self.logs=dict()
self.work_dir=os.path.abspath(".")
self.contentChanged=True
def updateToHead(self):
print "svn cleanup, update and log %s..."%self.name
client = pysvn.Client()
end_ver = pysvn.Revision(pysvn.opt_revision_kind.number, self.cur_revision)
self.logs.clear()
for compPath in self.paths:
client.cleanup(compPath)
updatedTo = client.update(compPath)
self.head_revision=updatedTo[0].number
log_messages = client.log(compPath, revision_end=end_ver)
for log_entry in log_messages:
#print log_entry.revision.number, log_entry.message
if(log_entry.revision.number!=self.cur_revision): #remove log of cur_verision
self.logs[log_entry.revision.number] = log_entry.message
if(len(self.logs)>0):
self.contentChanged=True
else:
self.contentChanged=False
def build(self):
raise NotImplementedError
def updateRootfs(self,rootfs_path,ipcamType):
raise NotImplementedError
class SvnRepo:
def __init__(self,repoName,revision):
self.name=repoName
self.cur_revision=revision
self.components=[]
self.head_revision=0
self.logs=dict()
def appendComponent(self,compName,path):
comp=None
#TODO: 组件名称与类的对应关系。项目细节，请酌情调整
if(compName=="comp1"):
comp=Comp1(path)
elif(compName=="comp2"):
comp=Comp2(path)
elif(compName=="comp3"):
comp=Comp3(path)
else:
print "The component named '%s' isn't supported!"%compName
return -1
if(comp==None):
print "Error! comp is null!"
return -1
comp.cur_revision=self.cur_revision
self.components.append(comp)
return 0
def updateToHead(self):
for comp in self.components:
comp.updateToHead()
return self.__verifyConsistency()
def __verifyConsistency(self):
self.head_revision=0
self.logs.clear()
for comp in self.components:
if(self.head_revision==0):
self.head_revision=comp.head_revision
if(self.head_revision!=comp.head_revision):
return -1
self.logs.update(comp.logs)
return 0
def build(self):
for comp in self.components:
if(comp.contentChanged==False):
continue
iRet=comp.build()
if(iRet<0):
return -1
return 0
def updateRootfs(self,rootfsPath,ipcamType):
for comp in self.components:
if(comp.contentChanged==False):
continue
comp.updateRootfs(rootfsPath,ipcamType)
class Project:
def __init__(self,name,confFile):
self.name=name
self.repos=[]
self.notes=""
self.logsCnt=0
self.nextBldNums=dict()
self.confFile=os.path.abspath(confFile)
self.config=readCfg(confFile)
repoNameList=self.config["repositories"].split(",")
for repoName in repoNameList:
strCur=self.config[repoName+"_revision"]
repo=SvnRepo(repoName,int(strCur))
compNameList=self.config[repoName].split(",")
for compName in compNameList:
repo.appendComponent(compName,self.config[compName])
self.repos.append(repo)
def updateToHead(self):
for repo in self.repos:
iRet=repo.updateToHead()
if(iRet<0):
return -1
self.logsCnt,self.notes = self.__getLogs()
return 0
def build(self):
for repo in self.repos:
iRet=repo.build()
if(iRet<0):
return -1
return 0
def __updateRootfs(self,rootfsPath,ipcamType):
for repo in self.repos:
repo.updateRootfs(rootfsPath,ipcamType)
def __getLogs(self):
notes=""
index=1
for repo in self.repos:
ver_list = repo.logs.keys()
if(len(ver_list)==0):
continue
ver_list.sort()
for ver in ver_list:
message = repo.logs[ver]
if(message[-1]=="/n"):
note = "%d.(%s%d) "%(index,repo.name,ver) + message
else:
note = "%d.(%s%d) "%(index,repo.name,ver) + message + "/n"
notes += note
index +=1
return index-1,notes
# Assume: webs, webcam, lrweb are ready in saveDir
def makeRamdisk(self, ipcamType):
#TODO: 所有组件build完成后调用本方法制作Linux根文件系统。项目细节，请酌情实现。
return 0
def smokeTest(self):
print "smoke test..."
#TODO: unit test here
return 0
class Comp1(SvnComponent):
def __init__(self,path):
SvnComponent.__init__(self,"comp1",path)
def __compileComp1(self, ipcamType, saveDir):
print "compile comp1 for %s..."%ipcamType
target = os.path.abspath(os.path.join(saveDir,"comp1_%s"%ipcamType))
clearCmd = "scons -c"
compileCmd = "scons"
#TODO: 不同目标板可能编译选项不一样。项目细节，请酌情实现。
try:
os.remove(target) #Attenrion: shutil.rmtree() isn't suitble for file
except OSError, e: # [Errno 2] No such file or directory
pass
print "enter "+self.paths[0]
os.chdir(self.paths[0])
print clearCmd
clearResult = commands.getoutput(clearCmd)
print clearResult
print compileCmd
(status,output) = commands.getstatusoutput(compileCmd)
print output
if (status != 0):
return -1
#TODO: 本处假设编译出的executable名字为comp1。请酌情修改。
shutil.copyfile("comp1", target)
return 0
def build(self):
#TODO: 本处假设目标板型号有两个，分别为ipcamType1和ipcamType2。请酌情修改。
iRet1=self.__compileComp1("ipcamType1",self.work_dir)
iRet2=self.__compileComp1("ipcamType2",self.work_dir)
if(iRet1<0 or iRet2<0):
return -1
return 0
def updateRootfs(self,rootfsPath,ipcamType):
print "update webcam in rootfs..."
comp1 = os.path.join(self.work_dir,"comp1_%s"%ipcamType)
#TODO: 本处假设comp1被安装到根文件系统的/etc/comp1。请酌情修改。
target = os.path.abspath(os.path.join(rootfsPath,"etc/comp1"))
shutil.copyfile(webcam,target)
class Comp2(SvnComponent):
def __init__(self,path):
SvnComponent.__init__(self,"comp2",path)
def __compileComp2(self, ipcamType, saveDir):
print "compile comp2 for %s..."%ipcamType
target = os.path.abspath(os.path.join(saveDir,"comp2_%s"%ipcamType))
clearCmd = "make clean"
compileCmd = "make all; make strip"
try:
os.remove(target) #Attenrion: shutil.rmtree() isn't suitble for file
except OSError, e: # [Errno 2] No such file or directory
pass
print "enter "+self.paths[0]
os.chdir(self.paths[0])
#TODO: 不同目标板可能编译选项不一样。项目细节，请酌情实现。
print clearCmd
clearResult = commands.getoutput(clearCmd)
print clearResult
print compileCmd
(status,output) = commands.getstatusoutput(compileCmd)
print output
if (status != 0):
return -1
#TODO: 本处假设编译出的executable名字为comp2。请酌情修改。
shutil.copyfile("comp2", target)
return 0
def __prepareWebpages(self,saveDir):
target = os.path.abspath(os.path.join(saveDir,"webpages"))
source = os.path.abspath(os.path.join(self.paths[0],"webpages"))
print "prepare web pages..."
shutil.rmtree(target, True) # [Errno 2] No such file or directory
shutil.copytree(source, target)
rmSvnDirsCmd = "find . -name .svn | xargs rm -fr"
print "enter "+target
os.chdir(target)
(status,output) = commands.getstatusoutput(rmSvnDirsCmd)
print output
def build(self):
#TODO: 本处假设目标板型号有两个，分别为ipcamType1和ipcamType2。请酌情修改。
iRet1=self.__compileComp2("ipcamType1",self.work_dir)
iRet2=self.__compileComp2("ipcamType2",self.work_dir)
if(iRet1!=0 or iRet2!=0):
return -1
self.__prepareWebpages(self.work_dir)
return 0
def updateRootfs(self,rootfsPath,ipcamType):
print "update comp2 in rootfs..."
comp2 = os.path.abspath(os.path.join(self.work_dir,"comp2_%s"%ipcamType))
webpages = os.path.abspath(os.path.join(self.work_dir,"webpages"))
#TODO: 本处假设comp2和webpages分别被安装到根文件系统的/etc/comp2和/var/www/html。请酌情修改。
target_comp2 = os.path.join(rootfsPath,"etc/comp2")
target_webpages = os.path.join(rootfsPath,"var/www/html")
shutil.copyfile(webs,target_comp2)
shutil.rmtree(target_webpages,True)
shutil.copytree(webpages, target_webpages)
class Comp3(SvnComponent):
def __init__(self,path):
#Python require that super class's __init__ must be called!
SvnComponent.__init__(self,"cfgfiles",path)
def __prepareComp3(self, ipcamType):
target = os.path.abspath(os.path.join(self.work_dir,"comp3_%s"%ipcamType))
source = ""
#TODO: 本处假设目标板型号有两个，分别为ipcamType1和ipcamType2。
#还假设comp3针对不同的目标板使用不同svn工作拷贝。请酌情修改。
if(ipcamType=="ipcamType1"):
source=self.paths[0]
else:
source=self.paths[1]
print "prepare comp3 for %s..."%ipcamType
shutil.rmtree(target, True) # [Errno 2] No such file or directory
shutil.copytree(source, target)
rmSvnDirsCmd = "find . -name .svn | xargs rm -fr"
print "enter "+target
os.chdir(target)
(status,output) = commands.getstatusoutput(rmSvnDirsCmd)
print output
def build(self):
self.__prepareComp3("ipcamType1")
self.__prepareComp3("ipcamType2")
return 0
def updateRootfs(self,rootfsPath,ipcamType):
print "update comp3 in rootfs..."
cfgfiles = os.path.abspath(os.path.join(self.work_dir,"comp3_%s"%ipcamType))
#TODO: 本处假设comp3被安装到根文件系统的/etc/conf。请酌情修改。
target = os.path.join(rootfsPath, "etc/conf")
shutil.rmtree(target,True)
shutil.copytree(cfgfiles,target)
def main():
print >>sys.stdout, __doc__
parser = OptionParser()
parser.add_option("-q", "--quiet", dest="sendmail", action="store_false",
default=True, help="don't send mails when build finished.")
parser.add_option("-f", "--file", dest="cfgfile", default="build.conf",
help="use another config file rather than './build.conf'")
(options, args) = parser.parse_args()
cfgFileFullPath = os.path.abspath(options.cfgfile) #Becareful to use relative path
print "use config file "+cfgFileFullPath
project = Project("IPCam",cfgFileFullPath)
if(options.sendmail==False):
project.config["send_mail"] = "no"
#Svn update components and fetch logs
iRet = project.updateToHead()
if(iRet<0):
print "components in the same repository are with different head revisions!"
sys.exit(-1)
if(project.logsCnt==0):
print "the whole project doesn't changed since last build!"
for repo in project.repos:
print repo.name+":(HEAD)"+str(repo.head_revision)+"<=>"+project.config["%s_revision"%repo.name]
sys.exit(0)
#Build components
iRet=project.build()
if(iRet!=0 and project.config["send_mail"].lower()=="yes"):
sendmails(project.config, False, "Compile failed!")
#Make ramdisk, iterate all types of ipcam
#TODO: 本处假设目标板型号有两个，分别为ipcamType1和ipcamType2。
for ipcamType in ["ipcamType1","ipcamType2"]:
project.makeRamdisk(ipcamType)
#Update build.conf
project.updateCfgfile()
#Smoke test
res = project.smokeTest()
if(res!=0):
msg = "smoke test failed!/r/n"
if(project.config["send_mail"].lower()=="yes"):
sendmails(project.config, False, msg + project.notes)
sys.exit(2)
#At last, notify the team about new versions
if(project.config["send_mail"].lower()=="yes"):
#TODO: 本处假设目标板型号有两个，分别为ipcamType1和ipcamType2。
msg = "note about new build (ipcamType1: %s, ipcamType2: %s): /r/n"%(project.nextBldNums["ipcamType1"],project.nextBldNums["ipcamType2"])
sendmails(project.config, True, msg+project.notes)
print "congratulations!"
if __name__ == '__main__':
main()
文件build.conf是以上脚本使用的配置文件。每次构建版本文件成功之后，都会在最后记录这次构建出的版本号以及各个SVN库的修订号。内容如下（请酌情修改）：
#This is the config file for autobuild.py.
#The regexp of a config line is r"(?P<item>[a-zA-Z0-9_]+)[ /t]+(?P<value>/S+)"
#If a item occurs multi times, the effective one is the latest one.
#Don't use any redundant space character in value part of a config line.
#Don't use relative anywhere, because current directory may change during running!
#Don't write comments at the end of a config line.
#repositories is a comma separated list
repositories HQ,SH
HQ comp1,comp2
#components' absoulte path
comp1 /home/kenny/svn_HQ/ipcam/comp1
comp2 /home/kenny/svn_HQ/ipcam/comp2
SH comp3
#Attention: a comma separated list, the first is for ipcamType1, the second is for ipcamType2
comp3 /home/kenny/svn_SH/ipcam/trunk/ipcamType1/conf,/home/kenny/svn_SH/ipcam/trunk/ipcamType2/conf
ipcamType1_ramdisk /home/kenny/ramdisk_ipcamType1
ipcamType2_ramdisk /home/kenny/ramdisk_ipcamType2
#Email addr prefix list is a comma separated list.
group frank_wang,kenny,alex,steason
core frank_wang,kenny
addr_domain example.com
#The SMTP server must be configed without authentication
smtp_server localhost
send_mail yes
HQ_revision 2340
SH_revision 228
ipcamType1_version 1.1.14.23
ipcamType2_version 1.1.13.24
HQ_revision 2357
SH_revision 244
ipcamType1_version 1.1.14.24
ipcamType2_version 1.1.13.25
扫一扫
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
