零、系列目录
写这套文章的时候，不会完全按照目录的顺序一篇一篇写， 大家可以到目录中直接找到对应的章节进行查看。
一、背景
在运行爬虫的时候，有些任务难免执行超时，针对超时的任务，我们采取的策略就是直接关闭这个任务的进程，避免任务阻塞。
二、脚本代码
先上干货，有需要的同学可以直接拿走了。
完整代码位置：点击跳转。
#!/bin/bash
function kill_timeout_process()
{
pid=$1
max_time=$2
start_time=$(cat /proc/$pid/stat | cut -d" " -f22)
if [[ "$start_time" == "" ]]
then
echo "[$pid] not exists."
return 0
fi
user_hz=$(getconf CLK_TCK)
sys_uptime=$(cat /proc/uptime | cut -d" " -f1)
runs_time=$(( ${sys_uptime%.*} - $start_time/$user_hz ))
echo "[$pid] runs for $runs_time seconds."
if [ "$runs_time" -ge "$max_time" ]
then
echo "[$pid] killed."
kill -9 $pid
fi
return 1
}
process_name="scrapy"
timeout=7200
for pid in `pgrep $process_name`
do
kill_timeout_process $pid $timeout
done
三、解析
如果有同学对上面的脚本原理感兴趣，咱们继续。
执行步骤
1、获取待比较的进程
2、获取进程的运行时间
3、判断运行时间是否超过设置时间
4、关闭超时进行
代码
针对上述步骤，我们一个一个来讲。
获取进程 pid
想要获取进程id，ps 可以、pgrep 也可以。
下面例子中，scrapy 是我想要查询的进程包含的命令。
ps 方式：
ps -ef | grep "scrapy" | grep -v grep | awk '{print $2}'
运行结果：
[root@docker]# ps -ef | grep "scrapy" | grep -v grep | awk '{print $2}'
5369
5371
14628
14634
16514
16516
17383
17384
pgrep 方式：
pgrep scrapy
运行结果：
[root@docker]# pgrep scrapy
5371
14634
17384
17857
27193
27587
27592
27599
27604
可以看到，如果只是单纯的需要 pid 的话，pgrep 的方式更简单，我们就选它。
获取进程的运行时间
获取进程运行时间的方式是：
进程运行时间 = 系统当前运行时间 - 进程启动时的系统时间
如何获取系统当前运行时间？
系统当前的运行时间信息在 /proc/uptime 这个文件里面，例如：
[root@docker]# cat /proc/uptime
5047164.56 34461345.27
可以看到，这个文件里面有两个数。第一个就表示系统到目前为止的运行时间，单位秒。
如何获取进程启动时的系统时间？
这里可以从 /proc/[pid]/stat 这个文件下手，这里保存着 pid 对应的进程的所有运行数据。 具体可见下文：
/proc/[pid]/stat
Status information about the process. This is used by ps(1).
It is defined in the kernel source file fs/proc/array.c.
The fields, in order, with their proper scanf(3) format speci‐
fiers, are listed below. Whether or not certain of these
fields display valid information is governed by a ptrace
access mode PTRACE_MODE_READ_FSCREDS | PTRACE_MODE_NOAUDIT
check (refer to ptrace(2)). If the check denies access, then
the field value is displayed as 0. The affected fields are
indicated with the marking [PT].
(1) pid %d
The process ID.
(2) comm %s
The filename of the executable, in parentheses.
This is visible whether or not the executable is
swapped out.
(3) state %c
One of the following characters, indicating process
state:
R Running
S Sleeping in an interruptible wait
D Waiting in uninterruptible disk sleep
Z Zombie
T Stopped (on a signal) or (before Linux 2.6.33)
trace stopped
t Tracing stop (Linux 2.6.33 onward)
W Paging (only before Linux 2.6.0)
X Dead (from Linux 2.6.0 onward)
x Dead (Linux 2.6.33 to 3.13 only)
K Wakekill (Linux 2.6.33 to 3.13 only)
W Waking (Linux 2.6.33 to 3.13 only)
P Parked (Linux 3.9 to 3.13 only)
(4) ppid %d
The PID of the parent of this process.
(5) pgrp %d
The process group ID of the process.
(6) session %d
The session ID of the process.
(7) tty_nr %d
The controlling terminal of the process. (The minor
device number is contained in the combination of
bits 31 to 20 and 7 to 0; the major device number is
in bits 15 to 8.)
(8) tpgid %d
The ID of the foreground process group of the con‐
trolling terminal of the process.
(9) flags %u
The kernel flags word of the process. For bit mean‐
ings, see the PF_* defines in the Linux kernel
source file include/linux/sched.h. Details depend
on the kernel version.
The format for this field was %lu before Linux 2.6.
(10) minflt %lu
The number of minor faults the process has made
which have not required loading a memory page from
disk.
(11) cminflt %lu
The number of minor faults that the process's
waited-for children have made.
(12) majflt %lu
The number of major faults the process has made
which have required loading a memory page from disk.
(13) cmajflt %lu
The number of major faults that the process's
waited-for children have made.
(14) utime %lu
Amount of time that this process has been scheduled
in user mode, measured in clock ticks (divide by
sysconf(_SC_CLK_TCK)). This includes guest time,
guest_time (time spent running a virtual CPU, see
below), so that applications that are not aware of
the guest time field do not lose that time from
their calculations.
(15) stime %lu
Amount of time that this process has been scheduled
in kernel mode, measured in clock ticks (divide by
sysconf(_SC_CLK_TCK)).
(16) cutime %ld
Amount of time that this process's waited-for chil‐
dren have been scheduled in user mode, measured in
clock ticks (divide by sysconf(_SC_CLK_TCK)). (See
also times(2).) This includes guest time,
cguest_time (time spent running a virtual CPU, see
below).
(17) cstime %ld
Amount of time that this process's waited-for chil‐
dren have been scheduled in kernel mode, measured in
clock ticks (divide by sysconf(_SC_CLK_TCK)).
(18) priority %ld
(Explanation for Linux 2.6) For processes running a
real-time scheduling policy (policy below; see
sched_setscheduler(2)), this is the negated schedul‐
ing priority, minus one; that is, a number in the
range -2 to -100, corresponding to real-time priori‐
ties 1 to 99. For processes running under a non-
real-time scheduling policy, this is the raw nice
value (setpriority(2)) as represented in the kernel.
The kernel stores nice values as numbers in the
range 0 (high) to 39 (low), corresponding to the
user-visible nice range of -20 to 19.
Before Linux 2.6, this was a scaled value based on
the scheduler weighting given to this process.
(19) nice %ld
The nice value (see setpriority(2)), a value in the
range 19 (low priority) to -20 (high priority).
(20) num_threads %ld
Number of threads in this process (since Linux 2.6).
Before kernel 2.6, this field was hard coded to 0 as
a placeholder for an earlier removed field.
(21) itrealvalue %ld
The time in jiffies before the next SIGALRM is sent
to the process due to an interval timer. Since ker‐
nel 2.6.17, this field is no longer maintained, and
is hard coded as 0.
(22) starttime %llu
The time the process started after system boot. In
kernels before Linux 2.6, this value was expressed
in jiffies. Since Linux 2.6, the value is expressed
in clock ticks (divide by sysconf(_SC_CLK_TCK)).
The format for this field was %lu before Linux 2.6.
(23) vsize %lu
Virtual memory size in bytes.
(24) rss %ld
Resident Set Size: number of pages the process has
in real memory. This is just the pages which count
toward text, data, or stack space. This does not
include pages which have not been demand-loaded in,
or which are swapped out.
(25) rsslim %lu
Current soft limit in bytes on the rss of the
process; see the description of RLIMIT_RSS in
getrlimit(2).
(26) startcode %lu [PT]
The address above which program text can run.
(27) endcode %lu [PT]
The address below which program text can run.
(28) startstack %lu [PT]
The address of the start (i.e., bottom) of the
stack.
(29) kstkesp %lu [PT]
The current value of ESP (stack pointer), as found
in the kernel stack page for the process.
(30) kstkeip %lu [PT]
The current EIP (instruction pointer).
(31) signal %lu
The bitmap of pending signals, displayed as a deci‐
mal number. Obsolete, because it does not provide
information on real-time signals; use
/proc/[pid]/status instead.
(32) blocked %lu
The bitmap of blocked signals, displayed as a deci‐
mal number. Obsolete, because it does not provide
information on real-time signals; use
/proc/[pid]/status instead.
(33) sigignore %lu
The bitmap of ignored signals, displayed as a deci‐
mal number. Obsolete, because it does not provide
information on real-time signals; use
/proc/[pid]/status instead.
(34) sigcatch %lu
The bitmap of caught signals, displayed as a decimal
number. Obsolete, because it does not provide
information on real-time signals; use
/proc/[pid]/status instead.
(35) wchan %lu [PT]
This is the "channel" in which the process is wait‐
ing. It is the address of a location in the kernel
where the process is sleeping. The corresponding
symbolic name can be found in /proc/[pid]/wchan.
(36) nswap %lu
Number of pages swapped (not maintained).
(37) cnswap %lu
Cumulative nswap for child processes (not main‐
tained).
(38) exit_signal %d (since Linux 2.1.22)
Signal to be sent to parent when we die.
(39) processor %d (since Linux 2.2.8)
CPU number last executed on.
(40) rt_priority %u (since Linux 2.5.19)
Real-time scheduling priority, a number in the range
1 to 99 for processes scheduled under a real-time
policy, or 0, for non-real-time processes (see
sched_setscheduler(2)).
(41) policy %u (since Linux 2.5.19)
Scheduling policy (see sched_setscheduler(2)).
Decode using the SCHED_* constants in linux/sched.h.
The format for this field was %lu before Linux
2.6.22.
(42) delayacct_blkio_ticks %llu (since Linux 2.6.18)
Aggregated block I/O delays, measured in clock ticks
(centiseconds).
(43) guest_time %lu (since Linux 2.6.24)
Guest time of the process (time spent running a vir‐
tual CPU for a guest operating system), measured in
clock ticks (divide by sysconf(_SC_CLK_TCK)).
(44) cguest_time %ld (since Linux 2.6.24)
Guest time of the process's children, measured in
clock ticks (divide by sysconf(_SC_CLK_TCK)).
(45) start_data %lu (since Linux 3.3) [PT]
Address above which program initialized and unini‐
tialized (BSS) data are placed.
(46) end_data %lu (since Linux 3.3) [PT]
Address below which program initialized and unini‐
tialized (BSS) data are placed.
(47) start_brk %lu (since Linux 3.3) [PT]
Address above which program heap can be expanded
with brk(2).
(48) arg_start %lu (since Linux 3.5) [PT]
Address above which program command-line arguments
(argv) are placed.
(49) arg_end %lu (since Linux 3.5) [PT]
Address below program command-line arguments (argv)
are placed.
(50) env_start %lu (since Linux 3.5) [PT]
Address above which program environment is placed.
(51) env_end %lu (since Linux 3.5) [PT]
Address below which program environment is placed.
(52) exit_code %d (since Linux 3.5) [PT]
The thread's exit status in the form reported by
waitpid(2).
可以看到，我们上文是通过 cat /proc/$pid/stat | cut -d" " -f22 来获取的该进程的 starttime —— 启动时间。这里需要注意，这里所说的启动时间，是指该进程启动时，系统当时的中断次数，这个次数除以系统的时钟频率（CLK_TCK）就得到了该进程的开始时间（单位秒）。
至于剩下的比较和判断，还有关闭进程，这些比较基础的内容就不一一讲解了。
四、总结
关闭超时进程的操作思路比较简单，主要是需要对 Linux 系统有一些了解，知道相关的内容（比如系统运行时间、进程运行时间等）保存在哪里，其他的都是基本操作了。
祝大家变的更强。
扫一扫
4192
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
