docker进程启动失败处理案例分享

docker进程启动失败处理

背景

• 有同学反馈在启动docker的时候遇到了如下问题：docker启动报错

  [root@wuxianfeng ~]# systemctl start docker
  Job for docker.service failed because the control process exited with error code. See "systemctl status docker.service" and "journalctl -xe" for details.
  
  

• 他说看了status，也一头雾水

  [root@wuxianfeng ~]# systemctl status docker
  ● docker.service - Docker Application Container Engine
     Loaded: loaded (/usr/lib/systemd/system/docker.service; enabled; vendor preset: disabled)
     Active: failed (Result: start-limit) since Mon 2022-09-05 10:49:47 CST; 33s ago
       Docs: https://docs.docker.com
    Process: 1519 ExecStart=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock (code=exited, status=1/FAILURE)
   Main PID: 1519 (code=exited, status=1/FAILURE)
  
  Sep 05 10:49:45 wuxianfeng systemd[1]: docker.service: main process exited, code=exited, status=1/FAILURE
  Sep 05 10:49:45 wuxianfeng systemd[1]: Failed to start Docker Application Container Engine.
  Sep 05 10:49:45 wuxianfeng systemd[1]: Unit docker.service entered failed state.
  Sep 05 10:49:45 wuxianfeng systemd[1]: docker.service failed.
  Sep 05 10:49:47 wuxianfeng systemd[1]: docker.service holdoff time over, scheduling restart.
  Sep 05 10:49:47 wuxianfeng systemd[1]: Stopped Docker Application Container Engine.
  Sep 05 10:49:47 wuxianfeng systemd[1]: start request repeated too quickly for docker.service
  Sep 05 10:49:47 wuxianfeng systemd[1]: Failed to start Docker Application Container Engine.
  Sep 05 10:49:47 wuxianfeng systemd[1]: Unit docker.service entered failed state.
  Sep 05 10:49:47 wuxianfeng systemd[1]: docker.service failed.
  

解决

• 出现无法启动docker的可能其实是非常多的，他依赖的东西发生了异常就有可能无法启动（比如selinux，比如权限，比如启动所需的配置文件等）
• 没有说固定的哪个方法就能解决所有的问题，日志是分析问题的重要手段。
• systemctl start docker的提示信息告诉了你两个处理问题的方向

systemctl status docker 查看docker的状态

• 很多问题在这里会输出线索，本例没有。也难怪他一头雾水。

journalctl -xe 查看系统日志

• 我摘录部分

-- Unit docker.service has begun starting up.
Sep 05 10:49:42 wuxianfeng dockerd[1515]: unable to configure the Docker daemon with file /etc/docker/daemon.json: invalid character '\n' in string lite
Sep 05 10:49:42 wuxianfeng systemd[1]: docker.service: main process exited, code=exited, status=1/FAILURE
Sep 05 10:49:42 wuxianfeng systemd[1]: Failed to start Docker Application Container Engine.
-- Subject: Unit docker.service has failed
-- Defined-By: systemd

• 本例就可以用这个信息来解决，注意上面的第二行清晰的提示了无法用你提供的/etc/docker/daemon.json来配置dockerd，因为有invalid character ‘\n’ in string lite。

• 查看下daemon.json的内容

  [root@wuxianfeng ~]# cat /etc/docker/daemon.json
  {
    "debug: true
  }
  

• 果然错了，少了个引号，加上后启动ok。

• 可以看到这个问题的处理其实非常简单，只需根据提示，稍微有点json的基础你就知道了。

引申一：docker其他调试方法

取自 《docker入门与实践》

开启 Debug 模式

在 dockerd 配置文件 daemon.json（默认位于 /etc/docker/）中添加

{
  "debug": true
}

重启守护进程。

$ sudo kill -SIGHUP $(pidof dockerd)

此时 dockerd 会在日志中输入更多信息供分析。

检查内核日志

$ sudo dmesg |grep dockerd
$ sudo dmesg |grep runc

Docker 不响应时处理

可以杀死 dockerd 进程查看其堆栈调用情况。

$ sudo kill -SIGUSR1 $(pidof dockerd)

重置 Docker 本地数据

注意，本操作会移除所有的 Docker 本地数据，包括镜像和容器等。

$ sudo rm -rf /var/lib/docker

docker logs 查看容器的日志

• 常用于解决容器启动、运行异常的分析

引申二：journalctl

• journalctl 是查询系统日志的一个工具，功能非常强大

• -x:查看指定目录

  -x, --catalog
     Augment log lines with explanation texts from the message catalog. This will add explanatory help texts to log messages in the output where this is available. These short help texts will explain the context of an error or log event, possible solutions, as well as pointers to support forums, developer documentation, and any other relevant manuals. Note that help texts are not available for all messages, but only for selected ones. For more information on the message catalog, please refer to the Message Catalog Developer  Documentation[4]
  

• -e:从尾部看日志

  -e, --pager-end
      Immediately jump to the end of the journal inside the implied pager tool. This implies -n1000 to guarantee that the pager will not buffer logs of unbounded size. This may be overridden with an explicit -n with some other numeric value while -nall will disable this cap. Note  that this option is only supported for the less(1) pager.
  

• 不做展开，可以参考：https://zhuanlan.zhihu.com/p/410995772