The Linux server failed to restart since /etc/fstab error

本文详细介绍了如何通过单用户模式登录服务器、调整分区挂载、编辑fstab文件去除错误行、验证挂载状态,并最终完成服务器重启的过程。此流程对于服务器维护和故障排除至关重要。

1. Login the server with single user mode

2. remount the / partion

# mount -o remount,rw /

3. Edit the /etc/fstab and remove comment the error line

4. verify with "mount -a" command

5. reboot the server

[zcc@localhost mysql]$ sudo systemctl status mysql ● mysql.service - MySQL Community Server Loaded: loaded (/etc/systemd/system/mysql.service; enabled; vendor preset: disabled) Active: failed (Result: exit-code) since 四 2025-07-24 00:22:15 PDT; 5s ago Process: 99005 ExecStart=/usr/local/mysql/bin/mysqld_safe --datadir=/var/lib/mysql (code=exited, status=1/FAILURE) 7月 24 00:22:14 localhost.localdomain systemd[1]: Starting MySQL Community Server... 7月 24 00:22:15 localhost.localdomain mysqld_safe[99005]: /usr/local/mysql/bin/mysqld_safe:行653: /var/…不够 7月 24 00:22:15 localhost.localdomain systemd[1]: mysql.service: control process exited, code=exited s...s=1 7月 24 00:22:15 localhost.localdomain systemd[1]: Failed to start MySQL Community Server. 7月 24 00:22:15 localhost.localdomain systemd[1]: Unit mysql.service entered failed state. 7月 24 00:22:15 localhost.localdomain systemd[1]: mysql.service failed. Hint: Some lines were ellipsized, use -l to show in full. [zcc@localhost mysql]$ mysql -u root -p Enter password: ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/tmp/mysql.sock' (2) [zcc@localhost mysql]$ sudo systemctl status mysql ● mysql.service - MySQL Community Server Loaded: loaded (/etc/systemd/system/mysql.service; enabled; vendor preset: disabled) Active: failed (Result: exit-code) since 四 2025-07-24 00:22:15 PDT; 3min 45s ago Process: 99005 ExecStart=/usr/local/mysql/bin/mysqld_safe --datadir=/var/lib/mysql (code=exited, status=1/FAILURE) 7月 24 00:22:14 localhost.localdomain systemd[1]: Starting MySQL Community Server... 7月 24 00:22:15 localhost.localdomain mysqld_safe[99005]: /usr/local/mysql/bin/mysqld_safe:行653: /va…不够 7月 24 00:22:15 localhost.localdomain systemd[1]: mysql.service: control process exited, code=exited...s=1 7月 24 00:22:15 localhost.localdomain systemd[1]: Failed to start MySQL Community Server. 7月 24 00:22:15 localhost.localdomain systemd[1]: Unit mysql.service entered failed state. 7月 24 00:22:15 localhost.localdomain systemd[1]: mysql.service failed. 这个怎么解决
07-25
[root@node4 ~]# kubeadm reset --force [preflight] Running pre-flight checks W0923 10:54:42.908233 30672 removeetcdmember.go:79] [reset] No kubeadm config, using etcd pod spec to get data directory [reset] No etcd config found. Assuming external etcd [reset] Please, manually reset etcd to prevent further issues [reset] Stopping the kubelet service [reset] Unmounting mounted directories in "/var/lib/kubelet" [reset] Deleting contents of config directories: [/etc/kubernetes/manifests /etc/kubernetes/pki] [reset] Deleting files: [/etc/kubernetes/admin.conf /etc/kubernetes/kubelet.conf /etc/kubernetes/bootstrap-kubelet.conf /etc/kubernetes/controller-manager.conf /etc/kubernetes/scheduler.conf] [reset] Deleting contents of stateful directories: [/var/lib/kubelet /var/lib/dockershim /var/run/kubernetes /var/lib/cni] The reset process does not clean CNI configuration. To do so, you must remove /etc/cni/net.d The reset process does not reset or clean up iptables rules or IPVS tables. If you wish to reset iptables, you must do so manually by using the "iptables" command. If your cluster was setup to utilize IPVS, run ipvsadm --clear (or similar) to reset your system's IPVS tables. The reset process does not clean your kubeconfig files and you must remove them manually. Please, check the contents of the $HOME/.kube/config file. [root@node4 ~]# rm -rf /etc/kubernetes /var/lib/kubelet [root@node4 ~]# systemctl stop kubelet [root@node4 ~]# [root@node4 ~]# [root@node4 ~]# [root@node4 ~]# systemctl status docker ● docker.service - Docker Application Container Engine Loaded: loaded (/etc/systemd/system/docker.service; enabled; vendor preset: disabled) Drop-In: /etc/systemd/system/docker.service.d └─docker-dns.conf, docker-options.conf Active: active (running) since 二 2025-09-23 10:36:29 CST; 18min ago Docs: http://docs.docker.com Main PID: 16450 (dockerd) Tasks: 22 Memory: 54.9M CGroup: /system.slice/docker.service └─16450 /usr/bin/dockerd --iptables=false --exec-opt native.cgroupdriver=systemd --data-root=/var/lib/docker --log-opt max-size=50m --log-opt max-file=5 --dns 10.233.0.3 --dns 202.101.224.68 --dns 202.101.224.69 --dns-search default.svc.cluster.local --dns... 9月 23 10:36:28 node4 dockerd[16450]: time="2025-09-23T10:36:28.714625957+08:00" level=info msg="ClientConn switching balancer to \"pick_first\"" module=grpc 9月 23 10:36:28 node4 dockerd[16450]: time="2025-09-23T10:36:28.755430797+08:00" level=info msg="[graphdriver] using prior storage driver: overlay2" 9月 23 10:36:28 node4 dockerd[16450]: time="2025-09-23T10:36:28.954323711+08:00" level=info msg="Loading containers: start." 9月 23 10:36:29 node4 dockerd[16450]: time="2025-09-23T10:36:29.206033565+08:00" level=info msg="Fixing inconsistent endpoint_cnt for network none. Expected=0, Actual=12" 9月 23 10:36:29 node4 dockerd[16450]: time="2025-09-23T10:36:29.337817459+08:00" level=info msg="Default bridge (docker0) is assigned with an IP address 172.17.0.0/16. Daemon option --bip can be used to set a preferred IP address" 9月 23 10:36:29 node4 dockerd[16450]: time="2025-09-23T10:36:29.495200799+08:00" level=info msg="Loading containers: done." 9月 23 10:36:29 node4 dockerd[16450]: time="2025-09-23T10:36:29.800389989+08:00" level=info msg="Docker daemon" commit=a89b842 graphdriver(s)=overlay2 version=20.10.17 9月 23 10:36:29 node4 dockerd[16450]: time="2025-09-23T10:36:29.800482890+08:00" level=info msg="Daemon has completed initialization" 9月 23 10:36:29 node4 systemd[1]: Started Docker Application Container Engine. 9月 23 10:36:29 node4 dockerd[16450]: time="2025-09-23T10:36:29.952645573+08:00" level=info msg="API listen on /var/run/docker.sock" [root@node4 ~]# systemctl restart docker containerd [root@node4 ~]# systemctl status docker ● docker.service - Docker Application Container Engine Loaded: loaded (/etc/systemd/system/docker.service; enabled; vendor preset: disabled) Drop-In: /etc/systemd/system/docker.service.d └─docker-dns.conf, docker-options.conf Active: active (running) since 二 2025-09-23 10:55:37 CST; 5s ago Docs: http://docs.docker.com Main PID: 30797 (dockerd) Tasks: 21 Memory: 48.8M CGroup: /system.slice/docker.service └─30797 /usr/bin/dockerd --iptables=false --exec-opt native.cgroupdriver=systemd --data-root=/var/lib/docker --log-opt max-size=50m --log-opt max-file=5 --dns 10.233.0.3 --dns 202.101.224.68 --dns 202.101.224.69 --dns-search default.svc.cluster.local --dns... 9月 23 10:55:36 node4 dockerd[30797]: time="2025-09-23T10:55:36.636353414+08:00" level=info msg="ccResolverWrapper: sending update to cc: {[{unix:///run/containerd/containerd.sock <nil> 0 <nil>}] <nil> <nil>}" module=grpc 9月 23 10:55:36 node4 dockerd[30797]: time="2025-09-23T10:55:36.636357438+08:00" level=info msg="ClientConn switching balancer to \"pick_first\"" module=grpc 9月 23 10:55:36 node4 dockerd[30797]: time="2025-09-23T10:55:36.687672605+08:00" level=info msg="[graphdriver] using prior storage driver: overlay2" 9月 23 10:55:36 node4 dockerd[30797]: time="2025-09-23T10:55:36.892456214+08:00" level=info msg="Loading containers: start." 9月 23 10:55:36 node4 dockerd[30797]: time="2025-09-23T10:55:36.998523887+08:00" level=info msg="Default bridge (docker0) is assigned with an IP address 172.17.0.0/16. Daemon option --bip can be used to set a preferred IP address" 9月 23 10:55:37 node4 dockerd[30797]: time="2025-09-23T10:55:37.065087090+08:00" level=info msg="Loading containers: done." 9月 23 10:55:37 node4 dockerd[30797]: time="2025-09-23T10:55:37.107876319+08:00" level=info msg="Docker daemon" commit=a89b842 graphdriver(s)=overlay2 version=20.10.17 9月 23 10:55:37 node4 dockerd[30797]: time="2025-09-23T10:55:37.107948567+08:00" level=info msg="Daemon has completed initialization" 9月 23 10:55:37 node4 systemd[1]: Started Docker Application Container Engine. 9月 23 10:55:37 node4 dockerd[30797]: time="2025-09-23T10:55:37.253910996+08:00" level=info msg="API listen on /var/run/docker.sock" [root@node4 ~]# [root@node4 ~]# [root@node4 ~]# [root@node4 ~]# [root@node4 ~]# [root@node4 ~]# [root@node4 ~]# [root@node4 ~]# [root@node4 ~]# [root@node4 ~]# [root@node4 ~]# [root@node4 ~]# [root@node4 ~]# [root@node4 ~]# [root@node4 ~]# [root@node4 ~]# [root@node4 ~]# [root@node4 ~]# [root@node4 ~]# [root@node4 ~]# [root@node4 ~]# [root@node4 ~]# [root@node4 ~]# ss -tulnp | grep 10248 [root@node4 ~]# [root@node4 ~]# [root@node4 ~]# [root@node4 ~]# kubeadm join 192.168.1.12:6443 \ > --token n0higb.w0qcokvgv3521iqh \ > --discovery-token-ca-cert-hash sha256:f2a8a9e2b9ec276e0a3ead1fa393ad40a2657105d48b12ff21fc305cc680f87d \ > --v=5 I0923 10:56:30.172295 30964 join.go:395] [preflight] found NodeName empty; using OS hostname as NodeName I0923 10:56:30.172656 30964 initconfiguration.go:115] detected and using CRI socket: /var/run/dockershim.sock [preflight] Running pre-flight checks I0923 10:56:30.172719 30964 preflight.go:90] [preflight] Running general checks I0923 10:56:30.172974 30964 checks.go:250] validating the existence and emptiness of directory /etc/kubernetes/manifests I0923 10:56:30.172986 30964 checks.go:287] validating the existence of file /etc/kubernetes/kubelet.conf I0923 10:56:30.172993 30964 checks.go:287] validating the existence of file /etc/kubernetes/bootstrap-kubelet.conf I0923 10:56:30.173001 30964 checks.go:103] validating the container runtime I0923 10:56:30.239105 30964 checks.go:129] validating if the "docker" service is enabled and active I0923 10:56:30.292863 30964 checks.go:336] validating the contents of file /proc/sys/net/bridge/bridge-nf-call-iptables I0923 10:56:30.292993 30964 checks.go:336] validating the contents of file /proc/sys/net/ipv4/ip_forward I0923 10:56:30.293010 30964 checks.go:654] validating whether swap is enabled or not I0923 10:56:30.293175 30964 checks.go:377] validating the presence of executable conntrack I0923 10:56:30.293327 30964 checks.go:377] validating the presence of executable ip I0923 10:56:30.293414 30964 checks.go:377] validating the presence of executable iptables I0923 10:56:30.293575 30964 checks.go:377] validating the presence of executable mount I0923 10:56:30.293650 30964 checks.go:377] validating the presence of executable nsenter I0923 10:56:30.293660 30964 checks.go:377] validating the presence of executable ebtables I0923 10:56:30.293668 30964 checks.go:377] validating the presence of executable ethtool I0923 10:56:30.293676 30964 checks.go:377] validating the presence of executable socat I0923 10:56:30.293684 30964 checks.go:377] validating the presence of executable tc I0923 10:56:30.293692 30964 checks.go:377] validating the presence of executable touch I0923 10:56:30.293702 30964 checks.go:525] running all checks I0923 10:56:30.345060 30964 checks.go:408] checking whether the given node name is valid and reachable using net.LookupHost I0923 10:56:30.345344 30964 checks.go:623] validating kubelet version I0923 10:56:30.374493 30964 checks.go:129] validating if the "kubelet" service is enabled and active I0923 10:56:30.380422 30964 checks.go:202] validating availability of port 10250 I0923 10:56:30.380703 30964 checks.go:287] validating the existence of file /etc/kubernetes/pki/ca.crt I0923 10:56:30.380715 30964 checks.go:437] validating if the connectivity type is via proxy or direct I0923 10:56:30.380739 30964 join.go:465] [preflight] Discovering cluster-info I0923 10:56:30.380760 30964 token.go:78] [discovery] Created cluster-info discovery client, requesting info from "192.168.1.12:6443" I0923 10:56:30.406267 30964 token.go:116] [discovery] Requesting info from "192.168.1.12:6443" again to validate TLS against the pinned public key I0923 10:56:30.425034 30964 token.go:133] [discovery] Cluster info signature and contents are valid and TLS certificate validates against pinned roots, will use API Server "192.168.1.12:6443" I0923 10:56:30.425084 30964 discovery.go:51] [discovery] Using provided TLSBootstrapToken as authentication credentials for the join process I0923 10:56:30.425102 30964 join.go:479] [preflight] Fetching init configuration I0923 10:56:30.425118 30964 join.go:517] [preflight] Retrieving KubeConfig objects [preflight] Reading configuration from the cluster... [preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml' I0923 10:56:30.468845 30964 interface.go:431] Looking for default routes with IPv4 addresses I0923 10:56:30.468870 30964 interface.go:436] Default route transits interface "p1p1" I0923 10:56:30.470524 30964 interface.go:208] Interface p1p1 is up I0923 10:56:30.470947 30964 interface.go:256] Interface "p1p1" has 2 addresses :[192.168.1.16/22 fe80::4e10:d5ff:fe52:a682/64]. I0923 10:56:30.470993 30964 interface.go:223] Checking addr 192.168.1.16/22. I0923 10:56:30.471022 30964 interface.go:230] IP found 192.168.1.16 I0923 10:56:30.471063 30964 interface.go:262] Found valid IPv4 address 192.168.1.16 for interface "p1p1". I0923 10:56:30.471126 30964 interface.go:442] Found active IP 192.168.1.16 W0923 10:56:30.471181 30964 utils.go:69] The recommended value for "clusterDNS" in "KubeletConfiguration" is: [10.233.0.10]; the provided value is: [169.254.25.10] I0923 10:56:30.475012 30964 preflight.go:101] [preflight] Running configuration dependant checks I0923 10:56:30.475059 30964 controlplaneprepare.go:211] [download-certs] Skipping certs download I0923 10:56:30.475084 30964 kubelet.go:110] [kubelet-start] writing bootstrap kubelet config file at /etc/kubernetes/bootstrap-kubelet.conf I0923 10:56:30.476555 30964 kubelet.go:118] [kubelet-start] writing CA certificate at /etc/kubernetes/pki/ca.crt I0923 10:56:30.477445 30964 kubelet.go:139] [kubelet-start] Checking for an existing Node in the cluster with name "node4" and status "Ready" I0923 10:56:30.488211 30964 kubelet.go:153] [kubelet-start] Stopping the kubelet [kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml" [kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env" [kubelet-start] Starting the kubelet [kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap... [kubelet-check] Initial timeout of 40s passed. [kubelet-check] It seems like the kubelet isn't running or healthy. [kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get "http://localhost:10248/healthz": dial tcp 127.0.0.1:10248: connect: connection refused. [kubelet-check] It seems like the kubelet isn't running or healthy. [kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get "http://localhost:10248/healthz": dial tcp 127.0.0.1:10248: connect: connection refused. [kubelet-check] It seems like the kubelet isn't running or healthy. [kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get "http://localhost:10248/healthz": dial tcp 127.0.0.1:10248: connect: connection refused. [kubelet-check] It seems like the kubelet isn't running or healthy. [kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get "http://localhost:10248/healthz": dial tcp 127.0.0.1:10248: connect: connection refused. ^C [root@node4 ~]# mkdir -p /etc/kubernetes/pki [root@node4 ~]# ls /etc/kubernetes/pki/ca.crt.bak ls: 无法访问/etc/kubernetes/pki/ca.crt.bak: 没有那个文件或目录 [root@node4 ~]#
最新发布
09-24
<think>好的,我现在需要帮助用户解决AppArmor服务启动失败的问题,控制进程退出时出现错误代码的情况。首先,我需要回顾用户之前的提问和我的回答,确保信息一致。 用户之前遇到了AppArmor服务未激活的问题,当时我提供了检查服务状态、启动服务、安装组件、检查内核支持、修复配置文件和强制重载等步骤。现在用户的问题更具体了,是服务启动失败并出现错误代码,这可能涉及到更深层次的原因,比如内核模块问题、配置文件错误、权限不足或者日志中的具体错误信息。 首先,我需要考虑用户可能遇到的常见错误。根据之前的回答,用户可能需要进一步检查日志,使用`journalctl -xeu apparmor`来查看详细错误信息。这可能包括配置文件语法错误、权限问题或者与其他安全模块的冲突。 其次,用户提到控制进程退出时出现错误代码,这通常意味着systemd在启动服务时遇到了问题。我需要引导用户检查服务状态,查看具体的错误信息,比如是否有“Failed to start AppArmor filesystem”之类的提示。这可能涉及到内核参数或挂载点的问题,比如`/sys/kernel/security`是否存在或正确挂载。 接下来,可能需要检查AppArmor的内核支持是否启用,用户之前已经执行过`grep CONFIG_SECURITY_APPARMOR`命令,但需要确认结果是否为`y`。如果没有启用,可能需要重新编译内核或加载模块。此外,挂载安全文件系统也是一个关键点,用户可能需要执行`mount | grep securityfs`来确认,如果没有挂载,需要手动挂载或修改fstab。 另外,配置文件的有效性也是一个常见问题。用户需要检查`/etc/apparmor.d/`下的配置文件是否有语法错误,使用`apparmor_parser`进行测试。如果有损坏的配置,可以暂时移动或禁用这些文件,再尝试启动服务。 还有,SELinux和AppArmor的冲突可能性,用户可能需要卸载SELinux相关包,确保两者不会同时运行。此外,软件包损坏也可能导致问题,重新安装AppArmor相关包可能有效。 最后,用户可能需要更新系统或内核,尤其是当问题出现在系统更新之后,可能存在兼容性问题。同时,检查系统日志中的其他错误,比如磁盘空间不足或内存问题,也可能影响服务启动。 在生成解决方案时,需要分步骤详细说明,确保用户能逐步排查问题。同时,相关的命令和日志分析是关键,帮助用户定位具体错误原因。最后,生成的相关问题需要围绕用户可能遇到的后续问题展开,比如如何调试、处理解析错误或清理缓存配置。</think>### AppArmor服务启动失败解决方案 当AppArmor服务启动失败并伴随控制进程错误代码时,通常由内核模块加载异常、配置文件冲突或权限问题导致。以下是系统性排查步骤: --- #### 1. **查看详细错误日志** 通过systemd日志定位具体错误原因: ```bash sudo journalctl -xeu apparmor --since "5 minutes ago" ``` 重点关注以下关键词: - `Failed to load AppArmor profiles`(配置文件加载失败) - `securityfs not mounted`(安全文件系统未挂载) - `Operation not permitted`(权限不足)[^1] --- #### 2. **检查内核模块状态** 确认AppArmor内核模块已加载: ```bash lsmod | grep apparmor # 若无输出表示未加载 sudo modprobe apparmor # 手动加载模块 ``` 若加载失败,需检查内核配置: ```bash grep CONFIG_SECURITY_APPARMOR /boot/config-$(uname -r) ``` 若返回`CONFIG_SECURITY_APPARMOR=y`,表示内核支持;否则需重新编译内核或更换内核版本。 --- #### 3. **挂载安全文件系统** AppArmor依赖`securityfs`文件系统: ```bash mount | grep securityfs # 应显示挂载在/sys/kernel/security ``` 若未挂载,手动挂载并永久生效: ```bash sudo mount -t securityfs securityfs /sys/kernel/security echo "securityfs /sys/kernel/security securityfs defaults 0 0" | sudo tee -a /etc/fstab ``` --- #### 4. **验证配置文件有效性** 检查所有profile是否存在语法错误: ```bash sudo apparmor_parser -T -r /etc/apparmor.d/ # 测试并重载配置 ``` 若报错具体profile(如`/etc/apparmor.d/usr.bin.nginx`): ```bash sudo mv /etc/apparmor.d/usr.bin.nginx /etc/apparmor.d/disable/ # 临时禁用 sudo systemctl restart apparmor ``` --- #### 5. **处理常见错误场景** - **SELinux冲突**:完全禁用SELinux ```bash sudo apt purge selinux-* && reboot ``` - **软件包损坏**:重装核心组件 ```bash sudo apt install --reinstall apparmor apparmor-utils ``` - **缓存文件异常**:清理缓存 ```bash sudo rm -rf /etc/apparmor.d/cache/* ``` --- #### 6. **更新系统与内核** 若问题出现在系统升级后: ```bash sudo apt update && sudo apt full-upgrade -y sudo reboot ``` --- ### 典型错误代码对应方案 | 错误代码 | 解决方案 | |---------|----------| | **137** | 内存不足导致崩溃,检查`dmesg`日志 | | **255** | 内核模块未加载,执行`modprobe apparmor` | | **203** | 配置文件权限错误,运行`sudo chmod 644 /etc/apparmor.d/*` | --- ### 相关问题 1. 如何调试AppArmor的访问拒绝日志? 2. AppArmor配置文件解析错误如何快速定位? 3. 如何清除AppArmor的缓存配置?
评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值