GitLab 升级 502 Error executing action `run` 'ruby_block[resource: /var/opt/gitlab/git-data/repositor

欢迎访问我的个人博客网站:http://www.yanmin99.com/

GitLab 升级 502问题的解决 Error executing action run on resource ‘ruby_block[directory resource: /var/opt/gitlab/git-data/repositories]’
  • 1、异常信息

    resolving cookbooks for run list: ["gitlab"]
    Synchronizing Cookbooks:
    - package (0.0.0)
    - runit (0.14.2)
    - gitlab (0.0.1)
    Installing Cookbook Gems:
    Compiling Cookbooks...
    Recipe: gitlab::default
    * directory[/etc/gitlab] action create (up to date)
    /sbin/init: Unbekannte Option »--version«
    -.mount                                                                                                 loaded active mounted   /
    Converging 284 resources
    * directory[/etc/gitlab] action create (up to date)
    * directory[Create /var/opt/gitlab] action create (up to date)
    * directory[/opt/gitlab/embedded/etc] action create (up to date)
    * template[/opt/gitlab/embedded/etc/gitconfig] action create (up to date)
    Recipe: gitlab::web-server
    * group[gitlab-www] action create (up to date)
    * user[gitlab-www] action create (up to date)
    Recipe: gitlab::users
    * directory[/var/opt/gitlab] action create (up to date)
    * group[git] action create (up to date)
    * user[git] action create (up to date)
    * template[/var/opt/gitlab/.gitconfig] action create (up to date)
    Recipe: gitlab::gitlab-shell
    * ruby_block[directory resource: /var/opt/gitlab/git-data] action run
    
      ================================================================================
      Error executing action `run` on resource 'ruby_block[directory resource: /var/opt/gitlab/git-data]'
      ================================================================================
    
    Mixlib::ShellOut::ShellCommandFailed
    
  • 2、解决方案

    chmod 2770 /var/opt/gitlab/git-data/repositories
* file[/var/log/gitlab/gitlab-kas/current] action touch (skipped due to only_if) Recipe: gitlab::database_migrations * ruby_block[check remote PG version] action nothing (skipped due to action :nothing) * rails_migration[gitlab-rails] action run[2025-07-14T15:04:58+08:00] WARN: gitlab-rails does not have a log_group or default logdir mode defined. Setting to 0700. * bash_hide_env[migrate gitlab-rails database] action run ================================================================================ Error executing action `run` on resource 'bash_hide_env[migrate gitlab-rails database]' ================================================================================ Mixlib::ShellOut::ShellCommandFailed ------------------------------------ Expected process to exit with [0], but received '137' ---- Begin output of "bash" ---- STDOUT: STDERR: ---- End output of "bash" ---- Ran "bash" returned 137 Cookbook Trace: (most recent call first) ---------------------------------------- /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/resources/rails_migration.rb:20:in `block in class_from_file' Resource Declaration: --------------------- # In /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/resources/rails_migration.rb 20: bash_hide_env "migrate #{new_resource.name} database" do 21: code <<-EOH 22: set -e 23: log_file="#{logging_settings[:log_directory]}/#{new_resource.logfile_prefix}-$(date +%Y-%m-%d-%H-%M-%S).log" 24: umask 077 25: /opt/gitlab/bin/gitlab-rake #{new_resource.rake_task} 2>& 1 | tee ${log_file} 26: STATUS=${PIPESTATUS[0]} 27: chown #{account_helper.gitlab_user}:#{account_helper.gitlab_group} ${log_file} 28: echo $STATUS > #{new_resource.helper.db_migrate_status_file} 29: exit $STATUS 30: EOH 31: 32: environment new_resource.environment if new_resource.property_is_set?(:environment) 33: new_resource.dependent_services.each do |svc| 34: notifies :restart, svc, :immediately 35: end 36: 37: not_if { new_resource.helper.migrated? } 38: sensitive false 39: end 40: end Compiled Resource: ------------------ # Declared in /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/resources/rails_migration.rb:20:in `block in class_from_file' bash_hide_env("migrate gitlab-rails database") do action [:run] default_guard_interpreter :default interpreter "bash" declared_type :bash_hide_env cookbook_name "gitlab" recipe_name "database_migrations" code " set -e\n log_file=\"/var/log/gitlab/gitlab-rails/gitlab-rails-db-migrate-$(date +%Y-%m-%d-%H-%M-%S).log\"\n umask 077\n /opt/gitlab/bin/gitlab-rake gitlab:db:configure 2>& 1 | tee ${log_file}\n STATUS=${PIPESTATUS[0]}\n chown git:git ${log_file}\n echo $STATUS > /var/opt/gitlab/gitlab-rails/upgrade-status/db-migrate-c6697a9cd49e1cfa45cc31b1ecdb90ae-bd824d1abb2\n exit $STATUS\n" environment "*sensitive value suppressed*" not_if { #code block } end System Info: ------------ chef_version=18.3.0 platform=centos platform_version=7.6.1810 ruby=ruby 3.1.5p253 (2024-04-023 revision 1945f8dc0e) [x86_64-linux] program_name=/opt/gitlab/embedded/bin/cinc-client executable=/opt/gitlab/embedded/bin/cinc-client ================================================================================ Error executing action `run` on resource 'rails_migration[gitlab-rails]' ================================================================================ Mixlib::ShellOut::ShellCommandFailed ------------------------------------ bash_hide_env[migrate gitlab-rails database] (gitlab::database_migrations line 20) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '137' ---- Begin output of "bash" ---- STDOUT: STDERR: ---- End output of "bash" ---- Ran "bash" returned 137 Cookbook Trace: (most recent call first) ---------------------------------------- /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/resources/rails_migration.rb:20:in `block in class_from_file' Resource Declaration: --------------------- # In /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/recipes/database_migrations.rb 51: rails_migration "gitlab-rails" do 52: rake_task 'gitlab:db:configure' 53: logfile_prefix 'gitlab-rails-db-migrate' 54: helper migration_helper 55: 56: environment env_variables 57: dependent_services dependent_services 58: notifies :run, "execute[clear the gitlab-rails cache]", :immediately 59: notifies :run, "ruby_block[check remote PG version]", :immediately 60: 61: only_if { migration_helper.attributes_node['auto_migrate'] } 62: end Compiled Resource: ------------------ # Declared in /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/recipes/database_migrations.rb:51:in `from_file' rails_migration("gitlab-rails") do action [:run] default_guard_interpreter :default declared_type :rails_migration cookbook_name "gitlab" recipe_name "database_migrations" rake_task "gitlab:db:configure" logfile_prefix "gitlab-rails-db-migrate" helper "*sensitive value suppressed*" environment "*sensitive value suppressed*" only_if { #code block } end System Info: ------------ chef_version=18.3.0 platform=centos platform_version=7.6.1810 ruby=ruby 3.1.5p253 (2024-04-023 revision 1945f8dc0e) [x86_64-linux] program_name=/opt/gitlab/embedded/bin/cinc-client executable=/opt/gitlab/embedded/bin/cinc-client [2025-07-14T15:05:48+08:00] INFO: Running queued delayed notifications before re-raising exception [2025-07-14T15:05:48+08:00] INFO: templatesymlink[Create a gitlab.yml and create a symlink to Rails root] sending run action to execute[clear the gitlab-rails cache] (delayed) Recipe: gitlab::gitlab-rails * execute[clear the gitlab-rails cache] action run ================================================================================ Error executing action `run` on resource 'execute[clear the gitlab-rails cache]' ================================================================================ Mixlib::ShellOut::ShellCommandFailed ------------------------------------ Expected process to exit with [0], but received '' ---- Begin output of /opt/gitlab/bin/gitlab-rake cache:clear ---- STDOUT: STDERR: ---- End output of /opt/gitlab/bin/gitlab-rake cache:clear ---- Ran /opt/gitlab/bin/gitlab-rake cache:clear returned Resource Declaration: --------------------- # In /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/recipes/gitlab-rails.rb 543: execute "clear the gitlab-rails cache" do 544: command "/opt/gitlab/bin/gitlab-rake cache:clear" 545: action :nothing 546: not_if { omnibus_helper.not_listening?('redis') || !node['gitlab']['gitlab_rails']['rake_cache_clear'] } 547: end 548: Compiled Resource: ------------------ # Declared in /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/recipes/gitlab-rails.rb:543:in `from_file' execute("clear the gitlab-rails cache") do action [:nothing] default_guard_interpreter :execute command "/opt/gitlab/bin/gitlab-rake cache:clear" declared_type :execute cookbook_name "gitlab" recipe_name "gitlab-rails" not_if { #code block } end System Info: ------------ chef_version=18.3.0 platform=centos platform_version=7.6.1810 ruby=ruby 3.1.5p253 (2024-04-023 revision 1945f8dc0e) [x86_64-linux] program_name=/opt/gitlab/embedded/bin/cinc-client executable=/opt/gitlab/embedded/bin/cinc-client [2025-07-14T15:06:56+08:00] INFO: version_file[Create version file for Gitaly] sending hup action to runit_service[gitaly] (delayed) Recipe: gitaly::enable * runit_service[gitaly] action hup[2025-07-14T15:06:56+08:00] INFO: runit_service[gitaly] signalled (HUP) [2025-07-14T15:06:56+08:00] INFO: runit_service[gitaly] sent hup - send hup to runit_service[gitaly] [2025-07-14T15:06:56+08:00] INFO: file[create /opt/gitlab/embedded/etc/90-omnibus-gitlab-kernel.shmmax.conf kernel.shmmax] sending run action to execute[reload all sysctl conf] (delayed) Recipe: package::sysctl * execute[reload all sysctl conf] action run [execute] * Applying /usr/lib/sysctl.d/00-system.conf ... net.bridge.bridge-nf-call-ip6tables = 0 net.bridge.bridge-nf-call-iptables = 0 net.bridge.bridge-nf-call-arptables = 0 * Applying /usr/lib/sysctl.d/10-default-yama-scope.conf ... kernel.yama.ptrace_scope = 0 * Applying /usr/lib/sysctl.d/50-default.conf ... kernel.sysrq = 16 kernel.core_uses_pid = 1 net.ipv4.conf.default.rp_filter = 1 net.ipv4.conf.all.rp_filter = 1 net.ipv4.conf.default.accept_source_route = 0 net.ipv4.conf.all.accept_source_route = 0 net.ipv4.conf.default.promote_secondaries = 1 net.ipv4.conf.all.promote_secondaries = 1 fs.protected_hardlinks = 1 fs.protected_symlinks = 1 * Applying /etc/sysctl.d/90-omnibus-gitlab-kernel.sem.conf ... kernel.sem = 250 32000 32 275 * Applying /etc/sysctl.d/90-omnibus-gitlab-kernel.shmall.conf ... kernel.shmall = 4194304 * Applying /etc/sysctl.d/90-omnibus-gitlab-kernel.shmmax.conf ... kernel.shmmax = 17179869184 * Applying /etc/sysctl.d/99-sysctl.conf ... net.ipv4.ip_forward = 1 net.bridge.bridge-nf-call-arptables = 1 net.bridge.bridge-nf-call-ip6tables = 1 net.bridge.bridge-nf-call-iptables = 1 net.ipv4.ip_local_reserved_ports = 30000-32767 vm.max_map_count = 262144 vm.swappiness = 1 fs.inotify.max_user_instances = 524288 kernel.pid_max = 65535 vm.overcommit_memory = 1 net.core.somaxconn = 511 * Applying /etc/sysctl.d/lvs_dr.conf ... net.ipv4.conf.all.arp_ignore = 1 net.ipv4.conf.lo.arp_ignore = 1 net.ipv4.conf.all.arp_announce = 2 net.ipv4.conf.lo.arp_announce = 2 * Applying /etc/sysctl.conf ... net.ipv4.ip_forward = 1 net.bridge.bridge-nf-call-arptables = 1 net.bridge.bridge-nf-call-ip6tables = 1 net.bridge.bridge-nf-call-iptables = 1 net.ipv4.ip_local_reserved_ports = 30000-32767 vm.max_map_count = 262144 vm.swappiness = 1 fs.inotify.max_user_instances = 524288 kernel.pid_max = 65535 vm.overcommit_memory = 1 net.core.somaxconn = 511 [2025-07-14T15:06:56+08:00] INFO: execute[reload all sysctl conf] ran successfully - execute sysctl -e --system Running handlers: [2025-07-14T15:06:56+08:00] ERROR: Running exception handlers There was an error running gitlab-ctl reconfigure: Multiple failures occurred: * Mixlib::ShellOut::ShellCommandFailed occurred in Cinc Client run: rails_migration[gitlab-rails] (gitlab::database_migrations line 51) had an error: Mixlib::ShellOut::ShellCommandFailed: bash_hide_env[migrate gitlab-rails database] (gitlab::database_migrations line 20) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '137' ---- Begin output of "bash" ---- STDOUT: STDERR: ---- End output of "bash" ---- Ran "bash" returned 137 * Mixlib::ShellOut::ShellCommandFailed occurred in delayed notification: execute[clear the gitlab-rails cache] (gitlab::gitlab-rails line 543) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '' ---- Begin output of /opt/gitlab/bin/gitlab-rake cache:clear ---- STDOUT: STDERR: ---- End output of /opt/gitlab/bin/gitlab-rake cache:clear ---- Ran /opt/gitlab/bin/gitlab-rake cache:clear returned Notes: Default admin account has been configured with following details: Username: root Password: You didn't opt-in to print initial root password to STDOUT. Password stored to /etc/gitlab/initial_root_password. This file will be cleaned up in first reconfigure run after 24 hours. NOTE: Because these credentials might be present in your log files in plain text, it is highly recommended to reset the password following https://docs.gitlab.com/ee/security/reset_user_password.html#reset-your-root-password. Running handlers complete [2025-07-14T15:06:56+08:00] ERROR: Exception handlers complete Infra Phase failed. 319 resources updated in 02 minutes 41 seconds Notes: Default admin account has been configured with following details: Username: root Password: You didn't opt-in to print initial root password to STDOUT. Password stored to /etc/gitlab/initial_root_password. This file will be cleaned up in first reconfigure run after 24 hours. NOTE: Because these credentials might be present in your log files in plain text, it is highly recommended to reset the password following https://docs.gitlab.com/ee/security/reset_user_password.html#reset-your-root-password. [2025-07-14T15:06:56+08:00] FATAL: Stacktrace dumped to /opt/gitlab/embedded/cookbooks/cache/cinc-stacktrace.out [2025-07-14T15:06:56+08:00] FATAL: --------------------------------------------------------------------------------------- [2025-07-14T15:06:56+08:00] FATAL: PLEASE PROVIDE THE CONTENTS OF THE stacktrace.out FILE (above) IF YOU FILE A BUG REPORT [2025-07-14T15:06:56+08:00] FATAL: --------------------------------------------------------------------------------------- [2025-07-14T15:06:56+08:00] FATAL: Chef::Exceptions::MultipleFailures: Multiple failures occurred: * Mixlib::ShellOut::ShellCommandFailed occurred in Cinc Client run: rails_migration[gitlab-rails] (gitlab::database_migrations line 51) had an error: Mixlib::ShellOut::ShellCommandFailed: bash_hide_env[migrate gitlab-rails database] (gitlab::database_migrations line 20) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '137' ---- Begin output of "bash" ---- STDOUT: STDERR: ---- End output of "bash" ---- Ran "bash" returned 137 * Mixlib::ShellOut::ShellCommandFailed occurred in delayed notification: execute[clear the gitlab-rails cache] (gitlab::gitlab-rails line 543) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '' ---- Begin output of /opt/gitlab/bin/gitlab-rake cache:clear ---- STDOUT: STDERR: ---- End output of /opt/gitlab/bin/gitlab-rake cache:clear ---- Ran /opt/gitlab/bin/gitlab-rake cache:clear returned
07-15
<think>我们正在处理一个GitLab在运行`gitlab-ctl reconfigure`时出现的错误。错误信息显示在执行数据库迁移(rails_migration)时失败,具体是在运行bash命令时返回了退出码137,随后在清理缓存时也出现了错误(退出码为空)。 ### 问题分析: 1. **数据库迁移失败**: - 命令:`bash_hide_env "migrate gitlab-rails database"` 执行了一个bash脚本,该脚本运行了`gitlab-rake gitlab:db:configure`。 - 错误:退出码137,通常表示进程被信号终止(SIGKILL)。常见原因包括内存不足(OOM Killer)或手动终止。 - 环境:CentOS 7.6,GitLab使用内嵌的Chef(cinc-client)进行配置。 2. **清理缓存失败**: - 命令:`/opt/gitlab/bin/gitlab-rake cache:clear` 执行时返回空退出码,可能意味着进程被中断或崩溃。 ### 可能原因: - **内存不足**:在数据库迁移过程中,GitLab需要足够的内存。如果系统内存不足,可能导致OOM Killer终止进程(表现为137错误)。 - **资源限制**:系统资源(如进程数、文件描述符)不足。 - **磁盘空间不足**:需要检查磁盘空间是否足够。 - **权限问题**:虽然日志中未直接提示,但也要考虑运行命令的用户权限。 ### 解决步骤: 1. **检查系统资源**: - 内存:使用`free -h`查看可用内存。GitLab官方推荐至少4GB内存,对于较大的实例可能需要更多。 - 交换空间:确保有足够的交换空间(可以使用`swapon --show`检查)。如果内存不足,可以增加交换空间。 - 磁盘空间:使用`df -h`检查,特别是`/`和`/var`目录。 2. **增加交换空间(临时)**: ```bash sudo dd if=/dev/zero of=/swapfile bs=1M count=4096 # 创建4GB的交换文件 sudo chmod 600 /swapfile sudo mkswap /swapfile sudo swapon /swapfile ``` 永久生效:将`/swapfile swap swap defaults 0 0`添加到`/etc/fstab`。 3. **调整GitLab配置以降低内存占用**: 编辑`/etc/gitlab/gitlab.rb`,可以尝试减少工作进程数: ```ruby unicorn['worker_processes'] = 2 # 默认是CPU核心数,可以降低 sidekiq['concurrency'] = 5 # 默认是25,可以降低 ``` 然后重新配置:`sudo gitlab-ctl reconfigure` 4. **重新运行迁移**: 在确保资源充足后,尝试手动运行数据库迁移: ```bash sudo gitlab-rake gitlab:db:configure ``` 5. **检查日志**: - 数据库迁移日志:`/var/log/gitlab/gitlab-rails/gitlab-rails-db-migrate-*.log` - 系统日志:`/var/log/messages` 或 `journalctl` 查看是否有OOM Killer的记录。 6. **清理缓存失败的问题**: 在数据库迁移成功后,可以尝试手动清理缓存: ```bash sudo gitlab-rake cache:clear ``` ### 注意事项: - 如果问题持续,考虑升级服务器配置(增加内存)。 - 确保备份数据后再进行操作。 ### 结论: 根据错误退出码137,最可能的原因是内存不足。建议先增加交换空间,然后调整GitLab配置以减少内存占用,最后重新运行配置。 §§相关问题§§ 1. 如何检查Linux系统是否由于内存不足而终止了进程? 2. 在GitLab中,除了调整`unicorn`和`sidekiq`的配置外,还有哪些配置可以降低内存占用? 3. 如果增加交换空间后问题仍然存在,还有哪些可能的原因和解决方案? 4. 如何永久添加交换文件并在系统重启后自动挂载? 5. 在运行GitLab数据库迁移时,如何查看实时的日志输出以监控迁移过程?</think>根据错误日志分析,GitLab 在运行 `gitlab-ctl reconfigure` 时出现两个关键失败: 1. **数据库迁移失败** - 执行 `bash_hide_env[migrate gitlab-rails database]` 时返回错误码 **137** - 典型原因:内存不足触发 OOM Killer 终止进程(137 = SIGKILL) - 相关资源:`rails_migration["gitlab-rails"]` 调用 Rake 任务 `gitlab:db:configure` 2. **缓存清理失败** - 执行 `gitlab-rake cache:clear` 时无退出码(可能被中断) - 发生在数据库迁移失败后的延迟通知中 ### 根本原因分析 1. **内存不足 (OOM)** - 数据库迁移需要大量内存,CentOS 7.6 默认配置可能不足 - 错误码 137 是 OOM Killer 终止进程的典型信号 - 日志中 `vm.overcommit_memory = 1` 配置可能加剧内存压力 2. **依赖服务问题** - 迁移任务依赖 Redis 服务(`not_if { omnibus_helper.not_listening?('redis') }`) - Redis 未正常运行可能导致后续缓存清理失败 ### 解决方案 #### 紧急处理 ```bash # 增加交换空间(临时缓解OOM) sudo dd if=/dev/zero of=/swapfile bs=1G count=4 sudo chmod 600 /swapfile sudo mkswap /swapfile sudo swapon /swapfile # 重启关键服务 sudo gitlab-ctl restart redis sudo gitlab-ctl restart postgresql # 手动重试数据库迁移 sudo gitlab-rake gitlab:db:configure ``` #### 永久修复 1. **调整内存配置** ```bash # 永久增加交换空间 echo '/swapfile none swap sw 0 0' | sudo tee -a /etc/fstab # 优化内核参数 sudo sysctl -w vm.overcommit_memory=0 sudo sysctl -w vm.swappiness=60 ``` 2. **修改 GitLab 配置 (`/etc/gitlab/gitlab.rb`)** ```ruby # 减少并发减轻内存压力 unicorn['worker_processes'] = 2 sidekiq['max_concurrency'] = 5 # 确保 Redis 有足够内存 redis['maxmemory'] = "1024mb" redis['maxmemory_policy'] = "allkeys-lru" ``` 3. **重运行配置** ```bash sudo gitlab-ctl reconfigure sudo gitlab-rake cache:clear ``` ### 验证步骤 ```bash # 检查迁移状态 sudo gitlab-rake db:migrate:status # 查看内存日志 grep -i "oom" /var/log/messages grep -i "kill" /var/log/gitlab/gitlab-rails/*.log ```
评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值