Message "error while loading shared libraries: /lib64/libc.so.6: ELF file OS ABI invalid" appearing

Environment

• Red Hat Enterprise Linux 5/6

Issue

• After invoking a program, the following error is printed on the screen:

Raw

error while loading shared libraries: <library_path>: ELF file OS ABI invalid

• What means "ELF file OS ABI invalid"?
• While applying the latest patches, why the messages are printed on screen?

Raw

getent: error while loading shared libraries: /lib64/libc.so.6: ELF file OS ABI invalid
warning: /etc/pam.d/system-auth created as /etc/pam.d/system-auth.rpmnew
warning: /etc/security/access.conf created as /etc/security/access.conf.rpmnew
warning: /etc/security/limits.conf created as /etc/security/limits.conf.rpmnew
/bin/sh: error while loading shared libraries: /lib64/libc.so.6: ELF file OS ABI invalid
Error in PREIN scriptlet in rpm package udev-095-14.29.el5.x86_64
error: %pre(udev-095-14.29.el5.x86_64) scriptlet failed, exit status 127
error:   install: %pre scriptlet failed (2), skipping udev-095-14.29.el5
/bin/sh: error while loading shared libraries: /lib64/libc.so.6: ELF file OS ABI invalid
Error in PREIN scriptlet in rpm package initscripts-8.45.42-1.el5_8.1.x86_64
error: %pre(initscripts-8.45.42-1.el5_8.1.x86_64) scriptlet failed, exit status 127
error:   install: %pre scriptlet failed (2), skipping initscripts-8.45.42-1.el5_8.1
/bin/sh: error while loading shared libraries: /lib64/libc.so.6: ELF file OS ABI invalid
Error in PREIN scriptlet in rpm package hal-0.5.8.1-64.el5.x86_64
error: %pre(hal-0.5.8.1-64.el5.x86_64) scriptlet failed, exit status 127
error:   install: %pre scriptlet failed (2), skipping hal-0.5.8.1-64.el5
/bin/sh: error while loading shared libraries: /lib64/libc.so.6: ELF file OS ABI invalid
Error in PREIN scriptlet in rpm package pm-utils-0.99.3-14.el5.x86_64
error: %pre(pm-utils-0.99.3-14.el5.x86_64) scriptlet failed, exit status 127
error:   install: %pre scriptlet failed (2), skipping pm-utils-0.99.3-14.el5

Resolution

• Replace the shared library with a version compatible with the current version of the Operating System.
• Remove the third party library which is causing the problem. Contact its application vendor for the same.
• If third party application using its own dynamic linker cannot run with the above error, it's possible that it could not find the correct dynamic linker. If there is a dynamic linker, which can be used with the application, in any directories of LD_LIBRARY_PATH, it uses the linker and it's possible that it doesn't find the correct library. Check the path in LD_LIBRARY_PATH is valid to run the application. Delete the directory including the wrong linker which can run with the application.

Root Cause

• The application being executed is trying to load a shared library using an incompatible Application Binary Interface (ABI), with the current RHEL version. The ABI describes the interface between an application and the operating system or another application, and is responsible for details like data types, sizes, function calling conventions, binary format of object-files, and some other details.
• It is not always possible to load the same library or run the same application in different major versions of the Red Hat Enterprise Linux, since that ABIs can not be compatible between these versions. Red Hat provides a compatibility policy related to the application binary interface between versions which covers the compatibility policy in details.
• The policy specification and the another information about compatibility can be found at https://access.redhat.com/home into the "Developer Guide" documentation under each RHEL version.

Diagnostic Steps

• While patching a server from RHEL 5.6 to 5.9, it looks like /lib64/libc.so.6 is symlinked to /lib64/libc-2.5.so. When applying the patch, the link is changed to /lib64/libc-2.12.so. As yum continues to update, multiple errors begin to get thrown for libc as follows.

Raw

getent: error while loading shared libraries: /lib64/libc.so.6: ELF file OS ABI invalid
warning: /etc/pam.d/system-auth created as /etc/pam.d/system-auth.rpmnew
warning: /etc/security/access.conf created as /etc/security/access.conf.rpmnew
warning: /etc/security/limits.conf created as /etc/security/limits.conf.rpmnew
/bin/sh: error while loading shared libraries: /lib64/libc.so.6: ELF file OS ABI invalid
Error in PREIN scriptlet in rpm package udev-095-14.29.el5.x86_64
error: %pre(udev-095-14.29.el5.x86_64) scriptlet failed, exit status 127
error:   install: %pre scriptlet failed (2), skipping udev-095-14.29.el5
/bin/sh: error while loading shared libraries: /lib64/libc.so.6: ELF file OS ABI invalid
Error in PREIN scriptlet in rpm package initscripts-8.45.42-1.el5_8.1.x86_64
error: %pre(initscripts-8.45.42-1.el5_8.1.x86_64) scriptlet failed, exit status 127
error:   install: %pre scriptlet failed (2), skipping initscripts-8.45.42-1.el5_8.1
/bin/sh: error while loading shared libraries: /lib64/libc.so.6: ELF file OS ABI invalid
Error in PREIN scriptlet in rpm package hal-0.5.8.1-64.el5.x86_64
error: %pre(hal-0.5.8.1-64.el5.x86_64) scriptlet failed, exit status 127
error:   install: %pre scriptlet failed (2), skipping hal-0.5.8.1-64.el5
/bin/sh: error while loading shared libraries: /lib64/libc.so.6: ELF file OS ABI invalid
Error in PREIN scriptlet in rpm package pm-utils-0.99.3-14.el5.x86_64
error: %pre(pm-utils-0.99.3-14.el5.x86_64) scriptlet failed, exit status 127
error:   install: %pre scriptlet failed (2), skipping pm-utils-0.99.3-14.el5

• The library file /lib64/libc-2.12.so is not compatible with Red Hat Enterprise Linux and not shipped by Red Hat.

Raw

[root@example ~]# ls -l /lib64/libc.so.6
lrwxrwxrwx 1 root root 11 Jul 31 17:06 /lib64/libc.so.6 -> libc-2.5.so

[root@example ~]# ls -l /lib64/libc-2.5.so
-rwxr-xr-x 1 root root 1722232 May 17 12:55 /lib64/libc-2.5.so

[root@example ~]# ls -l /lib64/libc-2.12.so
ls: /lib64/libc-2.12.so: No such file or directory

[root@example ~]# yum whatprovides "/lib64/libc-2.12.so"
Loaded plugins: product-id, rhnplugin, security, subscription-manager
This system is not registered to Red Hat Subscription Management. You can use subscription-manager to register.
This system is receiving updates from RHN Classic or RHN Satellite.
rhel-x86_64-server-5                                                                                                    | 1.3 kB     00:00     
rhel-x86_64-server-5/primary                                                                                            | 5.8 MB     00:00     
rhel-x86_64-server-5                                                                                                               15343/15343
rhel-x86_64-server-cluster-5                                                                                            | 1.3 kB     00:00     
rhel-x86_64-server-cluster-5/primary                                                                                    |  26 kB     00:00     
rhel-x86_64-server-cluster-5                                                                                                           205/205
rhel-x86_64-server-cluster-storage-5                                                                                    | 1.3 kB     00:00     
rhel-x86_64-server-scalefs-5                                                                                            | 1.1 kB     00:00     
Importing additional filelist information
rhel-x86_64-server-5/filelists                                                                                          |  43 MB     00:01     
rhel-x86_64-server-5                                                                                                               15343/15343
rhel-x86_64-server-cluster-5/filelists                                                                                  | 3.2 MB     00:00     
rhel-x86_64-server-cluster-5                                                                                                           205/205
rhel-x86_64-server-cluster-storage-5/filelists                                                                          |  99 kB     00:00     
rhel-x86_64-server-cluster-storage-5                                                                                                   225/225
rhel-x86_64-server-scalefs-5/filelists                                                                                  | 1.3 kB     00:00     
rhel-x86_64-server-scalefs-5                                                                                                               5/5
No Matches found

• The links get changed because ldconfig is called up during yum update process.

Raw

[root@example ~]# strace -o /tmp/strace.out.yum yum update
[root@example ~]# grep -i ldconfig /tmp/strace.out.yum
write(31, "/sbin/ldconfig\nfreetype\nx86_64\n0"..., 4096) = 4096
write(31, ".49\n6.el6\n1\n/sbin/ldconfig\nlibai"..., 4096) = 4096
write(44, "/sbin/ldconfig\nexit 0", 21) = 21
write(45, "/sbin/ldconfig\n[ -e /etc/blkid.t"..., 164) = 164
open("/usr/share/man/man8/ldconfig_selinux.8.gz;523deb25", O_WRONLY|O_CREAT|O_TRUNC, 0666) = 46
lstat("/usr/share/man/man8/ldconfig_selinux.8.gz", {st_mode=S_IFREG|0644, st_size=1335, ...}) = 0
rename("/usr/share/man/man8/ldconfig_selinux.8.gz;523deb25", "/usr/share/man/man8/ldconfig_selinux.8.gz") = 0
chown("/usr/share/man/man8/ldconfig_selinux.8.gz", 0, 0) = 0
chmod("/usr/share/man/man8/ldconfig_selinux.8.gz", 0644) = 0
utime("/usr/share/man/man8/ldconfig_selinux.8.gz", [2013/06/25-14:37:49, 2013/06/25-14:37:49]) = 0
write(46, "/sbin/ldconfig\n# create certific"..., 100) = 100
write(46, "/sbin/ldconfig", 14)         = 14
write(42, "/sbin/ldconfig >/dev/null 2>/dev"..., 37) = 37
write(42, "/sbin/ldconfig\n\n# Automatically "..., 93) = 93
lstat("/usr/share/man/man8/ldconfig_selinux.8.gz", {st_mode=S_IFREG|0644, st_size=1335, ...}) = 0
write(42, "/sbin/ldconfig", 14)         = 14

• The third party application may use the linker provided by the vender, and it doesn't use the linker (like /lib/ld-linux.so.2 and /lib64/ld-linux-x86_64.so.2) provided by Red Hat. In that case,
  the output of ldd may not show the libraries linked actually by the application, since ldd doesn't know the linker provided by the vendor. So, use strace to check which library is linked dynamically with such a application. Also, check which linker is used by the application with readelf -l .

Raw

# readelf -l /bin/ls |grep interpreter
      [Requesting program interpreter: /lib64/ld-linux-x86-64.so.2]