[GPU Driver]Purge the mismatched drivers and install the matched one to solve Version Mismatch

When running 'nvidia-smi' on ubuntu, I encountered the following output:

The output indicates a version mismatch between the NVIDIA driver installed on the system and the NVML (NVIDIA Management Library) library that nvidia-smi is attempting to use. This can happen due to various reasons, such as incomplete driver installations, multiple driver versions present on the system, or incorrect library paths.

Here's a step-by-step guide to diagnose and resolve this issue:

1. Check Current NVIDIA Driver Installation

First, verify the currently installed NVIDIA drivers and their versions.

dpkg -l | grep -i nvidia

The output was

From the output of the dpkg -l | grep -i nvidia command, it's evident that multiple NVIDIA driver versions are installed on the system (495, 510, 525, and 535). This can cause conflicts leading to the error you're experiencing.

To resolve this issue, I should remove all existing NVIDIA drivers and perform a clean installation of the desired driver version.

(Before making significant changes to the system, it's always a good practice to backup any important data to prevent accidental loss.)

2. Purge All Existing NVIDIA Drivers

To ensure a clean slate, remove all existing NVIDIA drivers and related packages. This includes removing all versions (495, 510, 525, and 535).

Execute the following commands:

sudo apt-get purge 'nvidia-*'
sudo apt-get autoremove

Explanation:

• sudo apt-get purge 'nvidia-*': Removes all packages that start with nvidia-.
• sudo apt-get autoremove: Cleans up any dependencies that are no longer needed.

3.Verify Removal of NVIDIA Packages

Ensure that all NVIDIA packages have been removed.

dpkg -l | grep -i nvidia

4. Update Package Repository

Refresh the package list to ensure you have the latest information.

sudo apt-get update

5. Install the corresponding NVIDIA Driver

Now, install the desired NVIDIA driver version.
You may find the following blog helpful [GPU Driver] How to find the corresponding GPU driver version for your GPU on Ubuntu-优快云博客

Execute the following command:

sudo apt-get install nvidia-driver-535

Alternative Method (Using ubuntu-drivers):

We can also use the ubuntu-drivers tool to automatically identify and install the recommended driver.

sudo ubuntu-drivers devices
sudo ubuntu-drivers autoinstall

If Ubuntu didn't find any drivers automatically,  you can try installing drivers manually. You may find the following blog helpful.

[GPU Driver] Install GPU Driver Manually for NVIDIA GPUs on Ubuntu_recommended: 536.40 or latest driver available-优快云博客

6. Reboot the System

After installing the driver, reboot the system to apply the changes.

sudo reboot

7. Verify the Installation

Once the system has rebooted, check if the NVIDIA driver is correctly installed and nvidia-smi works without errors.

nvidia-smi