Adding NVIDIA GRID vGPU drivers to a RHEL / CentOs VM results in a display failure and error: ”Oh no! Something has gone wrong!” when connecting via a remote connection after reboot

Answer ID 4165
Published 06/15/2016 03:13 PM
Updated 06/27/2016 11:26 AM

Adding NVIDIA GRID vGPU drivers to a RHEL / CentOs VM results in a display failure and error: "Oh no! Something has gone wrong!" when connecting via a remote connection after reboot


Error / Symptoms

A CentOs or RHEL Linux VM is deployed and can be connected to via a remote protocol, but once the NVIDIA vGPU Drivers are added into the VM and it's rebooted the display fails and an error "Oh no! Something has gone wrong!" is seen when connecting via a remote connection.

Root Cause

One possible cause of this error is that SELinux is blocking some aspects of the NVIDIA driver. You can investigate if this is the case by using SSH to access the VM and logging in as root and checking the messages file for errors like this below.

Feb 25 22:51:10 linuxws setroubleshoot: SELinux is preventing /usr/libexec/gnome-session-check-accelerated-helper from getattr access on the chr_file /dev/nvidiactl.

This can happen if SELinux is in enforcing mode. SELinux is a Linux Kernel security module and users should understand its role before changing permissions. See: https://en.wikipedia.org/wiki/Security-Enhanced_Linux.

Workaround

For some customers it may be acceptable to change their SELinux permissions.

To check current SELinux mode, run "sestatus":

# sestatus

SELinux status: enabled
SELinuxfs mount: /sys/fs/selinux
SELinux root directory: /etc/selinux
Loaded policy name: targeted
Current mode: permissive
Mode from config file: permissive
Policy MLS status: enabled
Policy deny_unknown status: allowed
Max kernel policy version: 28

Bolded options should be "permissive" to allow NVIDIA drivers to function.

Note: To switch SELinux to permissive mode without reboot (but not permamently) you can use "setenforce"

usage: setenforce [ Enforcing | Permissive | 1 | 0 ]

# setenforce 0

To permanent change SELinux settings

If you see this issue you can modify the SELinux behaviour by editing its configuration.

$ nano /etc/sysconfig/selinux

If you didn't log in as root, you'll need to append sudo to the commands

You should then see the contents of the config file as below.

Image

What we need to do is set the SELINUX= to either disabled, or permissive. Permissive has the advantage that it still continues to log any issues.

Once you've done that Ctrl+X hit Y to save changes, and then Enter to overwrite the old file and you can reboot.

Connecting from then on will take you into the desktop

Image

Where you can confirm the vGPU is attached using

$ nvidia-settings

Image

A video overviewing the steps outlined in this article has been provided by NVIDIA engineering: https://youtu.be/rMAvAB_-Z_Y

When editing config files, users should follow best practice and take a backup. Users do this at their own risk and NVIDIA bears no liability or support for errors made during this process.

Solution

Customers for whom having SELinux in enforcing mode is essential need to contact their OS vendor and request the version of SElinux in use permits NVIDIA drivers. This issue is documented in the NVIDIA vGPU driver release notes. NVIDIA is tracking this issue under the reference #200167868.

Customer needing to contact Redhat should quote RHEL Bug: https://bugzilla.redhat.com/show_bug.cgi?id=1322283 when asking for a fix.

References

Known Issues and Resolved issues: Release Notes GRID VGPU FOR VMWARE VSPHERE - Version 361.40 / 362.13 - April 4th, 2016

Known Issues and Resolved issues: Release Notes FOR GRID VIRTUAL GPU (VGPU) FOR CITRIX XENSERVER - Version 361.40 / 362.13 - April 4th, 2016

Relevant Products

NVIDIA GRID GPU enabled VMs using GRID cards such as K1, K2, M6, M60, M10

Linux OSs observed affected include CentOs 7.0 and (Redhat) RHEL 7.2 but other versions and OSs may exhibit the same behavior.

Was this answer helpful?
Your rating has been submitted, please tell us how we can make this answer more useful.

LIVE CHAT

Chat online with one of our support agents

CHAT NOW

ASK US A QUESTION

Contact Support for assistance

CONTACT US