Changes to the NVIDIA driver packages

The same set of changes described here have also been applied in the NVIDIA driver packages published by NVIDIA as part of the CUDA repositories. The changes will land in the 620 branch releases.

The same set of changes will land at the same time in the official driver installation guide.

DNF 5 reboot suggestion

The kernel module and the user space libraries must always be the same version, so after a driver update you really want to reboot. With DNF 4 on RHEL/EL 8 to 10, the packages have been adding themselves to the needs-restarting DNF plugin configuration for a while. DNF 5 did not have an equivalent until recently, but luckily my merge request to DNF 5 got merged without issues so we now have a configurable list of packages that can trigger the notification.

The snippets reside in /usr/share/dnf5/suggest-reboot.d/ and are available in Fedora (and eventually RHEL 11); after an update to the driver, “dnf5 needs-restarting” and the reboot hint at the end of the transaction tell you that a reboot is needed, like for a kernel update. This is also available in the 580 branch driver packages, the LTS driver that still supports older GPUs.

NVIDIA drivers on a system without an NVIDIA GPU

The NVIDIA driver packages can now be installed on any system, whether or not it has an NVIDIA GPU. That includes OS images that end up on very different hardware, and machines that only get an NVIDIA GPU later in their life, for example a laptop that gets plugged into a Thunderbolt dock or eGPU enclosure with an NVIDIA card inside.

Until now the packages assumed that if you installed them, you had the hardware. On a system without an NVIDIA GPU you would get services that fail at every boot, an autostart entry that complains at every login, and so on. None of it was harmful, but it made the driver a bad fit for generic images. Everything that needs the GPU now stays “dormant” and waits for the GPU to appear in the system.

Most of the changes are variations on the work done for Anatase (https://anatase.org/). Thanks to Antheas Kapenekakis for figuring it all out.

Kernel modules

The kernel modules are built as usual, through akmods (akmod-nvidia) or DKMS (dkms-nvidia), whether or not a GPU is present. They are not included in the initrd, and they are only loaded when the kernel finds a matching PCI device through its modalias. On a system without an NVIDIA GPU they just sit on disk. When a GPU appears, even after boot, udev loads the driver.

On top of the open kernel modules, nvidia-kmod and dkms-nvidia now carry two extra patches from Anatase:

  • Tunneled PCIe link speed. A GPU attached over Thunderbolt or USB4 sits behind a tunneled PCIe link, and the link often does not train to the best speed the tunnel supports. With this patch, whenever such a GPU is in use (at startup, and every time it wakes up from runtime power management), the driver stops the hardware from changing the link speed on its own at both ends of the link. It then asks the kernel’s PCI core to train the link once at the maximum speed available, and gives control back to the hardware when the GPU goes idle again. It is only enabled for Thunderbolt-attached GPUs and on kernels that export pcie_set_target_speed(), so internal GPUs are not affected. This is what makes the “plug in a dock with an NVIDIA GPU” case perform properly and not just work.
  • CEC over DisplayPort to HDMI adapters. This one is not about GPU detection, but it is a nice addition: it exposes the DisplayPort AUX channel to the kernel’s DRM helpers, so HDMI-CEC works through DP-to-HDMI adapters that support it. For example, you can control the Steam interface with a remote or turn on the TV once your HTPC set powers on.

I’m trying to push both patches internally in NVIDIA, let’s see if we can get them in.

Auotostart of programs

Previously the packages shipped systemd presets that enabled nvidia-persistenced and nvidia-powerd. An enabled service starts at every boot, GPU or not. The presets are gone. Both packages now ship a udev rule that pulls in the service when an NVIDIA display controller (VGA or 3D class) is added to the system:

ACTION=="add", SUBSYSTEM=="pci", ATTR{vendor}=="0x10de", ATTR{class}=="0x030000|0x030200", TAG+="systemd", ENV{SYSTEMD_WANTS}+="nvidia-persistenced.service"

On a system without an NVIDIA GPU the services never start. On a system with one they start at boot as before. When a GPU is hot-plugged, they start the moment it appears. There is nothing to enable or disable by hand.

nvidia-settings installs an autostart entry that restores the saved settings at login with “nvidia-settings --load-config-only“. Without an NVIDIA GPU this just fails at every login. The entry now checks for the device first:

Exec=sh -c "[ -e /dev/nvidia0 ] && exec /usr/bin/nvidia-settings --load-config-only"

No GPU means no device node, so that covers nvidia-settings as well.

CUDA in unprivileged containers

CUDA also needs the /dev/nvidia-uvm and /dev/nvidia-uvm-tools device nodes. They are normally created on demand: the NVIDIA libraries call nvidia-modprobe, a setuid helper that loads the nvidia-uvm module and creates the nodes.

Unprivileged containers (rootless Podman, Toolbx, Distrobox) can not call setuid programs on the system, so CUDA fails even if the rest of the GPU is passed through correctly. This is the standard mechanism for most of the non open source libraries that are part of the driver to make sure the correct device nodes are created upon execution.

nvidia-kmod-common now does this in advance with a udev rule. When an NVIDIA GPU is added, it runs “nvidia-modprobe -u -c 0“, which loads nvidia-uvm and creates both nodes. The nodes are then already there to pass into containers.

Updates to the Steam package

I’ve reviewed and updated the steam package to be noarch and no longer i686. As some of you are aware, since a few releases, the client is fully 64 bit, including supporting libraries, and it just ships some libraries and components for the integration on the 32 bit games (Steam overlay, etc.). This does not make 32 bit support optional, it’s still required to actually start the client.

As part of the change, I’ve also reviewed all the libraries and binaries that are required and the libraries that can be overridden. For those of you that are not aware, it works roughly like this:

  • libc.so.6/libdrm.so.2/libGL.so.1/libnsl.so.1 are the only real required libraries for installing on i686 and x8_64.
  • The rest can be considered “improvements” (additional hardware options compiled in specifically for Fedora, graphics drivers, specific distribution bugfixes, etc. – the list is quite long). Steam prefers the local libraries if available.
  • Some libraries are hardcoded (“pinned”) to be consumed only from the Steam Runtime, so even having those installed does not make any difference.

Long story short, I cross checked everything and I’ve revamped the SPEC file to be x86_64 centric with some i686 libraries pulled in as a dependency where it makes sense.

The SPEC file is smaller and tidier. Also for reasons I don’t understand, I can have the noarch package upgrade over i686, but not the same behaviour with x86_64 on i686. Anyway I think it makes more sense to have it noarch for the coming aarch64 variant or the steamrt3c runtime becoming the default.

This is how it will look like:

$ sudo dnf update steam
Updating and loading repositories:
Repositories loaded.
Package               Arch    Version              Repository      Size
Upgrading:
 steam                noarch  0:1.0.0.85-7.fc44    fedora-steam    19.4 MiB
   replacing steam    i686    0:1.0.0.85-6.fc44    fedora-steam    19.4 MiB

Transaction Summary:
 Upgrading:          1 package
 Replacing:          1 package

Total size of inbound packages is 19 MiB. Need to download 19 MiB.
After this operation, 0 B extra will be used (install 19 MiB, remove 19 MiB).
Is this ok [y/N]: 

NVIDIA driver 580 LTS repository

Soon we will get a release of drivers in the 590 branch. So branch 580 will be the last driver branch to support Maxwell, Pascal, and Volta based GPUs.

There is now a long term support for the 580 branch of NVIDIA drivers along with the main one. This repository contains only the 580 driver set in both DKMS and akmod format for supported Enterprise Linux (EPEL) and Fedora releases.

My plan is to maintain this repository until branch is End of Life, which should be around June 2028.

Until then, I’ll try to keep it in sync with the latest changes and updates going into the main repository.

To install this repository on Fedora:

# cd /etc/yum.repos.d
# wget https://negativo17.org/repos/fedora-nvidia-580.repo

On Red Hat Enterprise Linux and derivatives:

# cd /etc/yum.repos.d
# wget https://negativo17.org/repos/epel-nvidia-580.repo

As usual, the EPEL repository is required for Red Hat Enterprise Linux 8 and above.

The open source kernel modules have been removed, as they are not supported on Maxwell, Pascal and Volta GPUs. This makes the kernel module packages smaller and just with the modules supporting this hardware.

Starting with branch 590 the only use case for closed source modules on GPUs that are also supported by open modules is the vGPU case, which anyway as a matrix of supported driver versions and distributions. So starting from 590 I’ll remove the closed source modules and just stick with the open ones from then on.

If you plan to use the Multimedia repository as well, which contains 590 drivers, you can increase the priority of the NVIDIA 580 driver packages on Fedora (DNF5) with:

# dnf config-manager setopt fedora-nvidia-580.priority=90

or on EPEL (DNF4) with:

# dnf config-manager --save --setopt=epel-multimedia.priority=90

Source code stuff is available on Github as usual.

Plex desktop and HTPC client

Plex Desktop and HTPC clients are available again. Now Plex ships a precompiled QT 6 libraries with the private API enabled that allows for an easy packaging. The packages are available for EL9+ and Fedora on the x86_64 architecture only.

Just install the one you need via DNF. For the desktop application:

# dnf -y install PlexDesktop

For the HTPC application:

# dnf -y install plexHTPC

The packages come with every possible system library in use and can be installed in parallel. They both suggest all the VA-API drivers available in the multimedia repository.

Keep in mind that both sport MPV as the main player, so you can still customize the player options as you see fit. For example to force NVIDIA’s hardware decoding in both applications:

$ cat .local/share/plex/mpv.conf
hwdec=nvdec

The HTPC optional package that creates an auto starting “HTPC session” for set top box media players is not restored yet, I will add it in the next days.

The packages bundle QT 6 (for the web interface engine) with the private API enabled, FFMPeg with a progress API status reporting, a custom MPV and a CEC library in the HTPC client.

Why not using the official Flatpak installation directly? First of all I don’t like all the mounting and folder redirection of Flatpak, but also:

# flatpak install tv.plex.PlexDesktop tv.plex.PlexHTPC
Looking for matches…
Required runtime for tv.plex.PlexHTPC/x86_64/stable (runtime/org.freedesktop.Platform/x86_64/23.08) found in remote flathub
Do you want to install it? [Y/n]: y

tv.plex.PlexHTPC permissions:
ipc network pulseaudio x11 devices dbus access [1]

[1] org.freedesktop.PowerManagement, org.freedesktop.ScreenSaver

tv.plex.PlexDesktop permissions:
ipc network pulseaudio x11 devices dbus access [1]

[1] org.freedesktop.ScreenSaver


ID Branch Op Remote Download
1. org.freedesktop.Platform.GL.default 23.08 i flathub < 172.2 MB
2. org.freedesktop.Platform.GL.default 23.08-extra i flathub < 172.2 MB
3. org.freedesktop.Platform.GL.nvidia-555-58-02 1.4 i flathub < 305.0 MB
4. org.freedesktop.Platform.Locale 23.08 i flathub < 360.2 MB (partial)
5. org.freedesktop.Platform.openh264 2.2.0 i flathub < 944.3 kB
6. org.freedesktop.Platform 23.08 i flathub < 227.5 MB
7. tv.plex.PlexHTPC stable i flathub < 150.9 MB
8. tv.plex.PlexDesktop stable i flathub < 149.2 MB

Proceed with these changes to the system installation? [Y/n]:

That’s a whopping 1.5 GiB compressed download compared to the package footprint:

$ ls -1hs Plex*
106M PlexDesktop-1.96.0.177-3.fc40.x86_64.rpm
105M PlexHTPC-1.64.0.170-2.fc40.x86_64.rpm

That’s more than 7 times the size. Flatpak applications with duplicate libraries everywhere are not really my taste.

CUDA with unsupported GCC versions

When running the CUDA stack on a recent Fedora distribution, you’re very likely to hit the compatibility issue with the current GCC release not being yet supported by NVCC. This is quite easy to address, but not many people seem to know it.

At the moment of writing, CUDA 12.4 has GCC 13.x support, while Fedora 40 ships with GCC 14.

Since a few years I’ve been shipping a cuda-gcc package which appears as a drop in replacement for NVCC. It can be installed along with CUDA and the drivers from the Nvidia or multimedia repository or from a Fedora COPR if you are running the upstream CUDA packages provided by Nvidia.

This GCC version is hidden from the main path and is explicitly used by NVCC when compiling something. Installing the cuda-gcc-c++ package creates profile entries in /etc/profile.d that just do this:

# dnf -y install cuda-gcc-c++
$ cat /etc/profile.d/cuda-gcc.sh 
export NVCC_PREPEND_FLAGS='-ccbin /usr/bin/cuda'

Logout/login or reload your profile and you’re good to go.

This way, every time you invoke NVCC you are not using the system compiler but the one provided by the cuda-gcc package.

On a Red Hat Enterprise Linux based distribution you can achieve the same result by installting the development toolset of your choice and activating the environment for it. This is usually not an issue as NVCC is officially supported on those distributions.

Nvidia proprietary and open source kernel modules

With the latest bunch of updates to the Nvidia and Multimedia repositories, I’ve added the ability to switch to the two implementations of the kernel modules currently available in the Nvidia driver for Linux.

Since almost a year, the Nvidia driver ships with two different implementations of the kernel modules, one proprietary and one open source. The open source one as of drivers 545.x is now considered beta quality also for the workstations, so it seems a good moment to start shipping it.

The open source one is supposed to be the only one that will be kept in the future, but at the moment both are available and both differ in terms of functionality. You can read about the main differences in terms of functionality and what chips they support in the official documentation.

I did not want to introduce another variation of the kernel modules beside akmods, kABI and DKMS, this would have created even more confusion and lots of dependencies in the SPEC files for the variations. The new akmod and DKMS packages ship both sources (MIT/GPL and proprietary kernel modules) and allow you to switch between one or the other through a configuration file.

Considering that in the long run only the open source variant will remain, I wanted to make this as transparent as possible for the users. Basically, if you don’t care and just want something that works, nothing has changed for you.

The two sources get referenced as they are referenced inside the Nvidia run file, namely “kernel” for the original proprietary kernel modules and “kernel-open” for the new open source variation.

The following instructions show you how to switch between one implementation or the other.

DKMS

Check which version you have installed:

# modinfo -l nvidia
NVIDIA

Change the type of modules you want to use and trigger a rebuild and a reinstall:

# sed -i -e 's/kernel$/kernel-open/g' /etc/nvidia/kernel.conf
# dkms build -m nvidia/545.29.02 --force
# dkms install -m nvidia/545.29.02 --force

Now check again the license and you should see that it has changed to MIT/GPL:

# modinfo -l nvidia
Dual MIT/GPL
# reboot

To switch back, change the configuration again and then trigger the same process for rebuilding installing:

# sed -i -e 's/kernel-open$/kernel/g' /etc/nvidia/kernel.conf
# dkms build -m nvidia/545.29.02 --force
# dkms install -m nvidia/545.29.02 --force
# reboot

akmods

Check which version you have installed:

# modinfo -l nvidia
NVIDIA

Change the type of modules you want to use and trigger a rebuild and a reinstall:

# sed -i -e 's/kernel$/kernel-open/g' /etc/nvidia/kernel.conf
# akmods --rebuild

Now check again the license and you should see that it has changed to MIT/GPL:

# modinfo -l nvidia
Dual MIT/GPL
# reboot

To switch back, change the configuration again and then trigger the same process for rebuilding installing:

# sed -i -e 's/kernel-open$/kernel/g' /etc/nvidia/kernel.conf
# akmods --rebuild
# reboot

CUDA 9.0, cuDNN 7.0 and Wayland support in Fedora 27

The Nvidia repository now contains packages for Fedora 27. This is with the release candidate of CUDA 9, and it contains also cuDNN at version 7.0, which is the only version supported with CUDA 9 at the moment of writing.

The updated cuDNN 7.0 library has been added also to the other branches, this means it will be automatically upgraded from version 6.0 to 7.0. If you still need one of the previous versions, just remove it and install one of the compatibility packages:

# dnf list cuda-cudnn*
Installed Packages
cuda-cudnn.x86_64                   1:7-1.fc26         @fedora-nvidia
Available Packages
cuda-cudnn-devel.x86_64             1:7-1.fc26         fedora-nvidia 
cuda-cudnn5.1.x86_64                1:5.1-2.fc26       fedora-nvidia 
cuda-cudnn5.1-devel.x86_64          1:5.1-2.fc26       fedora-nvidia 
cuda-cudnn6.0.x86_64                1:6.0-1.fc26       fedora-nvidia 
cuda-cudnn6.0-devel.x86_64          1:6.0-1.fc26       fedora-nvidia 

CUDA 9 supports GCC 6.x and CLANG 3.9, so when it will be officially released, it will cover Fedora 25 and RHEL/CentOS compilers. With Fedora 27, there will be the usual need for a GCC compatibility package (like the compat-gcc53 package currently in the repository) as GCC is at version 7 and CLANG is at version 4.0.

I will try to provide a compat-gcc64 for Fedora 27+ at the time of the official CUDA 9 release.

Regarding the drivers, on Fedora 27 where Mutter 3.25+ is available, the modesetting part of the Nvidia drivers has been enabled by default, this means that at the login you can just select “GNOME” to run Gnome on Wayland. Please note that X 3D programs running on XWayland might not work properly.

CentOS 7.4 package rebases

As many have requested, I’ve enabled the updates for CentOS 7.4. As of this very moment, you need to enable the Continuous Release repository, which will get emptied when the final CentOS 7.4 images will be released. This means all the temporary packages I’ve put in the Red Hat Enterprise Linux 7.4 repository are now mainline.

If you are on CentOS 7.3 please proceed as follows:

# yum clean all
# yum-config-manager --enable cr
# yum update

You can then disable the cr repository (if you want) once CentOS 7.4 is out.

If you are on RHEL 7.4, you need to remove the temporary repository:

# yum clean all
# rm -f /etc/yum.repos.d/rhel74-temp.repo
# yum update

All Gstreamer packages are now at 1.10 as well as some other updates that have been already pushed into the Fedora repositories, like x265, HandBrake, etc..

RHEL 7.4 multimedia packages and Skype repository removal

The upgrade path from Red Hat Enterprise Linux 7.3 to 7.4 is a bit of a pain if you have the multimedia repository configured. This is because I’m rebuilding a few components for an upgraded libwebp package and because a lot of stuff has been rebased to versions that are in Fedora. Judging by the logs, I see that most of the downloads come from CentOS systems, so I just decided to hold on some updates that are required for the various package rebases for Red Hat Enterprise Linux 7.4. So until also CentOS releases version 7.4, I can’t make everyone happy and something (like Gstreamer plugin updates) will be stuck with 7.3 versions. Hopefully the new CentOS release will come quickly enough.

Also, I decided to stop rebuilding the base packages to use a newer libwebp version. This really had very few benefits and just a lot of pain due to the huge amount of packages involved in both x86_64 and i686 variants. The amount of packages affected by this weigh at around 3 gb.

In RHEL 7.4 there are additional WebKit variants that also would require a rebuild. So, as of today, to update the packages from the EPEL 7 multimedia repository you should run this command:

rpm -e --nodeps GraphicsMagick && yum distro-sync && yum -y install GraphicsMagick

Hopefully you would get an output similar to this:

Dependencies Resolved

==============================================================================================
 Package                         Arch        Version                  Repository         Size
==============================================================================================
Updating:
 compat-ffmpeg-libs              x86_64      1:2.8.12-2.el7           epel-multimedia   5.6 M
 ffmpeg                          x86_64      1:3.3.3-2.el7            epel-multimedia   1.5 M
 ffmpeg-libs                     i686        1:3.3.3-2.el7            epel-multimedia   6.1 M
 ffmpeg-libs                     x86_64      1:3.3.3-2.el7            epel-multimedia   6.3 M
 gstreamer1-plugins-bad          x86_64      1:1.4.5-5.el7            epel-multimedia   1.8 M
 libavdevice                     x86_64      1:3.3.3-2.el7            epel-multimedia    63 k
Downgrading:
 leptonica                       i686        1.72-2.el7               epel-multimedia   881 k
 leptonica                       x86_64      1.72-2.el7               epel              928 k
 libwebp                         i686        0.3.0-3.el7              base              169 k
 libwebp                         x86_64      0.3.0-3.el7              base              170 k
 lz4                             x86_64      1.7.3-1.el7              epel               82 k
 python-pillow                   x86_64      2.0.0-19.gitd1c6db8.el7  base              438 k
 webkitgtk                       x86_64      2.4.9-1.el7              epel               12 M
 webkitgtk3                      x86_64      2.4.9-6.el7              base               11 M
Installing for dependencies:
 libwebp0.6                      i686        0.6.0-1.el7              epel-multimedia   255 k
 libwebp0.6                      x86_64      0.6.0-1.el7              epel-multimedia   250 k

Transaction Summary
==============================================================================================
Install               ( 2 Dependent packages)
Upgrade    6 Packages
Downgrade  8 Packages

Total download size: 47 M
Is this ok [y/d/N]:

Basically libwebp should come again from the main CentOS/RHEL channels and the libwebp0.6 package should come from the multimedia repository. All the packages which were rebuilt for the previous libwebp 0.5 update should become synced again to their proper versions.

If you don’t get this output, but still get some dependency errors you have to do some debugging. For example, ffmpeg-libs.i686 requires libssh.i686, but the version of libssh in CentOS extras is different from the one in EPEL (it really depends on what kind of packages you have installed and with which repositories enabled) so I’m providing here the same version that is in CentOS extras but in both variants.

Update 16th August 2017

If you get many qt5 errors during the transactions, keep in mind that RHEL 7.4 has been rebased massively, and everyone else (including EPEL) is catching up. As of today, if you have the following errors (trimmed down) in a Yum transaction:

Error: Package: gvfs-1.30.4-3.el7.x86_64 (rhel-x86_64-server-7)
Error: Package: qt5-qtwebkit-5.6.1-3.b889f46git.el7.x86_64 (epel)
Error: Package: qt5-qtquickcontrols2-5.6.1-2.el7.x86_64 (@epel)
Error: Package: qt5-qtquickcontrols2-5.6.1-2.el7.x86_64 (@epel)
Error: Package: GraphicsMagick-1.3.26-3.el7.x86_64 (@epel-multimedia)
Error: Package: kf5-kdeclarative-5.36.0-1.el7.x86_64 (epel)
Error: Package: qt5-qtquickcontrols2-5.6.1-2.el7.x86_64 (@epel)
Error: Package: qt5-qtwebkit-5.6.1-3.b889f46git.el7.x86_64 (epel)
Transaction check error:
  file /usr/lib64/gstreamer-1.0/libgstopus.so from install of gstreamer1-plugins-bad-1:1.4.5-5.el7.x86_64 conflicts with file from package gstreamer1-plugins-base-1.10.4-1.el7.x86_64

You can do the following. For this:

Error: Package: GraphicsMagick-1.3.26-3.el7.x86_64 (@epel-multimedia)

Do:

rpm -e --nodeps GraphicsMagick && yum -y install GraphicsMagick

All of the QT7KDE 5 stuff:

Error: Package: qt5-qtwebkit-5.6.1-3.b889f46git.el7.x86_64 (epel)
Error: Package: qt5-qtquickcontrols2-5.6.1-2.el7.x86_64 (@epel)
Error: Package: qt5-qtquickcontrols2-5.6.1-2.el7.x86_64 (@epel)
Error: Package: kf5-kdeclarative-5.36.0-1.el7.x86_64 (epel)

Are in EPEL testing updates, so:

yum --enablerepo=epel-testing update

These:

Error: Package: gvfs-1.30.4-3.el7.x86_64 (rhel-x86_64-server-7)Transaction check error:
  file /usr/lib64/gstreamer-1.0/libgstopus.so from install of gstreamer1-plugins-bad-1:1.4.5-5.el7.x86_64 conflicts with file from package gstreamer1-plugins-base-1.10.4-1.el7.x86_64

are some of the packages that are rebased in RHEL 7.4. I’ve created a temporary repository for those, it will disappear once CentOS 7.4 is released as the packages will be integrated in the main multimedia repository. You can install it through:

yum-config-manager \
  --add-repo=https://negativo17.org/repos/rhel74-temp/rhel74-temp.repo

With the above repository it is possible to install all the other multimedia packages.

Skype repository removal

Skype 4.3 is 32 bit only, is now obsolete and has been superseded by a package that actually lists proper dependencies. It is also one of the packages that required one of the above WebKit rebuilds in i686 form for RHEL/CentOS 7 x86_64.

If you have it installed, just remove it with:

yum remove webkitgtk.i686

The repository has been deleted; to install the new Skype provided version, just head to the following official link.