Solvednvidia docker docker: Error response from daemon: Unknown runtime specified nvidia.

turian
1454

I know that this is a common bug, but I have read a variety of issues + bug reports as well as https://github.com/nvidia/nvidia-container-runtime#docker-engine-setup and still cannot resolve the issue.

1. Issue or feature description

$ docker run --runtime=nvidia --rm nvidia/cuda nvidia-smi
docker: Error response from daemon: Unknown runtime specified nvidia.

2. Steps to reproduce the issue

I follow the Ubuntu instructions here: https://github.com/NVIDIA/nvidia-docker
I have verified that nvidia-docker is not installed, and nvidia-docker2 is.

My daemon.json is reasonable and set by the package install:

$ cat /etc/docker/daemon.json
{

    "runtimes": {
        "nvidia": {
            "path": "nvidia-container-runtime",
            "runtimeArgs": []
        }
    }
}

I try to restart the dockerd as suggested here https://github.com/nvidia/nvidia-container-runtime#docker-engine-setup:

sudo pkill -SIGHUP dockerd

But still get:

$ docker run --runtime=nvidia --rm nvidia/cuda nvidia-smi
docker: Error response from daemon: Unknown runtime specified nvidia.

3. Information to attach (optional if deemed irrelevant)

  • Kernel version from uname -a
    Linux Ubuntu-1804-bionic-64-minimal 4.15.0-36-generic #39-Ubuntu SMP Mon Sep 24 16:19:09 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
  • Any relevant kernel output lines from dmesg
    n/a
  • Driver information from nvidia-smi -a
==============NVSMI LOG==============

Timestamp                           : Sat Oct  6 19:32:52 2018
Driver Version                      : 410.48

Attached GPUs                       : 1
GPU 00000000:01:00.0
    Product Name                    : GeForce GTX 1080
    Product Brand                   : GeForce
    Display Mode                    : Disabled
    Display Active                  : Disabled
    Persistence Mode                : Disabled
    Accounting Mode                 : Disabled
    Accounting Mode Buffer Size     : 4000
    Driver Model
        Current                     : N/A
        Pending                     : N/A
    Serial Number                   : N/A
    GPU UUID                        : GPU-761e2aa7-1578-4bdc-12c8-d7cac834813a
    Minor Number                    : 0
    VBIOS Version                   : 86.04.17.00.01
    MultiGPU Board                  : No
    Board ID                        : 0x100
    GPU Part Number                 : N/A
    Inforom Version
        Image Version               : G001.0000.01.03
        OEM Object                  : 1.1
        ECC Object                  : N/A
        Power Management Object     : N/A
    GPU Operation Mode
        Current                     : N/A
        Pending                     : N/A
    GPU Virtualization Mode
        Virtualization mode         : None
    IBMNPU
        Relaxed Ordering Mode       : N/A
    PCI
        Bus                         : 0x01
        Device                      : 0x00
        Domain                      : 0x0000
        Device Id                   : 0x1B8010DE
        Bus Id                      : 00000000:01:00.0
        Sub System Id               : 0x119E10DE
        GPU Link Info
            PCIe Generation
                Max                 : 3
                Current             : 3
            Link Width
                Max                 : 16x
                Current             : 16x
        Bridge Chip
            Type                    : N/A
            Firmware                : N/A
        Replays since reset         : 0
        Tx Throughput               : 1000 KB/s
        Rx Throughput               : 1000 KB/s
    Fan Speed                       : 41 %
    Performance State               : P0
    Clocks Throttle Reasons
        Idle                        : Not Active
        Applications Clocks Setting : Not Active
        SW Power Cap                : Active
        HW Slowdown                 : Not Active
            HW Thermal Slowdown     : Not Active
            HW Power Brake Slowdown : Not Active
        Sync Boost                  : Not Active
        SW Thermal Slowdown         : Not Active
        Display Clock Setting       : Not Active
    FB Memory Usage
        Total                       : 8119 MiB
        Used                        : 0 MiB
        Free                        : 8119 MiB
    BAR1 Memory Usage
        Total                       : 256 MiB
        Used                        : 2 MiB
        Free                        : 254 MiB
    Compute Mode                    : Default
    Utilization
        Gpu                         : 4 %
        Memory                      : 0 %
        Encoder                     : 0 %
        Decoder                     : 0 %
    Encoder Stats
        Active Sessions             : 0
        Average FPS                 : 0
        Average Latency             : 0
    FBC Stats
        Active Sessions             : 0
        Average FPS                 : 0
        Average Latency             : 0
    Ecc Mode
        Current                     : N/A
        Pending                     : N/A
    ECC Errors
        Volatile
            Single Bit
                Device Memory       : N/A
                Register File       : N/A
                L1 Cache            : N/A
                L2 Cache            : N/A
                Texture Memory      : N/A
                Texture Shared      : N/A
                CBU                 : N/A
                Total               : N/A
            Double Bit
                Device Memory       : N/A
                Register File       : N/A
                L1 Cache            : N/A
                L2 Cache            : N/A
                Texture Memory      : N/A
                Texture Shared      : N/A
                CBU                 : N/A
                Total               : N/A
        Aggregate
            Single Bit
                Device Memory       : N/A
                Register File       : N/A
                L1 Cache            : N/A
                L2 Cache            : N/A
                Texture Memory      : N/A
                Texture Shared      : N/A
                CBU                 : N/A
                Total               : N/A
            Double Bit
                Device Memory       : N/A
                Register File       : N/A
                L1 Cache            : N/A
                L2 Cache            : N/A
                Texture Memory      : N/A
                Texture Shared      : N/A
                CBU                 : N/A
                Total               : N/A
    Retired Pages
        Single Bit ECC              : N/A
        Double Bit ECC              : N/A
        Pending                     : N/A
    Temperature
        GPU Current Temp            : 43 C
        GPU Shutdown Temp           : 99 C
        GPU Slowdown Temp           : 96 C
        GPU Max Operating Temp      : N/A
        Memory Current Temp         : N/A
        Memory Max Operating Temp   : N/A
    Power Readings
        Power Management            : Supported
        Power Draw                  : 35.90 W
        Power Limit                 : 180.00 W
        Default Power Limit         : 180.00 W
        Enforced Power Limit        : 180.00 W
        Min Power Limit             : 90.00 W
        Max Power Limit             : 180.00 W
    Clocks
        Graphics                    : 974 MHz
        SM                          : 974 MHz
        Memory                      : 5005 MHz
        Video                       : 873 MHz
    Applications Clocks
        Graphics                    : N/A
        Memory                      : N/A
    Default Applications Clocks
        Graphics                    : N/A
        Memory                      : N/A
    Max Clocks
        Graphics                    : 1911 MHz
        SM                          : 1911 MHz
        Memory                      : 5005 MHz
        Video                       : 1708 MHz
    Max Customer Boost Clocks
        Graphics                    : N/A
    Clock Policy
        Auto Boost                  : N/A
        Auto Boost Default          : N/A
    Processes                       : None
  • Docker version from docker version
Client:
 Version:           18.06.1-ce
 API version:       1.38
 Go version:        go1.10.3
 Git commit:        e68fc7a
 Built:             Tue Aug 21 17:24:51 2018
 OS/Arch:           linux/amd64
 Experimental:      false

Server:
 Engine:
  Version:          18.06.1-ce
  API version:      1.38 (minimum version 1.12)
  Go version:       go1.10.3
  Git commit:       e68fc7a
  Built:            Tue Aug 21 17:28:38 2018
  OS/Arch:          linux/amd64
  Experimental:     false
  • NVIDIA packages version from dpkg -l '*nvidia*' or rpm -qa '*nvidia*'
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name                                                              Version                               Architecture                          Description
+++-=================================================================-=====================================-=====================================-========================================================================================================================================
un  libgldispatch0-nvidia                                             <none>                                <none>                                (no description available)
ii  libnvidia-cfg1-410:amd64                                          410.48-0ubuntu1                       amd64                                 NVIDIA binary OpenGL/GLX configuration library
un  libnvidia-cfg1-any                                                <none>                                <none>                                (no description available)
un  libnvidia-common                                                  <none>                                <none>                                (no description available)
ii  libnvidia-common-410                                              410.48-0ubuntu1                       all                                   Shared files used by the NVIDIA libraries
ii  libnvidia-compute-410:amd64                                       410.48-0ubuntu1                       amd64                                 NVIDIA libcompute package
ii  libnvidia-container-tools                                         1.0.0-1                               amd64                                 NVIDIA container runtime library (command-line tools)
ii  libnvidia-container1:amd64                                        1.0.0-1                               amd64                                 NVIDIA container runtime library
un  libnvidia-decode                                                  <none>                                <none>                                (no description available)
ii  libnvidia-decode-410:amd64                                        410.48-0ubuntu1                       amd64                                 NVIDIA Video Decoding runtime libraries
un  libnvidia-encode                                                  <none>                                <none>                                (no description available)
ii  libnvidia-encode-410:amd64                                        410.48-0ubuntu1                       amd64                                 NVENC Video Encoding runtime library
un  libnvidia-fbc1                                                    <none>                                <none>                                (no description available)
ii  libnvidia-fbc1-410:amd64                                          410.48-0ubuntu1                       amd64                                 NVIDIA OpenGL-based Framebuffer Capture runtime library
un  libnvidia-gl                                                      <none>                                <none>                                (no description available)
ii  libnvidia-gl-410:amd64                                            410.48-0ubuntu1                       amd64                                 NVIDIA OpenGL/GLX/EGL/GLES GLVND libraries and Vulkan ICD
un  libnvidia-ifr1                                                    <none>                                <none>                                (no description available)
ii  libnvidia-ifr1-410:amd64                                          410.48-0ubuntu1                       amd64                                 NVIDIA OpenGL-based Inband Frame Readback runtime library
un  nvidia-304                                                        <none>                                <none>                                (no description available)
un  nvidia-340                                                        <none>                                <none>                                (no description available)
un  nvidia-384                                                        <none>                                <none>                                (no description available)
un  nvidia-390                                                        <none>                                <none>                                (no description available)
ii  nvidia-compute-utils-410                                          410.48-0ubuntu1                       amd64                                 NVIDIA compute utilities
ii  nvidia-container-runtime                                          2.0.0+docker18.06.1-1                 amd64                                 NVIDIA container runtime
ii  nvidia-container-runtime-hook                                     1.4.0-1                               amd64                                 NVIDIA container runtime hook
ii  nvidia-cuda-dev                                                   9.1.85-3ubuntu1                       amd64                                 NVIDIA CUDA development files
ii  nvidia-cuda-doc                                                   9.1.85-3ubuntu1                       all                                   NVIDIA CUDA and OpenCL documentation
ii  nvidia-cuda-gdb                                                   9.1.85-3ubuntu1                       amd64                                 NVIDIA CUDA Debugger (GDB)
ii  nvidia-cuda-toolkit                                               9.1.85-3ubuntu1                       amd64                                 NVIDIA CUDA development toolkit
ii  nvidia-dkms-410                                                   410.48-0ubuntu1                       amd64                                 NVIDIA DKMS package
un  nvidia-dkms-kernel                                                <none>                                <none>                                (no description available)
un  nvidia-docker                                                     <none>                                <none>                                (no description available)
ii  nvidia-docker2                                                    2.0.3+docker18.06.1-1                 all                                   nvidia-docker CLI wrapper
un  nvidia-driver                                                     <none>                                <none>                                (no description available)
ii  nvidia-driver-410                                                 410.48-0ubuntu1                       amd64                                 NVIDIA driver metapackage
un  nvidia-driver-binary                                              <none>                                <none>                                (no description available)
un  nvidia-kernel-common                                              <none>                                <none>                                (no description available)
ii  nvidia-kernel-common-410                                          410.48-0ubuntu1                       amd64                                 Shared files used with the kernel module
un  nvidia-kernel-source                                              <none>                                <none>                                (no description available)
ii  nvidia-kernel-source-410                                          410.48-0ubuntu1                       amd64                                 NVIDIA kernel source package
un  nvidia-legacy-340xx-vdpau-driver                                  <none>                                <none>                                (no description available)
un  nvidia-libopencl1                                                 <none>                                <none>                                (no description available)
un  nvidia-libopencl1-dev                                             <none>                                <none>                                (no description available)
ii  nvidia-modprobe                                                   410.48-0ubuntu1                       amd64                                 Load the NVIDIA kernel driver and create device files
ii  nvidia-opencl-dev:amd64                                           9.1.85-3ubuntu1                       amd64                                 NVIDIA OpenCL development files
un  nvidia-opencl-icd                                                 <none>                                <none>                                (no description available)
un  nvidia-persistenced                                               <none>                                <none>                                (no description available)
ii  nvidia-prime                                                      0.8.8                                 all                                   Tools to enable NVIDIA's Prime
ii  nvidia-profiler                                                   9.1.85-3ubuntu1                       amd64                                 NVIDIA Profiler for CUDA and OpenCL
ii  nvidia-settings                                                   410.48-0ubuntu1                       amd64                                 Tool for configuring the NVIDIA graphics driver
un  nvidia-settings-binary                                            <none>                                <none>                                (no description available)
un  nvidia-smi                                                        <none>                                <none>                                (no description available)
un  nvidia-utils                                                      <none>                                <none>                                (no description available)
ii  nvidia-utils-410                                                  410.48-0ubuntu1                       amd64                                 NVIDIA driver support binaries
un  nvidia-vdpau-driver                                               <none>                                <none>                                (no description available)
ii  nvidia-visual-profiler                                            9.1.85-3ubuntu1                       amd64                                 NVIDIA Visual Profiler for CUDA and OpenCL
ii  xserver-xorg-video-nvidia-410                                     410.48-0ubuntu1                       amd64                                 NVIDIA binary Xorg driver
  • NVIDIA container library version from nvidia-container-cli -V
version: 1.0.0
build date: 2018-09-20T20:19+00:00
build revision: 881c88e2e5bb682c9bb14e68bd165cfb64563bb1
build compiler: x86_64-linux-gnu-gcc-7 7.3.0
build platform: x86_64
build flags: -D_GNU_SOURCE -D_FORTIFY_SOURCE=2 -DNDEBUG -std=gnu11 -O2 -g -fdata-sections -ffunction-sections -fstack-protector -fno-strict-aliasing -fvisibility=hidden -Wall -Wextra -Wcast-align -Wpointer-arith -Wmissing-prototypes -Wnonnull -Wwrite-strings -Wlogical-op -Wformat=2 -Wmissing-format-attribute -Winit-self -Wshadow -Wstrict-prototypes -Wunreachable-code -Wconversion -Wsign-conversion -Wno-unknown-warning-option -Wno-format-extra-args -Wno-gnu-alignof-expression -Wl,-zrelro -Wl,-znow -Wl,-zdefs -Wl,--gc-sections
  • NVIDIA container library logs (see troubleshooting)
    n/a

  • Docker command, image and tag used
    See above

22 Answers

✔️Accepted Answer

sngjuk
1290

I've also installed correctly, but forgot to restart daemon in ubuntu. it may resolve your error.

sudo systemctl daemon-reload
sudo systemctl restart docker

Other Answers:

maybe --gpus all instead of --runtime=nvidia.

I fixed this with:

sudo apt install -y nvidia-docker2
sudo systemctl daemon-reload
sudo systemctl restart docker

@francoisdelarbre Did you ever check your system have the only one of docker service? I have fixed the same issue by disabling the snap.docker.dockerd.service, it is found by the command systemctl list-unit-files | grep enable | grep docker

@cdpath

$ sudo apt install -y nvidia-docker2
Reading package lists... Done
Building dependency tree       
Reading state information... Done
E: Unable to locate package nvidia-docker2

Setting the GPG and remote repo for the package:
$ curl -s -L https://nvidia.github.io/nvidia-docker/gpgkey | sudo apt-key add -
$ distribution=$(. /etc/os-release;echo $ID$VERSION_ID)
$ curl -s -L https://nvidia.github.io/nvidia-docker/$distribution/nvidia-docker.list | sudo tee /etc/apt/sources.list.d/nvidia-docker.list

Save that file and refresh the package list
$ sudo apt-get update

Install nvidia-docker2 and reload the Docker configurations
$ sudo apt-get install -y nvidia-docker2
$ sudo pkill -SIGHUP dockerd

More Issues: