SolvedTheano Ubuntu 16.04 and CUDA

Hi all,

I have started working with Theano recently and I have encountered a problem when using Theano on a freshly installed Ubuntu Mate 16.04 platform. I have a NVIDIA K2200M video card which is CUDA-capable and is correctly installed, as the nvidia-smi command shows:

+------------------------------------------------------+                       
| NVIDIA-SMI 361.42     Driver Version: 361.42         |                       
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  Quadro K2200M       Off  | 0000:01:00.0     Off |                  N/A |
| N/A   31C    P8    N/A /  N/A |    212MiB /  2047MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID  Type  Process name                               Usage      |
|=============================================================================|
|    0      1090    G   /usr/lib/xorg/Xorg                             200MiB |
|    0      7931    G   /usr/lib/firefox/firefox                         1MiB |
+-----------------------------------------------------------------------------+

I have firstly installed NVIDIA-CUDA-toolkit from the Ubuntu Mate repo, getting the 7.5.18 version and then ran the code to test the GPU provided in www.deeplearning.net with the usual flags THEANO_FLAGS='device=gpu,floatX=float32 but the following error arose:

['nvcc', '-shared', '-O3', '-m64', '-Xcompiler', '-DCUDA_NDARRAY_CUH=c72d035fdf91890f3b36710688069b2e,-DNPY_NO_DEPRECATED_API=NPY_1_7_API_VERSION,-fPIC,-fvisibility=hidden', '-Xlinker', '-rpath,/home/bee/.theano/compiledir_Linux-4.4--generic-x86_64-with-Ubuntu-16.04-xenial-x86_64-2.7.11+-64/cuda_ndarray', '-I/usr/local/lib/python2.7/dist-packages/theano/sandbox/cuda', '-I/usr/local/lib/python2.7/dist-packages/numpy/core/include', '-I/usr/include/python2.7', '-I/usr/local/lib/python2.7/dist-packages/theano/gof', '-o', '/home/bee/.theano/compiledir_Linux-4.4--generic-x86_64-with-Ubuntu-16.04-xenial-x86_64-2.7.11+-64/cuda_ndarray/cuda_ndarray.so', 'mod.cu', '-L/usr/lib', '-lcublas', '-lpython2.7', '-lcudart']
ERROR (theano.sandbox.cuda): Failed to compile cuda_ndarray.cu: ('nvcc return status', 1, 'for cmd', 'nvcc -shared -O3 -m64 -Xcompiler -DCUDA_NDARRAY_CUH=c72d035fdf91890f3b36710688069b2e,-DNPY_NO_DEPRECATED_API=NPY_1_7_API_VERSION,-fPIC,-fvisibility=hidden -Xlinker -rpath,/home/bee/.theano/compiledir_Linux-4.4--generic-x86_64-with-Ubuntu-16.04-xenial-x86_64-2.7.11+-64/cuda_ndarray -I/usr/local/lib/python2.7/dist-packages/theano/sandbox/cuda -I/usr/local/lib/python2.7/dist-packages/numpy/core/include -I/usr/include/python2.7 -I/usr/local/lib/python2.7/dist-packages/theano/gof -o /home/bee/.theano/compiledir_Linux-4.4--generic-x86_64-with-Ubuntu-16.04-xenial-x86_64-2.7.11+-64/cuda_ndarray/cuda_ndarray.so mod.cu -L/usr/lib -lcublas -lpython2.7 -lcudart')
WARNING (theano.sandbox.cuda): CUDA is installed, but device gpu is not available  (error: cuda unavailable)

I have also tried using the flag device=cuda, as suggested in the Issue 4384. Firstly I thought it was my OS' fault as installing NVIDIA-CUDA-toolkit from the official Ubuntu Mate repo did not create any folder /usr/local/cuda-X.Y, so I could not follow the instructions given in "Using the GPU". Because of that I installed the NVIDIA-CUDA-toolkit following a guide to install CUDA in Ubuntu 16.04 and I could install everything as sugested in the aforementioned guide to using the GPU, in the right folder. Nevertheless, exactly the same error arose. I have reasons to believe it has something to do with Theano's implementation rather than a bad Ubuntu integration between the GPU and the NVIDIA-CUDA-toolkit as when I installed CUDA from the official NVIDIA webpage, I could run seamlessly any CUDA sample free of hassle but Theano didn't seem to find the GPU.

I have also tried to follow any possible piece of advice given here and in the Theano Google Group (running sudo nvidia-smi, running an example before trying to compute anything, etc.).

I tried (with my limited programming skills) to track the problem and it seems that the problem is found when compiling the cuda_ndarray in the theano.sandbox.cuda module. The strangest thing is that even when installing the CUDA toolking from the repo, which installs nvcc in /usr/bin for some unknown reason), the command nvcc_compiler.is_nvcc_available() returns True so the CUDA compiler is found but, whereas nvcc seems to be able to compile CUDA code, it cannot compile cuda_ndarray.

Any advice?

17 Answers

✔️Accepted Answer

There is no problem with Theano. I tried it and we need to work around 2 problems that aren't dependent of Theano.

  1. nvcc don't support the default g++ version. Follow this to fix this:

sudo apt-get install g++-4.9

sudo update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-4.9 20
sudo update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-5 10

sudo update-alternatives --install /usr/bin/g++ g++ /usr/bin/g++-4.9 20
sudo update-alternatives --install /usr/bin/g++ g++ /usr/bin/g++-5 10

sudo update-alternatives --install /usr/bin/cc cc /usr/bin/gcc 30
sudo update-alternatives --set cc /usr/bin/gcc

sudo update-alternatives --install /usr/bin/c++ c++ /usr/bin/g++ 30
sudo update-alternatives --set c++ /usr/bin/g++

  1. There is a bug in the version of libc. You can work around it by using this extra Theano flag:
    (see #4369)

nvcc.flags=-D_FORCE_INLINES

Other Answers:

Just for the record I was able to get this to work perfectly on Ubuntu 16.04 (with packaged drivers and CUDA) by updating ~/.theanorc to be

[global]
device=gpu
floatX=float32

[nvcc]
flags=-D_FORCE_INLINES

Then ran the script at http://deeplearning.net/software/theano/tutorial/using_gpu.html#using-gpu to confirm.