nvidia-smi shows ERR! on one device. What does this mean?

My device 1 is showing an error that I've never seen. The same code is running on device 0. It won't keep, dies with no error, on device 1 (due to the ERR!).

Is this a hardware or software thing? What is it called besides 'ERR!'?

I ran the device query code similar to / available in many CUDA starter tutorials, and the output is normal. https://devblogs.nvidia.com/parallelforall/how-query-device-properties-and-handle-errors-cuda-cc/

my keel is:

uname -a
Linux manjaro 4.1.18-2-MANJARO #1 SMP PREEMPT Sat Feb 27 23:18:59 UTC 2016 x86_64 GNU/Linux


~$ nvidia-smi 
Wed May 18 19:16:29 2016
+------------------------------------------------------+                       
| NVIDIA-SMI 361.28     Driver Version: 361.28         |                       
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce GTX 970     Off  | 0000:01:00.0     Off |                  N/A |
| 23%   69C    P2   140W / 170W |   1207MiB /  4092MiB |     99%      Default |
+-------------------------------+----------------------+----------------------+
|   1  GeForce GTX 970     Off  | 0000:02:00.0     Off |                  N/A |
|ERR!   58C    P0   ERR! / 170W |     16MiB /  4092MiB |      1%      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID  Type  Process name                               Usage          |
|=============================================================================|
|    0       645    C   /usr/local/bin/code-gpu-75                     1190MiB |
+-----------------------------------------------------------------------------+

Recent Questions...