[FIXED] Paramiko SSH connection does not have access to GPU with tensorflow

Issue

Using Python's paramiko I try to access a server I own remotely via an SSH connection. This server has a python script that I execute via my SSH connection (I actually execute a .sh file that calls this python script). The first line of this python script is:

print("Devices:", tf.config.experimental.list_physical_devices())

Here I only see:

Devices: [PhysicalDevice(name='/physical_device:CPU:0', device_type='CPU')]

If I execute the same .sh file after opening the SSH connection with PuTTY that same line of code generates the following result:

Devices: [PhysicalDevice(name='/physical_device:CPU:0', device_type='CPU'), PhysicalDevice(name='/physical_device:GPU:0', device_type='GPU')]

Here it does show the GPU as available. I suppose this is related to the way I open the paramiko SSH connection. This is the code:

ssh = paramiko.SSHClient()  
ssh.load_system_host_keys()
ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())

ssh.connect(ip, port=port, username=u, password=pass)

cmd_to_execute = "./" + shFileName + ".sh"

ssh_stdin, ssh_stdout, ssh_stderr = ssh.exec_command(cmd_to_execute)
    
for line in ssh_stdout:
    print(line.strip('\n'))
for line in ssh_stderr:
    print('ERR: ' + line.strip('\n'))
    
ssh.close()

Why do I not have access to the GPU when the SSH connection is established using paramiko?

EDIT 1

If I allow tensorflow logs to be displayed, I get this error: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory

And others that look more or less like this one but changing the .11.0

On the other hand, if I use PuTTY it loads the libraries correctly: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcudart.so.11.0

Solution

Following the comments made by Martin Prikryl and DR. Snoopy I was able to solve the issue by modifying the startup scripts to set the same PATH for all sessions. I did this by modifying two files.

$HOME/.profile file original code had this part

if [ -n "$BASH_VERSION" ]; then
    # include .bashrc if it exists
    if [ -f "$HOME/.bashrc" ]; then
    . "$HOME/.bashrc"
    fi
fi

and I changed it to:

# include .bashrc if it exists
if [ -f "$HOME/.bashrc" ]; then
. "$HOME/.bashrc"
fi

$HOME/.bashrc original code had this part:

# If not running interactively, don't do anything
case $- in
    *i*) ;;
      *) return;;
esac

and I changed it to:

# If not running interactively, don't do anything
#case $- in
#    *i*) ;;
#      *) return;;
#esac

This way now the same libraries are loaded without taking into account which way I established the connection.

Answered By - Bobbick