[FIXED] Mixing shell variables and python variables in IPython '!command'

Issue

Trying to figure out whether this behaviour on IPython (v7.12.0, on Amazon SageMaker) is a bug or I'm missing some proper way / documented constraint...

Say I have some Python variables like:

NODE_VER = "v16.14.2"
NODE_DISTRO = "linux-x64"

These commands both work fine in a notebook:

!echo $PATH
# Shows **contents of system path**
!echo /usr/local/lib/nodejs/node-{NODE_VER}-{NODE_DISTRO}/bin:
# Shows /usr/local/lib/nodejs/node-v16.14.2-linux-x64/bin

...But this does not:

!echo /usr/local/lib/nodejs/node-{NODE_VER}-{NODE_DISTRO}/bin:$PATH
# Shows:
# /usr/local/lib/nodejs/node-{NODE_VER}-{NODE_DISTRO}/bin:**contents of system path**

I've tried a couple of combinations of e.g. using $NODE_VER syntax instead (which produces node--/ instead of node-{NODE_VER}-{NODE_DISTRO}/, but seems like any combination using both shell variables (PATH) and Python variables (NODE_VER/NODE_DISTRO) fails.

Can anybody help me understand why and how to work around it?

My end goal, as you might have guessed already, is to actually add this folder to the PATH rather than just echoing it - something like:

!export PATH=/usr/local/lib/nodejs/node-{NODE_VER}-{NODE_DISTRO}/bin:$PATH

Solution

How to reference both a python and environment variable in jupyter bash magic?

Try

!echo /usr/local/lib/nodejs/node-{NODE_VER}-{NODE_DISTRO}/bin:$$PATH

$$PATH forces it to use the system variable rather than try to find a Python/local one.

Various examples:

In [130]: foo = 'foo*.txt'
In [131]: HOME = 'myvar'
In [132]: !echo $foo
foo1.txt foobar0.txt foobar2.txt foobar3.txt foo.txt
In [133]: !echo $foo $HOME
foo1.txt foobar0.txt foobar2.txt foobar3.txt foo.txt myvar
In [134]: !echo $foo $$HOME
foo1.txt foobar0.txt foobar2.txt foobar3.txt foo.txt /home/paul
In [135]: !echo $foo $PWD
/home/paul/mypy
In [136]: !echo $foo $$PWD
foo1.txt foobar0.txt foobar2.txt foobar3.txt foo.txt /home/paul/mypy
In [137]: !echo {foo} $PWD
{foo} /home/paul/mypy
In [138]: !echo {foo} $$PWD
foo1.txt foobar0.txt foobar2.txt foobar3.txt foo.txt /home/paul/mypy

Any variable not locally defined forces the behavior you see:

In [139]: !echo $abc

In [140]: !echo {foo} $abc
{foo}

It may put the substitution in a try/except block, and "give up" if there's any NameError.

This substitution can occur in most of the magics, not just !.

Answered By - hpaulj