[FIXED] Jupyter shell commands in a function

Issue

I'm attempting to create a function to load Sagemaker models within a jupyter notebook using shell commands. The problem arises when I try to store the function in a utilities.py file and source it for multiple notebooks.

Here are the contents of the utilities.py file that I am sourcing in a jupyter lab notebook.

def get_aws_sagemaker_model(model_loc):
    """
    TO BE USED IN A JUPYTER NOTEBOOK
    
    extracts a sagemaker model that has ran and been completed
    
    deletes the copied items and leaves you with the model
    
    note that you will need to have the package installed with correct
    versioning for whatever model you have trained
    ie. if you are loading an XGBoost model, have XGBoost installed
    
    Args:
        model_loc (str) : s3 location of the model including file name
        
    Return:
        model: unpacked and loaded model
    """ 
    
    import re
    import tarfile
    import os
    import pickle as pkl

    # extract the filename from beyond the last backslash
    packed_model_name = re.search("(.*\/)(.*)$" , model_loc)[2]
    
    # copy and paste model file locally
    command_string = "!aws s3 cp {model_loc} ."
    exec(command_string)
    
    # use tarfile to extract
    tar = tarfile.open(packed_model_name)
    
    # extract filename from tarfile
    unpacked_model_name = tar.getnames()[0]
    
    tar.extractall()
    tar.close()
    
    model = pkl.load(open(unpacked_model_name, 'rb'))
    
    # cleanup copied files and unpacked model
    os.remove(packed_model_name)
    os.remove(unpacked_model_name)
    
    return model

The error occurs when trying to execute the command string:

Traceback (most recent call last):

  File "/home/ec2-user/anaconda3/envs/env/lib/python3.8/site-packages/IPython/core/interactiveshell.py", line 3444, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)

  File "/tmp/ipykernel_10889/996524724.py", line 1, in <module>
    model = get_aws_sagemaker_model("my-model-loc")

  File "/home/ec2-user/SageMaker/env/src/utilities/model_helper_functions.py", line 167, in get_aws_sagemaker_model
    exec(command_string)

  File "<string>", line 1
    !aws s3 cp my-model-loc .
    ^
SyntaxError: invalid syntax

It seems like jupyter isn't receiving the command before exec checks the syntax. Is there a way around this besides copying the function into each jupyter notebook that I use?

Thank you!

Solution

You can use transform_cell method of IPython's shell to transform the IPython syntax into valid plain-Python:

from IPython import get_ipython
ipython = get_ipython()

code = ipython.transform_cell('!ls')
print(code)

which will show:

get_ipython().system('!ls')

You can use that as input for exec:

exec(code)

Or directly:

exec(ipython.transform_cell('!ls'))

Answered By - krassowski