[FIXED] How to pass the script path to %run magic command as a variable in databricks notebook?

Issue

I want to run a notebook in databricks from another notebook using %run. Also I want to be able to send the path of the notebook that I'm running to the main notebook as a parameter.
The reason for not using dbutils.notebook.run is that I'm storing nested dictionaries in the notebook that's called and I wanna use them in the main notebook.

I'm looking for Something like:

path = "/References/parameterDefinition/schemaRepository"

%run <path variable>

Solution

Unfortunately it's impossible to pass the path in %run as variable. You can pass variable as parameter only, and it's possible only in combination with with widgets - you can see the example in this answer. In this case you can have all your definitions in one notebook, and depending on the passed variable you can redefine the dictionary.

There will be a new functionality coming in the next months (approximately, see public roadmap webinar for more details) that will allow to import notebooks as libraries using the import statement. Potentially you can emulate the same functionality by exporting the notebook into the file on disk using the Export command of Workspace API, decoding the data & importing file's content, for example, if you have notebook called module1 with content

my_cool_dict = {"key1": "abc", "key2": 123}

then you can import it as following:

import requests
import base64
import os


api_url = dbutils.notebook.entry_point.getDbutils().notebook().getContext().apiUrl().get()
host_token = "your_PAT_token"

path = "/Users/..../module1"

# fetch notebook
response = requests.get(f"{api_url}/api/2.0/workspace/export",
    json = {"format": "SOURCE", "path": path},
    headers={"Authorization": f"Bearer {host_token}"}
  ).json()

# decode base64 encoded content
data = base64.b64decode(response["content"].encode("ascii"))

# write the file & __init__.py, so directory will considered a module
dir = os.path.join("/tmp","my_modules")
if not os.path.exists(dir):
    os.mkdir(dir)

with open(os.path.join(dir, os.path.split(path)[-1]+".py"), "wb") as f:
  f.write(data)
with open(os.path.join(dir, "__init__.py"), "wb") as f:
  f.write("\n".encode("ascii"))

# add our directory into system path
import sys
sys.path.append(dir)

# import notebook
from module1 import my_cool_dict

and see that we got our variable:

Answered By - Alex Ott