[FIXED] Replace runfile with exec/subprocess.call while passing a namespace

Issue

I have a python script that accepts an argument, say test.py 1 and generates some data. A second script experiment.py launches test.py repeatedly. I achieve this using runfile in Spyder IDE:

So, experiment.py reads something like:

experiments = {}
for i in range(10):
    experiments[str(i)] = {}
    runfile(`test.py`, args=str(i), wdir=`/path/to/script/`, namespace=experiments[str(i)])

## some post-processing using data from the dict experiments

This allows me to store variables of test.py for each call in a separate namespace. Everything works well, but (here comes my question):

How can I replace runfile with exec or subprocess.call and be able to pass a namespace and an argument list?

The objective is to run the script from command line rather than Spyder. Just to clarify, I am aware that the best way to perform such a task is to convert test.py into a function. However, I am trying to make-do with the current script test.py which is quite long.

Solution

Modify test.py to save its variables to a file and then loading the file in experiment.py instead of using exec use subprocess.run so that you can run the script from the command line and avoid using the Spyder-specific runfile function.

test.py will look like this

import sys
import json

arg = int(sys.argv[1])

# Your original code here

variables_to_save = {'variable1': variable1, 'variable2': variable2}
with open(f"variables_{arg}.json", "w") as f:
    json.dump(variables_to_save, f)

and experiment.py

import subprocess
import json

experiments = {}
for i in range(10):
    experiments[str(i)] = {}
    
    subprocess.run(["python", "test.py", str(i)], cwd="/path/to/script/")
    
    with open(f"variables_{i}.json", "r") as f:
        experiments[str(i)] = json.load(f)

Answered By - Saxtheowl