[FIXED] How to read python files from a directory and search for functions

Issue

I am looking to make a dictionary of python file names as the value and the functions defined in each file as the key. I am easily able to make a list of the filenames. For example:

    for child in Path('./loaders').iterdir():
        if child.is_file():
            f = child.name
            if f != "__init__.py":
                self.beamline_options.append(f.split('.')[0])

I also have a variable defined as self.beamline_loaders = {}, which I would like to add to this loop above in such a way:

    for child in Path('./loaders').iterdir():
        if child.is_file():
            f = child.name
            if f != "__init__.py":
                self.beamline_options.append(f.split('.')[0])
                self.beamline_loaders[f] = [def1, def2, def3, ...]

where def1, def2, ... would be the names of the functions in each file. All of this is within the same module. I am using this dictionary to populate a QComboBox based on the selection of a first combo-box (which consists of just the names in the list of beamline options). This could in theory be hard coded, but I would prefer not to hard code it.

Solution

It sounds like you're asking for a simple plugin system that can dynamically load functions from any python files found in a target directory. Presumably, you will also need to execute those functions at some point, rather than just display their names. In which case, one solution would be to use importlib to directly import the modules from the source files, and then use dir to scan the module's attributes for suitable function objects.

Below is a basic demo that implements that, and also shows how to populate some combo-boxes with the module and function names. The function objects themselves are loaded on demand and stored in a dict, and can be displayed via the "Load" button. (NB: the demo assumes that the "loaders" directory is in the same directory as the demo script):

import sys, types, importlib.util
from collections import defaultdict
from pathlib import Path
from PyQt5 import QtWidgets

BASEDIR = Path(__file__).resolve().parent.joinpath('loaders')

class Window(QtWidgets.QWidget):
    def __init__(self):
        super().__init__()
        self.buttonLoad = QtWidgets.QPushButton('Load')
        self.buttonLoad.clicked.connect(self.handleLoad)
        self.comboOptions = QtWidgets.QComboBox()
        self.comboOptions.currentTextChanged.connect(self.handleOptions)
        self.comboLoaders = QtWidgets.QComboBox()
        layout = QtWidgets.QHBoxLayout(self)
        layout.addWidget(self.comboOptions)
        layout.addWidget(self.comboLoaders)
        layout.addWidget(self.buttonLoad)
        self.scanOptions()

    def scanOptions(self):
        self.beamline_options = {}
        self.beamline_loaders = defaultdict(dict)
        for entry in BASEDIR.iterdir():
            if entry.is_file() and entry.match('[!_.]*.py'):
                self.beamline_options[entry.stem] = entry
        self.comboOptions.addItems(self.beamline_options)

    def getLoaders(self, option):
        if option not in self.beamline_loaders:
            entry = self.beamline_options[option]
            modname = f'{BASEDIR.name}.{option}'
            spec = importlib.util.spec_from_file_location(modname, entry)
            module = importlib.util.module_from_spec(spec)
            sys.modules[modname] = module
            spec.loader.exec_module(module)
            for name in dir(module):
                if not name.startswith('_'):
                    obj = getattr(module, name)
                    if isinstance(obj, types.FunctionType):
                        self.beamline_loaders[option][name] = obj
        return self.beamline_loaders[option]

    def getLoader(self, option, loader):
        if option and loader:
            return self.getLoaders(option)[loader]

    def handleOptions(self, option):
        self.comboLoaders.clear()
        self.comboLoaders.addItems(self.getLoaders(option))

    def handleLoad(self):
        option = self.comboOptions.currentText()
        loader = self.comboLoaders.currentText()
        func = self.getLoader(option, loader)
        QtWidgets.QMessageBox.information(self, 'Load', repr(func))

if __name__ == '__main__':

    app = QtWidgets.QApplication(['Test'])
    window = Window()
    window.setGeometry(600, 100, 400, 50)
    window.show()
    app.exec_()

Answered By - ekhumoro