[FIXED] Customize module search path (PYTHONPATH) via pipenv

Issue

I have a Python project consisting of a Jupyter notebook, several scripts in a bin directory and modules in a src directory, with dependencies in a Pipfile:

myproject
├── myproject.ipynb
├── Pipfile
├── Pipfile.lock
├── bin
│   ├── bar.py
│   └── foo.py
└── src
    ├── baz.py
    └── qux.py

The scripts foo.py and bar.py use the standard shebang

#!/usr/bin/env python

and can be run with pipenv shell:

mymachine:myproject myname$ pipenv shell
(myproject-U308romt) bash-3.2$ bin/foo.py
foo

However, I can't easily access the modules in src from the scripts. If I add

import src.baz as baz

to foo.py, I get:

ModuleNotFoundError: No module named 'src'

One solution I tried is to add a .env file under myproject:

PYTHONPATH=${PYTHONPATH}:${PWD}

This works thanks to pipenv's automatic loading of .env, but checking the .env file into the git distribution of the project would collide with the traditional use of .env to store secrets such as passwords -- in fact, my default .gitignore for Python projects already excludes .env for just this reason.

$ git add .env
The following paths are ignored by one of your .gitignore files:
.env
Use -f if you really want to add them.

Alternatively, I could move src under bin, but then the Jupyter notebook would have to reference the modules as bin.src.baz etc., which is also a hassle.

My current workaround is just to add a symlink:

myproject
├── Pipfile
├── Pipfile.lock
├── bin
│   ├── bar.py
│   ├── foo.py
│   └── src -> ../src
└── src
    ├── baz.py
    └── qux.py

This works, and I suppose has the benefit of being transparent, but it seems like there should be some way to leverage pipenv to solve the same problem.

Is there a portable, distributable way to put these modules on the search path?

Solution

I'm not sure there's a perfect solution for this, but in the interest of being explicit rather than implicit (PEP 20), I've decided to check in a file that needs to be sourced before running any script. This is one extra manual step but you can put this in a Makefile for instance.

env.sh

export PYTHONPATH=${PYTHONPATH}:${PWD}

Makefile

bar:
    source env.sh && pipenv run python scripts/bar.py
.PHONY: migrate

The solution is a bit similar to the approach Go takes with its GOPATH.

I think the other solutions are not as good:

• pipenv aims to solve dependencies, I could be wrong but I did not find anything related to the problem of the PYTHONPATH.
• Linking folders won't scale really well if you start having other scripts folder.

Answered By - charlax