[FIXED] create volume in docker compose base on client directory path

Issue

I want to use a local folder for example dataset in my docker container. Also I have a python request module in which client send to API server a directory path. I want to use this path in docker compose as volume to use it in docker container.

How to send this path to docker compose. API request is :

def train():
    url = 'http://127.0.0.1:8000/train'  # server url
    data = {'label': ['fire'],
            'image_path': '/path/to/directory',
            'label_path': '/path/to/directory',
            'image_size': 640,
            'validation_split': 0.2,
            'save_dir': 'results/'}
    r = requests.post(url, json=data)
    print(r.text)

how to send local path 'image_path' and 'label_path'to docker compose and use it as volumes. my docker compose :

version: "3.8"
services:
    inference:
        container_name: "inference"
        build: "./backbone"
        ports:
            - '5000:5000'
        volumes:
            - ./backbone:/code
            # - ./volumes/weights:/weights
        command: "python3 app.py"
        environment:
            - PORT=5000
            - MODEL_PATH=/code/weights/best.pt
        ipc: host
        shm_size: 1024M

    train:
        container_name: "train"
        build: "./train"
        shm_size: '2gb'
        command: "python3 main_train.py"
        environment:
            - RESPONSE_URL=http://127.0.0.1:8000/response
            - LOGGER_URL=http://127.0.0.1:8000/logger
            - PORT=8000
            - IS_LOGGER_ON=False
        ports:
            - '8000:8000'
        volumes:
            - ./train:/code
        ipc: host

Solution

You need a couple of changes to make this work successfully.

In the server code, you need to not accept a full path. There are a number of security concerns around doing this (can you retrieve the application code? the file of database credentials? system files like /etc/passwd?). Instead, set it to have a configurable data path, and only accept files within that path.

DATA_PATH = os.environ.get('DATA_PATH', 'data') # default relative to current directory

def handle(data):
  if '/' in data['image_file']:
    raise InvalidInputError()
  image_path = os.path.join(DATA_PATH, data['image_file'])
  ...

When you set this up in a Docker image, you can specify a fixed path for that data path. Using a simple path in a subdirectory of the container filesystem root is fine.

# train/Dockerfile
FROM python
...
ENV DATA_DIR=/data        # set this variable only because the application wants it
RUN mkdir -p "$DATA_DIR"  # create an empty directory by default
CMD ["./main_train.py"]

Now when you launch this in Compose, you know the (fixed) container-side path that will hold the data directory, so you can mount content there.

services:
  train:
    build: ./train
    ports:
      - '8000:8000'
    volumes:
      - ./dataset_3:/data

It's usually the case that details like the filesystem layout in an image, paths for injected data or configuration, and the port that the service inside the container uses are fixed properties of the image. So it's safe to specify "the data will always be in /data" and use that as a mount point; you do not need to specify it as a variable anywhere in the Docker setup. Similarly, you should not need to set a $PORT environment variable in your Compose setup since the container-side port number will generally be a fixed property of your image.

Answered By - David Maze