[FIXED] Python Ram usage increases after new thread starts

Issue

In a True while loop, I'm starting a new Thread which in turn starts a shell Subprocess (python script). Due to other reasons, the python scripts need to be called using the shell subprocess. The subprocess runs for a certain time and then quits.

After each thread is created the main program (cmd) requires more RAM usage.

Code for starting thread:

while True:
  cameras = camera_queue.get_next_cameras()
  session_id = uuid.uuid4()

  task = threading.Thread(target=start_thread, args=(cameras, f'{session_id}'))
  task.start()
  time.sleep(camera_queue.batch_size * CAMERA_INTERVAL)

Subprocess code (start_thread):

subprocess.Popen(
  f'start /MIN /wait python Backend\\Recorder\\secondary.py "{cameras}" "{logger_id}"',
  shell=True,
)

This program will start around 1000 subprocesses per hour. I have checked and the subprocesses do close and there never exists more subprocesses than 1000.

Is there a way to clean up the usage? If you force quit the current cmd usage the ram usage instantly disappears while the shells stay open. After running the program for 1 hour the ram usage is at 3GB and after around 10 hours around 15-20GB.

Solution

@RolandSmith solution half worked. For some reason, by just passing the creation flag the scripts would never run / would instantly give a popen exception. A solution to this was to pass the shell=True still.

subprocess.Popen(
  ["python",
    "Backend\\Recorder\\dcr_unit.py",
    f"{cameras}",
    f"{session_id}"],
  creationflags=subprocess.CREATE_NO_WINDOW,
  shell=True
)

I noticed that for some reason the popen objects would still run in the background even though the instance was done executing its code. But when passing the creation flags and then running the script just through cmd the memory leak was pretty much 0. For some reason when running this script through a code editor like vscode would cause the memory leak tenfold. The program accumulated around 10GB of ram per 10000 new instances.

Answered By - Casper Kristiansson