[FIXED] How to make `tqdm.set_postfix_str` faster?

Issue

When I use tqdm, I want to track a score caculated during the iterations and thus I used the set_postfix_str method. But I found it could dramatically slow down the program:

from tqdm import tqdm
import time
N = 10000

pbar = tqdm(range(N), total=N, desc="N")
start = time.time()
for i in pbar:
    pass
print("w/o set_postfix_str: {:.2f}".format(time.time() - start))

pbar = tqdm(range(N), total=N, desc="N")
start = time.time()
for i in pbar:
    pass
    pbar.set_postfix_str(s="{}".format(i))
print("w set_postfix_str: {:.2f}".format(time.time() - start))

Output:

N: 100%|██████████| 10000/10000 [00:00<00:00, 1239195.20it/s]
w/o set_postfix_str: 0.01
N: 100%|██████████| 10000/10000 [00:12<00:00, 774.67it/s, 9999]
w set_postfix_str: 12.91

I tried to set the miniters in tqdm but it didn't help. How to make it faster?

Solution

how about setting up the bar_format parameter to your liking

import time
from tqdm import tqdm
N = 10000000 #I increased the number to see something in my machine

pbar = tqdm(range(N), total=N, desc="N")
start = time.time()
for i in pbar:
    pass
print("w/o set_postfix_str: {:.2f}".format(time.time() - start))

pbar = tqdm(range(N), total=N, desc="N",bar_format='{l_bar}{bar}| {n_fmt}/{total_fmt} [{elapsed}<{remaining}, {rate_fmt} {n_fmt}]')
start = time.time()
for i in pbar:
    pass
print("w custom bar: {:.2f}".format(time.time() - start))

the bar_format by default is '{l_bar}{bar}{r_bar}' in this case we want to change the {r_bar} part which in turn default to '| {n_fmt}/{total_fmt} [{elapsed}<{remaining}, ' '{rate_fmt}{postfix}]', I try some combination with just passing the postfix arguments but with no success, so explicity putting all in the bar_format was the solution I found, and thus we change the postfix part to what we want, that in your example is equivalent to n_fmt and thus the new bar_format is

'{l_bar}{bar}| {n_fmt}/{total_fmt} [{elapsed}<{remaining}, {rate_fmt} {n_fmt}]'

and a quick test

C:\Users\copperfield\Desktop>test.py
N: 100%|███████████████████████████| 10000000/10000000 [00:01<00:00, 5168526.86it/s]
w/o set_postfix_str: 1.93
N: 100%|██████████████████| 10000000/10000000 [00:01<00:00, 5320703.94it/s 10000000]
w custom bar: 1.88

C:\Users\copperfield\Desktop>

and turns out to be a little bit faster funnily enough

---

with further tinkering, and also looking at the source code of tqdm, and in order to no use this custom bar thing, because said that we want something that is calculated in the loop and not something that the progress bar keep track of, like in the previous example, this set_postfix_str also take a second argument refresh that by default is True, just change it to False, it still would be slow compared to the simple case, but not by that extreme amount, in my test only by like ~3 time slower

pbar = tqdm(range(N),total=N, desc="N")
start = time.time()
for i in pbar:
    pbar.set_postfix_str(f"{i}",refresh=False)

print("w set_postfix_str refresh=False: {:.2f}".format(time.time() - start))

and after taking a peek at the source code, the only thing this does is set up an attribute, so lets cut the middle man and do it directly

pbar = tqdm(range(N),total=N, desc="N")
start = time.time()
for i in pbar:
    pbar.postfix=f"{i}"

print("w .postfix: {:.2f}".format(time.time() - start))

this improve the time an little an now is just some ~1.7 times slower

and a quick test

C:\Users\copperfield\Desktop>test.py
N: 100%|██████████████████████████████████████████████████████████████| 10000000/10000000 [00:01<00:00, 5467326.75it/s]
w/o set_postfix_str: 1.83
N: 100%|█████████████████████████████████████████████████████| 10000000/10000000 [00:01<00:00, 5497884.24it/s 10000000]
w custom bar: 1.82
N: 100%|█████████████████████████████████████████████████████| 10000000/10000000 [00:05<00:00, 1874176.95it/s, 9999999]
w set_postfix_str refresh=False: 5.34
N: 100%|█████████████████████████████████████████████████████| 10000000/10000000 [00:03<00:00, 3088319.84it/s, 9999999]
w .postfix: 3.24

C:\Users\copperfield\Desktop>

Answered By - Copperfield