Issue
I've been fighting to parallelise some code for a while due to Pickling errors. I'm aware of proposed work arounds using dill or cloudpickle (e.g. here or here or here), but I didn't get any of these to work. In my frustration, I have gone to the concurrent.futures official document page and tried running the minimum working example they give for a ProcessPoolExecutor (see here). Specifically:
import concurrent.futures
import math
PRIMES = [
112272535095293,
112582705942171,
112272535095293,
115280095190773,
115797848077099,
1099726899285419]
def is_prime(n):
if n < 2:
return False
if n == 2:
return True
if n % 2 == 0:
return False
sqrt_n = int(math.floor(math.sqrt(n)))
for i in range(3, sqrt_n + 1, 2):
if n % i == 0:
return False
return True
def main():
with concurrent.futures.ProcessPoolExecutor() as executor:
for number, prime in zip(PRIMES, executor.map(is_prime, PRIMES)):
print('%d is prime: %s' % (number, prime))
if __name__ == '__main__':
main()
Running this gives the now all to familiar Pickling error:
_pickle.PicklingError: Can't pickle <function is_prime at 0x000001F26AAD6E50>: attribute lookup is_prime on __main__ failed
I can't really imagine that the official documentation would be wrong, so what am I doing wrong here? If the documentation is somehow outdated, what is the way to work around this?
I'm using Python 3.9 in PyCharm on Windows if that matters.
I have tried running the code in Spyder and it works without any error. I have changed the run configuration within PyCharm to use exactly the same python interpretator (verified with sys.executable) and environment (as verified with pip list) as Spyder is using, but yet the error is still thrown.
Solution
Uncheck the "Run with Python Console" option in the run-configuration used to execute this script (top-menu "Run" -> "Edit Configurations...")
Answered By - Pavel Karateev
0 comments:
Post a Comment
Note: Only a member of this blog may post a comment.