[FIXED] create dask local cluster and share it among different Jupiter notebook

Issue

For Dask on a single machine, I would need a global Memory limit. Since i am running/testing a lot of dask application. I created a separate process

from dask.distributed import Client
client = Client(memory_limit='12GB')  

And try to connect to it in another process using

client = Client('127.0.0.1:38719')

However, this line stagnates there.. Did I do anything wrong?

Solution

The easiest thing to do is to write the scheduler file that contains connection information:

from dask.distributed import Client
client = Client(memory_limit='12GB')
client.write_scheduler_file("dask_scheduler.json")

In a different notebook, you would use:

from dask.distributed import Client
client = Client(scheduler_file='dask_scheduler.json')

Answered By - SultanOrazbayev