[FIXED] In distributed computing, what are world size and rank?

Issue

I've been reading through some documentation and example code with the end goal of writing scripts for distributed computing (running PyTorch), but the concepts confuse me.

Let's assume that we have a single node with 4 GPUs, and we want to run our script on those 4 GPUs (i.e. one process per GPU). In such a scenario, what are the rank world size and rank? I often find the explanation for world size: Total number of processes involved in the job, so I assume that that is four in our example, but what about rank?

To explain it further, another example with multiple nodes and multiple GPUs could be useful, too.

Solution

These concepts are related to parallel computing. It would be helpful to learn a little about parallel computing, e.g., MPI.

You can think of world as a group containing all the processes for your distributed training. Usually, each GPU corresponds to one process. Processes in the world can communicate with each other, which is why you can train your model distributedly and still get the correct gradient update. So world size is the number of processes for your training, which is usually the number of GPUs you are using for distributed training.

Rank is the unique ID given to a process, so that other processes know how to identify a particular process. Local rank is the a unique local ID for processes running in a single node, this is where my view differs with @zihaozhihao.

Let's take a concrete example. Suppose we run our training in 2 servers (some articles also call them nodes) and each server/node has 4 GPUs. The world size is 4*2=8. The ranks for the processes will be [0, 1, 2, 3, 4, 5, 6, 7]. In each node, the local rank will be [0, 1, 2, 3].

I have also written a post about MPI collectives and basic concepts. The link is here.

Answered By - jdhao