[FIXED] pytorch subset on dataloader decrease number of data

Issue

i'm using random_split()

dataset_train, dataset_valid = random_split(dataset, [int(len(dataset) * 0.8), int(len(dataset) * 0.2+1)]) 

len(dataset_train) is 1026 and len(dataset_valid) is 257 but put this two vriable into dataloader decrease number of data

loader_train = DataLoader(dataset_train, batch_size=batch_size, shuffle=True, num_workers=0)
loader_val = DataLoader(dataset_valid, batch_size=batch_size, shuffle=True, num_workers=0)
print (len(loader_train))
print (len(loader_val))

output is :

257, 65

I don't know why decrease the size of dataset. please any help. thanks.

Solution

You can use data.Subset which works as a wrapper for data.Dataset instances. You need to provide a generator or list of indices that you want to retain in the constructed dataset

1. Here is a minimal setup example:

   >>> dataset_train = data.TensorDataset(torch.arange(100))
   

2. Construct the subset by wrapping dataset_train:

   >>> subset = data.Subset(ds, range(10)) # will select the first 10 of dataset_train
   

3. Finally construct your dataloader:

   >>> loader_train = data.DataLoader(subset, batch_size=2)
   

---

As an illustration here is loader_train:

>>> for x in dl:
...    print(x)
[tensor([0, 1])]
[tensor([2, 3])]
[tensor([4, 5])]
[tensor([6, 7])]
[tensor([8, 9])]

Answered By - Ivan