[FIXED] RuntimeError: CUDA error: CUBLAS_STATUS_INVALID_VALUE on PyTorch Lightning

Issue

I am working on a tutorial of PyTorch Lightning.

https://pytorch-lightning.readthedocs.io/en/stable/starter/introduction.html

Because I wanted to try GPU training, I changed definition of trainer as below.

trainer = pl.Trainer(limit_train_batches=100, max_epochs=1, gpus=1)

Then I got the following error.

RuntimeError                              Traceback (most recent call last)
Cell In [3], line 4
      1 # train the model (hint: here are some helpful Trainer arguments for rapid idea iteration)
      2 # trainer = pl.Trainer(limit_train_batches=100, max_epochs=3)
      3 trainer = pl.Trainer(limit_train_batches=100, max_epochs=3, accelerator='gpu', devices=1)
----> 4 trainer.fit(model=autoencoder, train_dataloaders=train_loader)

File ~/miniconda3/envs/py38-cu116/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py:696, in Trainer.fit(self, model, train_dataloaders, val_dataloaders, datamodule, ckpt_path)
    677 r"""
    678 Runs the full optimization routine.
    679 
   (...)
    693     datamodule: An instance of :class:`~pytorch_lightning.core.datamodule.LightningDataModule`.
    694 """
    695 self.strategy.model = model
--> 696 self._call_and_handle_interrupt(
    697     self._fit_impl, model, train_dataloaders, val_dataloaders, datamodule, ckpt_path
    698 )

File ~/miniconda3/envs/py38-cu116/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py:650, in Trainer._call_and_handle_interrupt(self, trainer_fn, *args, **kwargs)
    648         return self.strategy.launcher.launch(trainer_fn, *args, trainer=self, **kwargs)
    649     else:
--> 650         return trainer_fn(*args, **kwargs)
    651 # TODO(awaelchli): Unify both exceptions below, where `KeyboardError` doesn't re-raise
    652 except KeyboardInterrupt as exception:

[...]

File ~/miniconda3/envs/py38-cu116/lib/python3.8/site-packages/pytorch_lightning/core/module.py:1450, in LightningModule.backward(self, loss, optimizer, optimizer_idx, *args, **kwargs)
   1433 def backward(
   1434     self, loss: Tensor, optimizer: Optional[Optimizer], optimizer_idx: Optional[int], *args, **kwargs
   1435 ) -> None:
   1436     """Called to perform backward on the loss returned in :meth:`training_step`. Override this hook with your
   1437     own implementation if you need to.
   1438 
   (...)
   1448             loss.backward()
   1449     """
-> 1450     loss.backward(*args, **kwargs)

File ~/miniconda3/envs/py38-cu116/lib/python3.8/site-packages/torch/_tensor.py:396, in Tensor.backward(self, gradient, retain_graph, create_graph, inputs)
    387 if has_torch_function_unary(self):
    388     return handle_torch_function(
    389         Tensor.backward,
    390         (self,),
   (...)
    394         create_graph=create_graph,
    395         inputs=inputs)
--> 396 torch.autograd.backward(self, gradient, retain_graph, create_graph, inputs=inputs)

File ~/miniconda3/envs/py38-cu116/lib/python3.8/site-packages/torch/autograd/__init__.py:173, in backward(tensors, grad_tensors, retain_graph, create_graph, grad_variables, inputs)
    168     retain_graph = create_graph
    170 # The reason we repeat same the comment below is that
    171 # some Python versions print out the first line of a multi-line function
    172 # calls in the traceback and some print out the last line
--> 173 Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
    174     tensors, grad_tensors_, retain_graph, create_graph, inputs,
    175     allow_unreachable=True, accumulate_grad=True)

RuntimeError: CUDA error: CUBLAS_STATUS_INVALID_VALUE when calling `cublasSgemm( handle, opa, opb, m, n, k, &alpha, a, lda, b, ldb, &beta, c, ldc)`

The only thing I added to the tutorial code is gpus=1, so I cannot figure out what is the problem. How can I fix this?

FYI, I tried giving devices=1, accelerator='ddp' instead of gpus=1, and got a following error.

ValueError: You selected an invalid accelerator name: `accelerator='ddp'`. Available names are: cpu, cuda, hpu, ipu, mps, tpu.

My environments are:

• CUDA 11.6
• Python 3.8.13
• PyTorch 1.12.1
• PyTorch Lightning 1.7.7

Solution

Though I'm not sure about the reason, the issue disappeared when I used Python 3.10 instead of 3.8.

Answered By - Yusuke196