@cuDnn无法找到算法进行卷积
最近在复现一个项目，但是在运行训练文件时一直在报Unable to find a valid cuDNN algorithm to run convolution 这个问题，搜了一下貌似这个问题很小众，也没看到什么好的解决办法。目前已经尝试过更换cuDNN版本到8.0.3还是会有同样的错误，现在版本为7.6.5 ,还更改了pytoch版本1.5和1.6都试过了，依旧报错。 当前配置情况： albumentations0.4.3 Cython>=0.25 future0.18.0 lmdb0.97 loguru0.3.2 mypy_extensions0.4.1 lvis0.5.1 numpy1.16.4 opencv-python4.1.2.30 scikit-learn0.21.3 sentencepiece0.1.85 torch1.6.0 torchvision0.7 tqdm==4.36.0
错误代码： Traceback (most recent call last): File “scripts/pretrain_virtex.py”, line 250, in args=(_A, ), File “/media/gc/168606691BA7B4F0/virtex/virtex/utils/distributed.py”, line 84, in launch _job_worker(0, job_fn, 1, 1, 0, dist_url, args) File “/media/gc/168606691BA7B4F0/virtex/virtex/utils/distributed.py”, line 118, in _job_worker job_fn(*args) File “scripts/pretrain_virtex.py”, line 158, in main output_dict = model(batch) File “/home/gc/anaconda3/envs/zlt/lib/python3.6/site-packages/torch/nn/modules/module.py”, line 550, in call result = self.forward(*input, **kwargs) File “/media/gc/168606691BA7B4F0/virtex/apex/amp/_initialize.py”, line 197, in new_fwd **applier(kwargs, input_caster)) File “/media/gc/168606691BA7B4F0/virtex/virtex/models/captioning.py”, line 117, in forward visual_features = self.visual(batch[“image”]) File “/home/gc/anaconda3/envs/zlt/lib/python3.6/site-packages/torch/nn/modules/module.py”, line 550, in call result = self.forward(*input, **kwargs) File “/media/gc/168606691BA7B4F0/virtex/virtex/modules/visual_backbones.py”, line 139, in forward out = layer(image) if idx == 0 else layer(out) File “/home/gc/anaconda3/envs/zlt/lib/python3.6/site-packages/torch/nn/modules/module.py”, line 550, in call result = self.forward(*input, **kwargs) File “/home/gc/anaconda3/envs/zlt/lib/python3.6/site-packages/torch/nn/modules/container.py”, line 100, in forward input = module(input) File “/home/gc/anaconda3/envs/zlt/lib/python3.6/site-packages/torch/nn/modules/module.py”, line 550, in call result = self.forward(*input, **kwargs) File “/home/gc/anaconda3/envs/zlt/lib/python3.6/site-packages/torchvision/models/resnet.py”, line 112, in forward out = self.conv3(out) File “/home/gc/anaconda3/envs/zlt/lib/python3.6/site-packages/torch/nn/modules/module.py”, line 550, in call result = self.forward(*input, **kwargs) File “/home/gc/anaconda3/envs/zlt/lib/python3.6/site-packages/torch/nn/modules/conv.py”, line 351, in forward return self._conv_forward(input, self.weight) File “/home/gc/anaconda3/envs/zlt/lib/python3.6/site-packages/torch/nn/modules/conv.py”, line 347, in _conv_forward self.padding, self.dilation, self.groups) RuntimeError: Unable to find a valid cuDNN algorithm to run convolution
