/home/vuxuanni/.conda/envs/yxn_pytorch_1/bin/python /home/vuxuanni/code/Compaired-HIT-IIL/train_fivek_skff_no_patches.py
CUDA visible devices: 4
CUDA Device Name: NVIDIA GeForce RTX 3090
batch_size:1
Total_params: 8946598
length of dataset:10
Train loader length: 10
/home/vuxuanni/.conda/envs/yxn_pytorch_1/lib/python3.8/site-packages/cupy/cuda/compiler.py:464: UserWarning: cupy.cuda.compile_with_cache has been deprecated in CuPy v10, and will be removed in the future. Use cupy.RawModule or cupy.RawKernel instead.
warnings.warn(
Traceback (most recent call last):
File "/home/vuxuanni/code/Compaired-HIT-IIL/train_fivek_skff_no_patches.py", line 123, in <module>
train()
File "/home/vuxuanni/code/Compaired-HIT-IIL/train_fivek_skff_no_patches.py", line 67, in train
GCMModel_out_warp, rgb_mask, raw_demosaic_warp, raw_demosaic_mask = get_backwarp(x, GCMModel_out, target,
File "/home/vuxuanni/code/Compaired-HIT-IIL/model/warp_modules.py", line 65, in get_backwarp
flow = get_flow(tenFirst, tenSecond, net)
File "/home/vuxuanni/code/Compaired-HIT-IIL/model/warp_modules.py", line 82, in get_flow
flow = estimate(tenFirst, tenSecond, net)
File "/home/vuxuanni/code/Compaired-HIT-IIL/model/warp_modules.py", line 33, in estimate
input=net(tenPreprocessedFirst, tenPreprocessedSecond),
File "/home/vuxuanni/.conda/envs/yxn_pytorch_1/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
return forward_call(*input, **kwargs)
File "/home/vuxuanni/.conda/envs/yxn_pytorch_1/lib/python3.8/site-packages/torch/nn/parallel/data_parallel.py", line 166, in forward
return self.module(*inputs[0], **kwargs[0])
File "/home/vuxuanni/.conda/envs/yxn_pytorch_1/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
return forward_call(*input, **kwargs)
File "/home/vuxuanni/code/Compaired-HIT-IIL/pwc_net/pwc_net.py", line 270, in forward
objEstimate = self.netSix(tenFirst[-1], tenSecond[-1], None)
File "/home/vuxuanni/.conda/envs/yxn_pytorch_1/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
return forward_call(*input, **kwargs)
File "/home/vuxuanni/code/Compaired-HIT-IIL/pwc_net/pwc_net.py", line 147, in forward
tenVolume = torch.nn.functional.leaky_relu(input=correlation.FunctionCorrelation(
File "/home/vuxuanni/code/Compaired-HIT-IIL/pwc_net/correlation/correlation.py", line 398, in FunctionCorrelation
return _FunctionCorrelation.apply(tenFirst, tenSecond)
File "/home/vuxuanni/code/Compaired-HIT-IIL/pwc_net/correlation/correlation.py", line 316, in forward
cupy_launch('kernel_Correlation_updateOutput', cupy_kernel('kernel_Correlation_updateOutput', {
File "cupy/_util.pyx", line 64, in cupy._util.memoize.decorator.ret
File "/home/vuxuanni/code/Compaired-HIT-IIL/pwc_net/correlation/correlation.py", line 276, in cupy_launch
return cupy.cuda.compile_with_cache(strKernel).get_function(strFunction)
File "/home/vuxuanni/.conda/envs/yxn_pytorch_1/lib/python3.8/site-packages/cupy/cuda/compiler.py", line 468, in compile_with_cache
return _compile_module_with_cache(*args, **kwargs)
File "/home/vuxuanni/.conda/envs/yxn_pytorch_1/lib/python3.8/site-packages/cupy/cuda/compiler.py", line 496, in _compile_module_with_cache
return _compile_with_cache_cuda(
File "/home/vuxuanni/.conda/envs/yxn_pytorch_1/lib/python3.8/site-packages/cupy/cuda/compiler.py", line 565, in _compile_with_cache_cuda
mod.load(cubin)
File "cupy/cuda/function.pyx", line 263, in cupy.cuda.function.Module.load
File "cupy/cuda/function.pyx", line 265, in cupy.cuda.function.Module.load
File "cupy_backends/cuda/api/driver.pyx", line 210, in cupy_backends.cuda.api.driver.moduleLoadData
File "cupy_backends/cuda/api/driver.pyx", line 60, in cupy_backends.cuda.api.driver.check_status
cupy_backends.cuda.api.driver.CUDADriverError: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
Traceback (most recent call last):
File "cupy_backends/cuda/api/driver.pyx", line 217, in cupy_backends.cuda.api.driver.moduleUnload
File "cupy_backends/cuda/api/driver.pyx", line 60, in cupy_backends.cuda.api.driver.check_status
cupy_backends.cuda.api.driver.CUDADriverError: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered
Exception ignored in: 'cupy.cuda.function.Module.__dealloc__'
Traceback (most recent call last):
File "cupy_backends/cuda/api/driver.pyx", line 217, in cupy_backends.cuda.api.driver.moduleUnload
File "cupy_backends/cuda/api/driver.pyx", line 60, in cupy_backends.cuda.api.driver.check_status
cupy_backends.cuda.api.driver.CUDADriverError: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered 如何解决该问题
最新发布