Not implemented on the CPU

[wang-kangkang]

i meet this error:

Traceback (most recent call last):
File "wkktest.py", line 22, in
res = model(im_t.unsqueeze(0))
File "/ssd2/wangkangkang/anaconda3_torch/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
return forward_call(*input, **kwargs)
File "/ssd2/wangkangkang/projects/test_detreg/another/DETReg-main/models/deformable_detr.py", line 138, in forward
return self.forward_samples(samples)
File "/ssd2/wangkangkang/projects/test_detreg/another/DETReg-main/models/deformable_detr.py", line 167, in forward_samples
query_embeds)
File "/ssd2/wangkangkang/anaconda3_torch/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
return forward_call(*input, **kwargs)
File "/ssd2/wangkangkang/projects/test_detreg/another/DETReg-main/models/deformable_transformer.py", line 153, in forward
memory = self.encoder(src_flatten, spatial_shapes, level_start_index, valid_ratios, lvl_pos_embed_flatten, mask_flatten)
File "/ssd2/wangkangkang/anaconda3_torch/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
return forward_call(*input, **kwargs)
File "/ssd2/wangkangkang/projects/test_detreg/another/DETReg-main/models/deformable_transformer.py", line 256, in forward
output = layer(output, pos, reference_points, spatial_shapes, level_start_index, padding_mask)
File "/ssd2/wangkangkang/anaconda3_torch/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
return forward_call(*input, **kwargs)
File "/ssd2/wangkangkang/projects/test_detreg/another/DETReg-main/models/deformable_transformer.py", line 221, in forward
src2 = self.self_attn(self.with_pos_embed(src, pos), reference_points, src, spatial_shapes, level_start_index, padding_mask)
File "/ssd2/wangkangkang/anaconda3_torch/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
return forward_call(*input, **kwargs)
File "/ssd2/wangkangkang/projects/test_detreg/another/DETReg-main/models/ops/modules/ms_deform_attn.py", line 113, in forward
value, input_spatial_shapes, input_level_start_index, sampling_locations, attention_weights, self.im2col_step)
File "/ssd2/wangkangkang/projects/test_detreg/another/DETReg-main/models/ops/functions/ms_deform_attn_func.py", line 26, in forward
value, value_spatial_shapes, value_level_start_index, sampling_locations, attention_weights, ctx.im2col_step)
RuntimeError: Not implemented on the CPU

another error:
i run these:
cd ./models/ops
sh ./make.sh
and when i run python test.py, it report:
RuntimeError: CUDA out of memory. Tried to allocate 7.50 GiB (GPU 0; 31.75 GiB total capacity; 22.52 GiB already allocated; 656.44 MiB free; 23.52 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF

my device is v100.
how to run the detreg code?

[yongmayer]

I had the same issue. Could you please shred some light?

[vadimkantorov]

There is a ms_deform_attn_core_pytorch function that supports CPU.

In the meanwhile, you can patch

DETReg/models/ops/modules/ms_deform_attn.py

Line 112 in 32f39af

output = MSDeformAttnFunction.apply(

as follows:

from ..functions import MSDeformAttnFunction, ms_deform_attn_core_pytorch

        if value.is_cuda:
            output = MSDeformAttn.MSDeformAttnFunction.apply(value, input_spatial_shapes, input_level_start_index, sampling_locations, attention_weights, self.im2col_step)
        else:
            output = ms_deform_attn_core_pytorch(value, input_spatial_shapes, sampling_locations, attention_weights)

From my earlier checks, it's correct, but please test it as well

[wang-kangkang]

it because the author's google colab code wrong. you need change the input variable to cuda format for model run, it means you need change

im_t, _ = transforms(im, None)
res = model(im_t.unsqueeze(0))

to

im_t, _ = transforms(im, None)
im_t = im_t.unsqueeze(0)
res = model(im_t.cuda())

I can't remember whether there is the same error in other positions, if there is, use the same method to deal with

[yongmayer]

it because the author's google colab code wrong. you need change the input variable to cuda format for model run, it means you need change

im_t, _ = transforms(im, None)
res = model(im_t.unsqueeze(0))

to

im_t, _ = transforms(im, None)
im_t = im_t.unsqueeze(0)
res = model(im_t.cuda())

I can't remember whether there is the same error in other positions, if there is, use the same method to deal with

Thanks! This make the "Not implemented on the CPU" error disappear. But I had another error when I tried to run author's colab code and would appreciate your help.

---

RuntimeError Traceback (most recent call last)
/home/yma/prog/DETReg/Class_Agnostic_Detection_with_DETReg.ipynb Cell 9' in <cell line: 7>()
5 #res = model(im_t.unsqueeze(0))
6 im_t = im_t.unsqueeze(0)
----> 7 res = model(im_t.cuda())
8 scores = torch.sigmoid(res['pred_logits'][..., 1])
9 pred_boxes = res['pred_boxes']

File ~/prog/anaconda3/envs/physics/lib/python3.10/site-packages/torch/nn/modules/module.py:1110, in Module._call_impl(self, *input, **kwargs)
1106 # If we don't have any hooks, we want to skip the rest of the logic in
1107 # this function, and just call forward.
1108 if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks
1109 or _global_forward_hooks or _global_forward_pre_hooks):
-> 1110 return forward_call(*input, **kwargs)
1111 # Do not call functions when jit is used
1112 full_backward_hooks, non_full_backward_hooks = [], []

File ~/prog/DETReg/models/deformable_detr.py:138, in DeformableDETR.forward(self, samples)
136 if not isinstance(samples, NestedTensor):
137 samples = nested_tensor_from_tensor_list(samples)
--> 138 return self.forward_samples(samples)

File ~/prog/DETReg/models/deformable_detr.py:141, in DeformableDETR.forward_samples(self, samples)
140 def forward_samples(self, samples):
--> 141 features, pos = self.backbone(samples)
142 srcs = []
143 masks = []

File ~/prog/anaconda3/envs/physics/lib/python3.10/site-packages/torch/nn/modules/module.py:1110, in Module._call_impl(self, *input, **kwargs)
1106 # If we don't have any hooks, we want to skip the rest of the logic in
1107 # this function, and just call forward.
1108 if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks
1109 or _global_forward_hooks or _global_forward_pre_hooks):
-> 1110 return forward_call(*input, **kwargs)
1111 # Do not call functions when jit is used
1112 full_backward_hooks, non_full_backward_hooks = [], []

File ~/prog/DETReg/models/backbone.py:134, in Joiner.forward(self, tensor_list)
133 def forward(self, tensor_list: NestedTensor):
--> 134 xs = self0
135 out: List[NestedTensor] = []
136 pos = []

File ~/prog/anaconda3/envs/physics/lib/python3.10/site-packages/torch/nn/modules/module.py:1110, in Module._call_impl(self, *input, **kwargs)
1106 # If we don't have any hooks, we want to skip the rest of the logic in
1107 # this function, and just call forward.
1108 if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks
1109 or _global_forward_hooks or _global_forward_pre_hooks):
-> 1110 return forward_call(*input, **kwargs)
1111 # Do not call functions when jit is used
1112 full_backward_hooks, non_full_backward_hooks = [], []

File ~/prog/DETReg/models/backbone.py:88, in BackboneBase.forward(self, tensor_list)
86 def forward(self, tensor_list):
87 if isinstance(tensor_list, NestedTensor):
---> 88 xs = self.body(tensor_list.tensors)
89 out: Dict[str, NestedTensor] = {}
90 for name, x in xs.items():

File ~/prog/anaconda3/envs/physics/lib/python3.10/site-packages/torch/nn/modules/module.py:1110, in Module._call_impl(self, *input, **kwargs)
1106 # If we don't have any hooks, we want to skip the rest of the logic in
1107 # this function, and just call forward.
1108 if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks
1109 or _global_forward_hooks or _global_forward_pre_hooks):
-> 1110 return forward_call(*input, **kwargs)
1111 # Do not call functions when jit is used
1112 full_backward_hooks, non_full_backward_hooks = [], []

File ~/prog/anaconda3/envs/physics/lib/python3.10/site-packages/torchvision/models/_utils.py:63, in IntermediateLayerGetter.forward(self, x)
61 out = OrderedDict()
62 for name, module in self.items():
---> 63 x = module(x)
64 if name in self.return_layers:
65 out_name = self.return_layers[name]

File ~/prog/anaconda3/envs/physics/lib/python3.10/site-packages/torch/nn/modules/module.py:1110, in Module._call_impl(self, *input, **kwargs)
1106 # If we don't have any hooks, we want to skip the rest of the logic in
1107 # this function, and just call forward.
1108 if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks
1109 or _global_forward_hooks or _global_forward_pre_hooks):
-> 1110 return forward_call(*input, **kwargs)
1111 # Do not call functions when jit is used
1112 full_backward_hooks, non_full_backward_hooks = [], []

File ~/prog/anaconda3/envs/physics/lib/python3.10/site-packages/torch/nn/modules/conv.py:447, in Conv2d.forward(self, input)
446 def forward(self, input: Tensor) -> Tensor:
--> 447 return self._conv_forward(input, self.weight, self.bias)

File ~/prog/anaconda3/envs/physics/lib/python3.10/site-packages/torch/nn/modules/conv.py:443, in Conv2d._conv_forward(self, input, weight, bias)
439 if self.padding_mode != 'zeros':
440 return F.conv2d(F.pad(input, self._reversed_padding_repeated_twice, mode=self.padding_mode),
441 weight, bias, self.stride,
442 _pair(0), self.dilation, self.groups)
--> 443 return F.conv2d(input, weight, bias, self.stride,
444 self.padding, self.dilation, self.groups)

RuntimeError: Input type (torch.cuda.FloatTensor) and weight type (torch.FloatTensor) should be the same

[wang-kangkang]

1112 full_backward_hooks, non_full_backward_hooks = [], []

I can't remember if I meet this error. as a matter of experience, this error means the weight's datatype is float32, but the input datatype is float64. you need debug and check the datatype, then change it

[vadimkantorov]

I would say that input tensor device does not match mode weight tensor device (one is cpu, another is cuda), so more debugging / patching is needed

[wang-kangkang]

I would say that input tensor device does not match mode weight tensor device (one is cpu, another is cuda), so more debugging / patching is needed

Your solution is indeed more meaningful, but I recommend changing the input variable device on colab first so that novice users can run successfully

[refael-kohen]

I also encountered this problem, I have changed the code as follows, and now it is work:

In section: Define model and load checkpoint:

1.  Add this line after line 12:

    model.cuda()

2. Change 'cpu' to 'cuda' in line 13:
   checkpoint = 
torch.hub.load_state_dict_from_url("https://github.com/amirbar/DETReg/releases/download/1.0.0/checkpoint_imagenet.pth", 
progress=True, map_location=torch.device('cuda'))

In section: Load image and predict class agnostic results:

Change the lines 4-5, as @wang-kangkang wrote:

im_t, _ = transforms(im, None)
im_t = im_t.unsqueeze(0)
res = model(im_t.cuda())

In section: Plot results:
Add .cuda() in the end of the line:

pred_boxes_ = box_cxcywh_to_xyxy(pred_boxes) * torch.Tensor([img_w, img_h, img_w, img_h]).cuda()

Other problems that I solved:

• Need to add 'model' in line 10 of first section:
  args = {'model': 'deformable_detr', ....
  (or another model)

• Need to add this package to the requirement.txt file:
  requests

[MrCrightH]

Would you like to share how to train with a single GPU.

[MrCrightH]

I would say that input tensor device does not match mode weight tensor device (one is cpu, another is cuda), so more debugging / patching is needed

hello! Do you know how to train this model with only one GPU？