[Question] How to correctly integrate FDAM (especially FreqScale) into Mask DINO (Detectron2)?

[justdays]

Hi @Linwei-Chen,

Thank you for your great work and for releasing the code for FDAM! I'm currently trying to integrate it into the official Mask DINO (Detectron2 version) for instance segmentation, following the results reported in Table 3 of your ICCV paper.

I understand the standard FDAM integration shown in the README is for a ViT backbone Block:

1. Replace Attention with AttentionwithAttInv.
2. Add GroupDynamicScale after Attention.
3. Add GroupDynamicScale after MLP.

However, Mask DINO's Transformer Decoder uses object queries (tgt) of shape (num_queries, batch, dim), which lack a spatial structure (H, W). It seems GroupDynamicScale (which uses rfft2 and requires a (B, C, H, W) input) cannot be directly applied to these sequences.

1. AttInv: I successfully integrated AttentionwithAttInv into DeformableTransformerDecoderLayer to replace self_attn.
2. FreqScale: I am stuck here. I see GroupDynamicScale requires spatial dimensions.

Could you please clarify the intended integration path for Mask DINO?

• Is the FreqScale component used at all for the Mask DINO experiments? If so, what is the correct location? For example, should it be applied to the memory features (Key/Value) from the Pixel Decoder (which do have spatial structure)?
• Or does the Mask DINO integration in the paper only use the AttInv part?

Any guidance or pointers to relevant configs/files in the repo would be greatly appreciated. Thank you in advance for your time and help!

Best,
zetao

[Linwei-Chen]

Hi @zetao,

Thank you for your interest in our work and for trying to integrate FDAM into Mask DINO! Your detailed questions are very clear.

To answer your question directly: Yes, the FreqScale (GroupDynamicScale) component is indeed used. The core of the solution is that you simply need to pass the spatial information (H and W) from Mask DINO into the module.

Here is the clarification on how to properly integrate it:

1. Where to apply FreqScale:
You are absolutely correct that object queries (num_queries, batch, dim) lack a 2D spatial structure. Therefore, 2D FFT operations cannot be applied to them. Instead of applying GroupDynamicScale to the object queries, it should be applied to the features that inherently have spatial structures, such as the memory features (Key/Value) from the Pixel Decoder or the multi-scale feature maps from the backbone.

2. How to pass H and W:
In Mask DINO (which inherits from Deformable DETR), the spatial dimensions of the flattened feature maps are stored in the spatial_shapes variable. You can resolve the shape issue with the following steps:

• Modify the Forward Pass: Update your GroupDynamicScale function/module to accept H and W as explicit input arguments.
• Reshape Before FFT: Use the H and W (obtained from spatial_shapes or the original feature maps) to unflatten the sequence back to (B, C, H, W).
• Apply and Flatten: Perform the rfft2 operation, apply the FreqScale, and then flatten the tensor back to its original sequence shape (L, B, C) or (B, L, C) to continue the Mask DINO pipeline.

Summary:
The AttInv can be used in the cross-attention/self-attention mechanisms, while FreqScale must be applied to components where H and W can be extracted and passed in (like the image feature maps).

Hope this clears up the confusion! Please feel free to let me know if you run into any further issues during the implementation.

Best,
Linwei

[justdays]

That perfectly clears it up. I really appreciate your help!