RTDETR融合OverLock中的ContMix模块

​RT-DETR使用教程 RT-DETR使用教程RT-DETR改进汇总贴RT-DETR更新汇总贴《OverLoCK An Overview-first-Look-Closely-next ConvNet with Context-Mixing Dynamic Kernels》一、 模块介绍论文链接https://arxiv.org/abs/2502.20087代码链接https://github.com/LMMMEng/OverLoCK论文速览自上而下的注意力在人类视觉系统中起着至关重要的作用其中大脑最初会获得场景的粗略概览以发现突出的线索即首先概览然后进行更仔细、更细粒度的检查即接下来仔细观察。然而现代卷积网络仍然局限于金字塔结构该结构连续对特征图进行采样以进行感受野扩展而忽略了这一关键的仿生原理。我们提出了 OverLoCK这是第一个显式包含自上而下的注意力机制的纯 ConvNet 主干架构。与金字塔式骨干网络不同我们的设计采用分支架构具有三个协同子网络1 编码低/中级特征的 Base-Net;2 一个轻量级的 Overview-Net通过粗略的全局上下文建模即首先概述产生动态的自上而下的关注;3 一个强大的 Focus-Net它执行由自上而下的注意力引导的更细粒度的感知即仔细观察下一个。为了充分释放自上而下注意力的力量我们进一步提出了一种新的上下文混合动态卷积 ContMix它可以有效地模拟长距离依赖性同时即使在输入分辨率增加时也能保留固有的局部归纳偏差解决了现有卷积中的关键限制。与现有方法相比我们的 OverLoCK 表现出显着的性能改进。例如OverLoCK-T 实现了 84.2% 的 Top-1 准确率大大超过了 ConvNeXt-B而只使用了大约三分之一的 FLOPs/参数。总结本文更新其中ConrMix模块代码及使用教程。​⭐⭐本文二创模块仅更新于付费群中往期免费教程可看下方链接⭐⭐RT-DETR更新汇总贴含免费教程文章浏览阅读264次。RT-DETR使用教程缝合教程 RT-DETR中的yaml文件详解labelimg使用教程_rt-deterhttps://xy2668825911.blog.csdn.net/article/details/143696113https://xy2668825911.blog.csdn.net/article/details/143696113二、二创融合模块2.1 相关代码# https://blog.csdn.net/StopAndGoyyy?spm1011.2124.3001.5343\ # (CVPR 2025) OverLoCK An Overview-first-Look-Closely-next ConvNet with Context-Mixing Dynamic Kernels # https://github.com/LMMMEng/OverLoCK/tree/main try: from natten.functional import na2d_av from einops import rearrange, einsum except: pass class ContMix(nn.Module): def __init__(self, dim64, ctx_dim32, kernel_size7, smk_size5, num_heads2, ): super().__init__() ctx_dim dim // 2 self.kernel_size kernel_size self.smk_size smk_size self.num_heads num_heads * 2 head_dim dim // self.num_heads self.scale head_dim ** -0.5 self.weight_query nn.Sequential( nn.Conv2d(dim // 2, dim // 2, kernel_size1, biasFalse), # 32 - 32 nn.BatchNorm2d(dim // 2), ) self.weight_key nn.Sequential( nn.AdaptiveAvgPool2d(7), nn.Conv2d(ctx_dim, dim // 2, kernel_size1, biasFalse), nn.BatchNorm2d(dim // 2), ) self.weight_proj nn.Conv2d(49, kernel_size ** 2 smk_size ** 2, kernel_size1) self.dyconv_proj nn.Sequential( nn.Conv2d(dim, dim, kernel_size1, biasFalse), nn.BatchNorm2d(dim), ) self.get_rpb() def get_rpb(self): self.rpb_size1 2 * self.smk_size - 1 self.rpb1 nn.Parameter(torch.empty(self.num_heads, self.rpb_size1, self.rpb_size1)) self.rpb_size2 2 * self.kernel_size - 1 self.rpb2 nn.Parameter(torch.empty(self.num_heads, self.rpb_size2, self.rpb_size2)) nn.init.zeros_(self.rpb1) nn.init.zeros_(self.rpb2) torch.no_grad() def generate_idx(self, kernel_size): rpb_size 2 * kernel_size - 1 idx_h torch.arange(0, kernel_size) idx_w torch.arange(0, kernel_size) idx_k ((idx_h.unsqueeze(-1) * rpb_size) idx_w).view(-1) return (idx_h, idx_w, idx_k) def apply_rpb(self, attn, rpb, height, width, kernel_size, idx_h, idx_w, idx_k): num_repeat_h torch.ones(kernel_size, dtypetorch.long) num_repeat_w torch.ones(kernel_size, dtypetorch.long) num_repeat_h[kernel_size // 2] height - (kernel_size - 1) num_repeat_w[kernel_size // 2] width - (kernel_size - 1) bias_hw (idx_h.repeat_interleave(num_repeat_h).unsqueeze(-1) * ( 2 * kernel_size - 1)) idx_w.repeat_interleave(num_repeat_w) bias_idx bias_hw.unsqueeze(-1) idx_k bias_idx bias_idx.reshape(-1, int(kernel_size ** 2)) bias_idx torch.flip(bias_idx, [0]) rpb torch.flatten(rpb, 1, 2)[:, bias_idx] rpb rpb.reshape(1, int(self.num_heads), int(height), int(width), int(kernel_size ** 2)) return attn rpb def forward(self, x): B, C, H, W x.shape query, key torch.chunk(x, 2, dim1) # 32, 32 query self.weight_query(query) * self.scale key self.weight_key(key) query rearrange(query, b (g c) h w - b g c (h w), gself.num_heads) key rearrange(key, b (g c) h w - b g c (h w), gself.num_heads) weight einsum(query, key, b g c n, b g c l - b g n l) weight rearrange(weight, b g n l - b l g n).contiguous() weight self.weight_proj(weight) weight rearrange(weight, b l g (h w) - b g h w l, hH, wW) attn1, attn2 torch.split(weight, split_size_or_sections[self.smk_size ** 2, self.kernel_size ** 2], dim-1) rpb1_idx self.generate_idx(self.smk_size) rpb2_idx self.generate_idx(self.kernel_size) attn1 self.apply_rpb(attn1, self.rpb1, H, W, self.smk_size, *rpb1_idx) attn2 self.apply_rpb(attn2, self.rpb2, H, W, self.kernel_size, *rpb2_idx) attn1 torch.softmax(attn1, dim-1) attn2 torch.softmax(attn2, dim-1) value rearrange(x, b (m g c) h w - m b g h w c, m2, gself.num_heads) x1 na2d_av(attn1, value[0], kernel_sizeself.smk_size) x2 na2d_av(attn2, value[1], kernel_sizeself.kernel_size) x torch.cat([x1, x2], dim1) x rearrange(x, b g h w c - b (g c) h w, hH, wW) x self.dyconv_proj(x) return x2.2 更改yaml文件 以自研模型加入为例yam文件解读YOLO系列 “.yaml“文件解读_yolo yaml文件-CSDN博客打开更改ultralytics/cfg/models/rt-detr路径下的rtdetr-l.yaml文件替换原有模块。​​# Ultralytics YOLO , AGPL-3.0 license # RT-DETR-l object detection model with P3-P5 outputs. For details see https://docs.ultralytics.com/models/rtdetr # ⭐⭐Powered by https://blog.csdn.net/StopAndGoyyy, 技术指导QQ:2668825911⭐⭐ # Parameters nc: 80 # number of classes scales: # model compound scaling constants, i.e. modelyolov8n-cls.yaml will call yolov8-cls.yaml with scale n # [depth, width, max_channels] l: [1.00, 1.00, 512] # n: [ 0.33, 0.25, 1024 ] # s: [ 0.33, 0.50, 1024 ] # m: [ 0.67, 0.75, 768 ] # l: [ 1.00, 1.00, 512 ] # x: [ 1.00, 1.25, 512 ] # ⭐⭐Powered by https://blog.csdn.net/StopAndGoyyy, 技术指导QQ:2668825911⭐⭐ backbone: # [from, repeats, module, args] - [-1, 1, Conv, [64, 3, 2]] # 0-P1/2 - [-1, 1, Conv, [128, 3, 2]] # 1-P2/4 - [-1, 2, CCRI, [128, 5, True, False]] - [-1, 1, Conv, [256, 3, 2]] # 3-P3/8 - [-1, 4, CCRI, [256, 3, True, True]] - [-1, 1, Conv, [512, 3, 2]] # 5-P4/16 - [-1, 4, CCRI, [512, 3, True, True]] - [-1, 1, Conv, [1024, 3, 2]] # 7-P5/32 - [-1, 2, CCRI, [1024, 3, True, False]] head: - [-1, 1, Conv, [256, 1, 1, None, 1, 1, False]] # 9 input_proj.2 - [-1, 1, ContMix, []] - [-1, 1, Conv, [256, 1, 1]] # 11, Y5, lateral_convs.0 - [-1, 1, nn.Upsample, [None, 2, nearest]] - [6, 1, Conv, [256, 1, 1, None, 1, 1, False]] # 13 input_proj.1 - [[-2, -1], 1, Concat, [1]] - [-1, 2, RepC4, [256]] # 15, fpn_blocks.0 - [-1, 1, Conv, [256, 1, 1]] # 16, Y4, lateral_convs.1 - [-1, 1, nn.Upsample, [None, 2, nearest]] - [4, 1, Conv, [256, 1, 1, None, 1, 1, False]] # 18 input_proj.0 - [[-2, -1], 1, Concat, [1]] # cat backbone P4 - [-1, 2, RepC4, [256]] # X3 (20), fpn_blocks.1 - [-1, 1, Conv, [256, 3, 2]] # 22, downsample_convs.0 - [[-1, 16], 1, Concat, [1]] # cat Y4 - [-1, 2, RepC4, [256]] # F4 (23), pan_blocks.0 - [-1, 1, Conv, [256, 3, 2]] # 24, downsample_convs.1 - [[-1, 11], 1, Concat, [1]] # cat Y5 - [-1, 2, RepC4, [256]] # F5 (26), pan_blocks.1 - [[20, 23, 26], 1, RTDETRDecoder, [nc]] # Detect(P3, P4, P5) # ⭐⭐Powered by https://blog.csdn.net/StopAndGoyyy, 技术指导QQ:2668825911⭐⭐2.2 修改train.py文件创建Train_RT脚本用于训练。from ultralytics.models import RTDETR import os os.environ[KMP_DUPLICATE_LIB_OK] True if __name__ __main__: model RTDETR(modelultralytics/cfg/models/rt-detr/rtdetr-l.yaml) # model.load(yolov8n.pt) model.train(data./data.yaml, epochs2, batch1, device0, imgsz640, workers2, cacheFalse, ampTrue, mosaicFalse, projectruns/train, nameexp)​​在train.py脚本中填入修改好的yaml路径运行即可训。​​