模型训练 不同批次大小与学习率下的训练损失对比3以 ConvNeXt 为例 论文中的一些做法flyfish一、ImageNet-1K 从头全量训练300 epochs原文We train ConvNeXts for 300 epochs using AdamW [46] with a learning rate of 4e-3. There is a 20-epoch linear warmup and a cosine decaying schedule afterward. We use a batch size of 4096 and a weight decay of 0.05.参数基准学习率4e-3批次大小4096学习率调度20轮线性预热linear warmup 余弦衰减cosine decaying schedule二、ImageNet-22K 大规模预训练90 epochs原文We pre-train ConvNeXts on ImageNet-22K for 90 epochs with a warmup of 5 epochs. We do not use EMA. Other settings follow ImageNet-1K.参数基准学习率与ImageNet-1K训练一致为4e-3批次大小与ImageNet-1K训练一致为4096学习率调度5轮线性预热 余弦衰减其余超参全部继承1K训练配置三、ImageNet-1K 分类微调基于22K预训练权重30 epochs原文We fine-tune ImageNet22K pre-trained models on ImageNet-1K for 30 epochs. We use AdamW, a learning rate of 5e-5, cosine learning rate schedule, layer-wise learning rate decay [6,12], no warmup, a batch size of 512, and weight decay of 1e-8.In fine-tuning, we use layer-wise learning rate decay [6, 12] with every 3 consecutive blocks forming a group.When the model is fine-tuned at 384² resolution, we use a crop ratio of 1.0 (i.e., no cropping) during testing following [2,74,80], instead of 0.875 at 224².参数基准学习率5e-5所有分辨率统一批次大小512学习率调度无预热余弦衰减分层学习率衰减每3个连续block为一组224×224分辨率衰减系数 0.8ConvNeXt-B/ 0.95ConvNeXt-L384×384分辨率衰减系数 0.7四、下游任务COCO 目标检测/实例分割原文We conduct a lightweight sweep for COCO experiments including learning rate {1e-4, 2e-4}, layer-wise learning rate decay [6] {0.7, 0.8, 0.9, 0.95}, and stochastic depth rate {0.3, 0.4, 0.5, 0.6, 0.7, 0.8}.Following Swin Transformer [45], we use multi-scale training, AdamW optimizer, and a 3× schedule.参数学习率网格搜索范围{1e-4, 2e-4}分层学习率衰减搜索范围 {0.7, 0.8, 0.9, 0.95}优化器AdamW训练策略3× schedule 多尺度训练五、下游任务ADE20K 语义分割原文The hyperparameters we sweep for ADE20K experiments include learning rate {8e-5, 1e-4}, layer-wise learning rate decay {0.8, 0.9}, and stochastic depth rate {0.3, 0.4, 0.5}. All model variants are trained for 160K iterations with a batch size of 16.参数学习率网格搜索范围{8e-5, 1e-4}批次大小16分层学习率衰减搜索范围 {0.8, 0.9}训练迭代数160K iterations训练范式参考原文In our study, we use a training recipe that is close to DeiT’s [73] and Swin Transformer’s [45].ConvNeXt 的学习率、批次大小等超参设计整体参考了同期视觉Transformer的标准训练范式。 模型训练 不同批次大小与学习率下的训练损失对比（3）
模型训练 不同批次大小与学习率下的训练损失对比3以 ConvNeXt 为例 论文中的一些做法flyfish一、ImageNet-1K 从头全量训练300 epochs原文We train ConvNeXts for 300 epochs using AdamW [46] with a learning rate of 4e-3. There is a 20-epoch linear warmup and a cosine decaying schedule afterward. We use a batch size of 4096 and a weight decay of 0.05.参数基准学习率4e-3批次大小4096学习率调度20轮线性预热linear warmup 余弦衰减cosine decaying schedule二、ImageNet-22K 大规模预训练90 epochs原文We pre-train ConvNeXts on ImageNet-22K for 90 epochs with a warmup of 5 epochs. We do not use EMA. Other settings follow ImageNet-1K.参数基准学习率与ImageNet-1K训练一致为4e-3批次大小与ImageNet-1K训练一致为4096学习率调度5轮线性预热 余弦衰减其余超参全部继承1K训练配置三、ImageNet-1K 分类微调基于22K预训练权重30 epochs原文We fine-tune ImageNet22K pre-trained models on ImageNet-1K for 30 epochs. We use AdamW, a learning rate of 5e-5, cosine learning rate schedule, layer-wise learning rate decay [6,12], no warmup, a batch size of 512, and weight decay of 1e-8.In fine-tuning, we use layer-wise learning rate decay [6, 12] with every 3 consecutive blocks forming a group.When the model is fine-tuned at 384² resolution, we use a crop ratio of 1.0 (i.e., no cropping) during testing following [2,74,80], instead of 0.875 at 224².参数基准学习率5e-5所有分辨率统一批次大小512学习率调度无预热余弦衰减分层学习率衰减每3个连续block为一组224×224分辨率衰减系数 0.8ConvNeXt-B/ 0.95ConvNeXt-L384×384分辨率衰减系数 0.7四、下游任务COCO 目标检测/实例分割原文We conduct a lightweight sweep for COCO experiments including learning rate {1e-4, 2e-4}, layer-wise learning rate decay [6] {0.7, 0.8, 0.9, 0.95}, and stochastic depth rate {0.3, 0.4, 0.5, 0.6, 0.7, 0.8}.Following Swin Transformer [45], we use multi-scale training, AdamW optimizer, and a 3× schedule.参数学习率网格搜索范围{1e-4, 2e-4}分层学习率衰减搜索范围 {0.7, 0.8, 0.9, 0.95}优化器AdamW训练策略3× schedule 多尺度训练五、下游任务ADE20K 语义分割原文The hyperparameters we sweep for ADE20K experiments include learning rate {8e-5, 1e-4}, layer-wise learning rate decay {0.8, 0.9}, and stochastic depth rate {0.3, 0.4, 0.5}. All model variants are trained for 160K iterations with a batch size of 16.参数学习率网格搜索范围{8e-5, 1e-4}批次大小16分层学习率衰减搜索范围 {0.8, 0.9}训练迭代数160K iterations训练范式参考原文In our study, we use a training recipe that is close to DeiT’s [73] and Swin Transformer’s [45].ConvNeXt 的学习率、批次大小等超参设计整体参考了同期视觉Transformer的标准训练范式。