Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

Appearance settings

疑似梯度爆炸 #334

Copy link
Copy link

Description

@kyo-dai
Issue body actions

我是单卡训练,因为设备性能不足只能更改batchsize为16,但是训练结果显示,大约从20epoch开始损失一直这么大,没有减小的趋势,但我并没有改代码的其他内容,是什么原因,学习率过低了吗?
Epoch: [34/80] Total time: 1:01:24 (0.4984 s / it)
Averaged stats: lr: 0.000013 loss: 31.4573 (31.6999) loss_vfl: 0.6523 (0.6711) loss_bbox: 0.1718 (0.1814) loss_giou: 0.5420 (0.5580) loss_fgl: 0.9576 (0.9586) loss_vfl_aux_0: 0.6924 (0.7414) loss_bbox_aux_0: 0.1809 (0.1943) loss_giou_aux_0: 0.5732 (0.5833) loss_fgl_aux_0: 0.9859 (0.9925) loss_ddf_aux_0: 0.0465 (0.0524) loss_vfl_aux_1: 0.6758 (0.7233) loss_bbox_aux_1: 0.1727 (0.1843) loss_giou_aux_1: 0.5473 (0.5635) loss_fgl_aux_1: 0.9565 (0.9616) loss_ddf_aux_1: 0.0063 (0.0074) loss_vfl_aux_2: 0.6636 (0.7034) loss_bbox_aux_2: 0.1720 (0.1819) loss_giou_aux_2: 0.5437 (0.5587) loss_fgl_aux_2: 0.9569 (0.9588) loss_ddf_aux_2: 0.0011 (0.0012) loss_vfl_aux_3: 0.6641 (0.6820) loss_bbox_aux_3: 0.1719 (0.1815) loss_giou_aux_3: 0.5429 (0.5580) loss_fgl_aux_3: 0.9577 (0.9587) loss_ddf_aux_3: 0.0002 (0.0002) loss_vfl_aux_4: 0.6494 (0.6732) loss_bbox_aux_4: 0.1718 (0.1814) loss_giou_aux_4: 0.5420 (0.5580) loss_fgl_aux_4: 0.9576 (0.9586) loss_ddf_aux_4: 0.0001 (0.0001) loss_vfl_pre: 0.6992 (0.7439) loss_bbox_pre: 0.1814 (0.1945) loss_giou_pre: 0.5710 (0.5819) loss_vfl_enc_0: 0.7163 (0.7446) loss_bbox_enc_0: 0.2078 (0.2262) loss_giou_enc_0: 0.6412 (0.6537) loss_vfl_dn_0: 0.4922 (0.4909) loss_bbox_dn_0: 0.2619 (0.2536) loss_giou_dn_0: 0.5637 (0.5719) loss_fgl_dn_0: 1.0806 (1.0913) loss_ddf_dn_0: 0.2193 (0.2250) loss_vfl_dn_1: 0.4231 (0.4260) loss_bbox_dn_1: 0.1960 (0.1881) loss_giou_dn_1: 0.4617 (0.4596) loss_fgl_dn_1: 0.9825 (0.9864) loss_ddf_dn_1: 0.0265 (0.0283) loss_vfl_dn_2: 0.4050 (0.4107) loss_bbox_dn_2: 0.1827 (0.1780) loss_giou_dn_2: 0.4467 (0.4432) loss_fgl_dn_2: 0.9741 (0.9764) loss_ddf_dn_2: 0.0056 (0.0063) loss_vfl_dn_3: 0.3950 (0.4020) loss_bbox_dn_3: 0.1742 (0.1748) loss_giou_dn_3: 0.4411 (0.4389) loss_fgl_dn_3: 0.9761 (0.9772) loss_ddf_dn_3: 0.0007 (0.0009) loss_vfl_dn_4: 0.3960 (0.3984) loss_bbox_dn_4: 0.1685 (0.1740) loss_giou_dn_4: 0.4403 (0.4376) loss_fgl_dn_4: 0.9764 (0.9778) loss_ddf_dn_4: 0.0000 (0.0001) loss_vfl_dn_5: 0.3958 (0.3977) loss_bbox_dn_5: 0.1685 (0.1740) loss_giou_dn_5: 0.4404 (0.4376) loss_fgl_dn_5: 0.9765 (0.9778) loss_ddf_dn_5: 0.0000 (0.0000) loss_vfl_dn_pre: 0.4922 (0.4915) loss_bbox_dn_pre: 0.2697 (0.2594) loss_giou_dn_pre: 0.5670 (0.5709)
Test: [ 0/79] eta: 0:03:32 time: 2.6882 data: 1.8266 max mem: 17421
Test: [10/79] eta: 0:01:26 time: 1.2489 data: 0.2143 max mem: 17421
Test: [20/79] eta: 0:01:10 time: 1.1158 data: 0.0665 max mem: 17421
Test: [30/79] eta: 0:00:56 time: 1.1131 data: 0.0667 max mem: 17421
Test: [40/79] eta: 0:00:45 time: 1.1162 data: 0.0663 max mem: 17421
Test: [50/79] eta: 0:00:32 time: 1.0364 data: 0.0663 max mem: 17421
Test: [60/79] eta: 0:00:21 time: 1.0394 data: 0.0666 max mem: 17421
Test: [70/79] eta: 0:00:10 time: 1.1245 data: 0.0666 max mem: 17421
Test: [78/79] eta: 0:00:01 time: 1.0807 data: 0.0603 max mem: 17421
Test: Total time: 0:01:27 (1.1121 s / it)
Metrics: {'f1': 0.6701090231742727, 'precision': 0.6669847588406903, 'recall': 0.6732626943718178, 'iou': 0.4351902008130304, 'TPs': 24463, 'FPs': 12214, 'FNs': 11872}
Averaged stats:
Accumulating evaluation results...
COCOeval_opt.accumulate() finished...
DONE (t=3.74s).
IoU metric: bbox
Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.527
Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.704
Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.572
Average Precision (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.338
Average Precision (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.570
Average Precision (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.712
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 1 ] = 0.390
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets= 10 ] = 0.654
Average Recall (AR) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.721
Average Recall (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.539
Average Recall (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.768
Average Recall (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.892
Average Recall (AR) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.916
Average Recall (AR) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.785
best_stat: {'epoch': 34, 'coco_eval_bbox': 0.5272979428131327}

Reactions are currently unavailable

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions

    Morty Proxy This is a proxified and sanitized view of the page, visit original site.