[Improvement] Training custom classes of ava dataset #555

irvingzhang0512 · 2021-01-21T01:22:49Z

Target

Training some of the 80 ava classes to save training time and hopefully get better results for selected classes.

TODO

dataset/evaluation codes.
unittest
docs
sample config
model zoo, compare results.
Add input arg topk for BBoxHeadAVA, because num_classes may be smaller than 5.
~~check whether exclude_file_xxx will affect the results.~~

results

slowonly_kinetics_pretrained_r50_4*16

custom classes	mAP(train 80 classes)	mAP (train custom classes only)	selected classes comment
range(1, 15)	0.3460	0.3399	all `PERSON_MOVEMENT` classes
`[11, 12, 14, 15, 79, 80]`	0.7066	0.7011	AP(80 classes ckpt) > 0.6
`[1,4,8,9,13,17,28,49,74]`	0.4339	0.4397	AP(80 classes ckpt) in[0.3, 0.6)
`[3, 6, 10, 27, 29, 38, 41, 48, 51, 53, 54, 59, 61, 64, 70, 72]`	0.1948	0.3311	AP(80 classes ckpt) in[0.1, 0.3)
`[11,12,17,74,79,80]`	0.6520	0.6438	> 50000 samples
`[1,8,14,59]`	0.4307	0.5549	[5000, 50000) samples
`[3,4,6,9,10,15,27,28,29,38,41,48,49,54,61,64,65,66,67,70,77]`	0.2384	0.3269	[1000, 5000) samples
`[22,37,47,51,63,68,72,78]`	0.0753	0.3209	[500, 1000) samples
`[2,5,7,13,20,24,26,30,34,36,42,45,46,52,56,57,58,60,62,69,73,75,76]`	0.0348	0.1806	[100, 500) samples
`[16,18,19,21,23,25,31,32,33,35,39,40,43,44,50,53,55,71]`	0.0169	0.1984	<100 samples

insights

I think ava dataset suffers from series class imbalance. Training custom classes helps to get better results for classes with fewer samples.

codecov · 2021-01-21T01:38:45Z

Codecov Report

Merging #555 (3045485) into master (4fcbcb0) will increase coverage by 0.29%.
The diff coverage is 87.60%.

@@            Coverage Diff             @@
##           master     #555      +/-   ##
==========================================
+ Coverage   84.17%   84.47%   +0.29%     
==========================================
  Files         121      124       +3     
  Lines        8613     8797     +184     
  Branches     1427     1478      +51     
==========================================
+ Hits         7250     7431     +181     
- Misses       1014     1015       +1     
- Partials      349      351       +2

Flag	Coverage Δ
unittests	`84.46% <87.60%> (+0.29%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
mmaction/models/backbones/resnet3d.py	`90.61% <ø> (+0.21%)`	⬆️
mmaction/models/backbones/resnet3d_slowfast.py	`84.61% <ø> (+0.40%)`	⬆️
mmaction/models/backbones/resnet3d_slowonly.py	`78.57% <ø> (+3.57%)`	⬆️
...maction/models/roi_extractors/single_straight3d.py	`77.58% <14.28%> (-5.75%)`	⬇️
mmaction/core/evaluation/recall.py	`11.42% <25.00%> (-0.34%)`	⬇️
mmaction/models/backbones/resnet_tin.py	`19.56% <25.00%> (-0.29%)`	⬇️
...action/core/bbox/assigners/max_iou_assigner_ava.py	`51.02% <50.00%> (ø)`
mmaction/models/backbones/resnet.py	`93.15% <50.00%> (-1.74%)`	⬇️
mmaction/models/builder.py	`55.26% <50.00%> (ø)`
mmaction/models/heads/roi_head.py	`28.26% <50.00%> (ø)`
... and 18 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 4fcbcb0...b94c253. Read the comment docs.

configs/detection/ava/slowonly_kinetics_pretrained_r50_4x16x1_20e_ava_rgb_custom_classes.py

mmaction/datasets/ava_dataset.py

configs/detection/ava/README.md

kennymckormick · 2021-01-21T03:01:41Z

Some minor comments

irvingzhang0512 · 2021-01-22T03:05:57Z

Maybe this pr is not that useful as I expected... Training custom classes leads to lower mAP... Most classes with hight AP(>0.3) get worse results.

slowonly_kinetics_pretrained_r50_4*16

custom classes	mAP(trained 80 classes)	mAP (training custom classes only)	comment
range(1, 15)	0.3460	0.3399	training all `PERSON_MOVEMENT` classes

I'll do more tests

select classes with fewer samples only
finetune from 80 classes ckpt
another backbone
tune lr

@innerlee @kennymckormick any suggestions?

kennymckormick · 2021-01-22T07:50:33Z

Maybe this pr is not that useful as I expected... Training custom classes leads to lower mAP... Most classes with hight AP(>0.3) get worse results.

slowonly_kinetics_pretrained_r50_4*16

custom classes mAP(trained 80 classes) mAP (training custom classes only) comment
range(1, 15) 0.3460 0.3399 training all PERSON_MOVEMENT classes
I'll do more tests

select classes with fewer samples only

finetune from 80 classes ckpt

another backbone

tune lr

@innerlee @kennymckormick any suggestions?

Maybe you can try to pick the best model among epoch 11 -> 20, and see if its performance is better.

irvingzhang0512 · 2021-01-22T07:56:31Z

Maybe this pr is not that useful as I expected... Training custom classes leads to lower mAP... Most classes with hight AP(>0.3) get worse results.

slowonly_kinetics_pretrained_r50_4*16

custom classes mAP(trained 80 classes) mAP (training custom classes only) comment
range(1, 15) 0.3460 0.3399 training all PERSON_MOVEMENT classes
I'll do more tests

select classes with fewer samples only

finetune from 80 classes ckpt

another backbone

tune lr

@innerlee @kennymckormick any suggestions?

Maybe you can try to pick the best model among epoch 11 -> 20, and see if its performance is better.

Actually, I choose the best ckpt(epoch no.14), not the last one..

irvingzhang0512 · 2021-01-25T09:17:53Z

Finally get some good results...

slowonly_kinetics_pretrained_r50_4*16

custom classes	mAP(train 80 classes)	mAP (train custom classes only)	comment
range(1, 15)	0.3460	0.3399	training all `PERSON_MOVEMENT` classes
`[11, 12, 14, 15, 79, 80]`	0.7066	0.7011	classes with mAP(80 classes ckpt) > 0.6
`[1, 8, 9, 13, 17, 28, 49, 74]`	0.4243	0.4459	classes with mAP(80 classes ckpt) in[0.3, 0.6)
`[3, 6, 10, 27, 29, 38, 41, 48, 51, 53, 54, 59, 61, 64, 70, 72]`	0.1948	0.3311	classes with mAP(80 classes ckpt) in[0.1, 0.3)

I think ava suffers from data imbalance issue. For slowonly_kinetics_pretrained_r50_4*16 ava model

class	cnt of samples	AP
stand	164932	0.7833
sit	98770	0.7709
answer phone	3251	0.7082
talk to (e.g., self, a person, a group)	107742	0.6987
walk	40472	0.6434
watch (a person)	157920	0.6354
drive (e.g., a car, a truck)	1174	0.5321
carry/hold (an object)	78144	0.5127
listen to (a person)	103358	0.5107
lie/sleep	5298	0.4779
dance	3196	0.4612
martial art	2068	0.3788
swim	107	0.3687
bend/bow (at the waist)	8148	0.3376
ride (e.g., a bike, a car, a horse)	4663	0.3256
fight/hit (a person)	2602	0.2951
touch (an object)	15981	0.2639
watch (e.g., TV)	1917	0.2615
kiss (a person)	697	0.2456
run/jog	3296	0.2435
read	2048	0.2407
eat	2868	0.2276
drink	2194	0.1937
shovel	65	0.182
play musical instrument	1286	0.1776
crouch/kneel	2196	0.1603
hug (a person)	1043	0.1566
open (e.g., a window, a car door)	1228	0.1414
smoke	2855	0.1137
get up	1113	0.1129
sail boat	634	0.1005
close (e.g., a door, a box)	778	0.0995
hand clap	1181	0.0989
sing to (e.g., self, a person, a group)	1548	0.0802
fall down	290	0.0753
fishing	77	0.0572
grab (a person)	1866	0.0536
work on a computer	172	0.0529
write	749	0.0526
hand shake	604	0.0442
give/serve (an object) to (a person)	1008	0.0423
cut	174	0.0394
shoot	221	0.0364
text on/look at a cellphone	393	0.0349
dig	38	0.0328
enter	217	0.0314
put down	528	0.0278
take (an object) from (a person)	598	0.0243
row boat	62	0.0239
lift (a person)	376	0.0223
jump/leap	241	0.0189
dress/put on clothing	313	0.0174
lift/pick up	497	0.0168
push (another person)	350	0.0156
push (an object)	378	0.0139
climb (e.g., a mountain)	244	0.013
crawl	150	0.0125
listen (e.g., to music)	593	0.0082
pull (an object)	329	0.0078
throw	247	0.0077
hand wave	327	0.0068
take a photo	178	0.0023
turn (e.g., a screwdriver)	133	0.0023
press	87	0.0017
cook	61	0.0015
hit (an object)	167	0.0015
play with pets	121	0.0015
exit	53	0.0009
play with kids	122	0.0009
stir	54	0.0007
play board game	92	0.0006
point to (an object)	96	0.0006
clink glass	52	0.0004
kick (a person)	49	0.0004
chop	36	0.0003
brush teeth	18	0.0002
catch (an object)	80	0.0002
extract	7	0.0002
kick (an object)	34	0.0002
paint	63	0.0001

We could train custom classes with fewer samples to get better results.

kennymckormick · 2021-01-25T09:23:50Z

Good results! You can choose a successful example to add configs and checkpoints. Also remember to add ur results to README file.

irvingzhang0512 · 2021-01-25T12:01:12Z

ckpt/json/log are uploaded here
@kennymckormick Would you kindly help to upload these files to your server?

irvingzhang0512 · 2021-01-29T02:40:06Z

kindly ping @kennymckormick

kennymckormick · 2021-02-01T03:29:18Z

kindly ping @kennymckormick

Got it

irvingzhang0512 · 2021-02-01T03:35:12Z

Add ckpt trained by custom classes in model zoo along with other models may be a little bit misleading. How about create a table for custom classes only...

kennymckormick · 2021-02-01T03:35:27Z

ckpt/json/log are uploaded here
@kennymckormick Would you kindly help to upload these files to your server?

Sorry seems I have no permission to download the files. Would you please upload the ckpts with other tools? (like dropbox, googledrive, baiduyun or transfer.sh)

irvingzhang0512 · 2021-02-01T03:41:06Z

ckpt/json/log are uploaded here
@kennymckormick Would you kindly help to upload these files to your server?

Sorry seems I have no permission to download the files. Would you please upload the ckpts with other tools? (like dropbox, googledrive, baiduyun or transfer.sh)

I'll upload these to baiduyun tonight... Cannot use other tools while in the company.

kennymckormick · 2021-02-01T03:47:48Z

Add ckpt trained by custom classes in model zoo along with other models may be a little bit misleading. How about create a table for custom classes only...

I think we need a pair of checkpoint and config file for users to try this feature. Besides, I think we can create a separate table for custom classes.

irvingzhang0512 · 2021-02-01T14:22:09Z

@kennymckormick
链接：https://pan.baidu.com/s/1XLL3yOqFr1IWM2eX5_nNGQ
提取码：os3p

kennymckormick · 2021-02-02T05:53:15Z

@kennymckormick
链接：https://pan.baidu.com/s/1XLL3yOqFr1IWM2eX5_nNGQ
提取码：os3p

gotcha

kennymckormick · 2021-02-02T13:52:47Z

@innerlee ready now

kennymckormick · 2021-02-02T13:54:46Z

Would u plz help solve the lint issue @irvingzhang0512 , I did't install markdown lint on the machine I commit

kennymckormick · 2021-02-03T07:36:42Z

@innerlee

configs/detection/ava/README.md

innerlee · 2021-02-03T08:11:07Z

Sorry I need to get a understanding on the motivation. Does custom==subset here? What if you simply filter the annotation file? Or use some re-sampling strategy to solve the class imbalance problem?

kennymckormick · 2021-02-03T08:14:18Z

Sorry I need to get a understanding on the motivation. Does custom==subset here? What if you simply filter the annotation file? Or use some re-sampling strategy to solve the class imbalance problem?

Here custom==subset. The motivation is: 1. Sometimes you only need to recognize a subset of action classes, and training on the subset led to better performance. 2. Adding a line in the config file is better than regenerating annotation files when doing filtering.

innerlee · 2021-02-03T08:20:08Z

So an alternative implementation would be, writing a scirpt to fitler a subset of the annotation file, right?

An even better alternative is, implementing a sampling strategy on a class-basis, so that you can adjust the sampling ratio of any class to any propotions, including exclusion by setting 0 ratio?

kennymckormick · 2021-02-03T08:30:00Z

So an alternative implementation would be, writing a scirpt to fitler a subset of the annotation file, right?

An even better alternative is, implementing a sampling strategy on a class-basis, so that you can adjust the sampling ratio of any class to any propotions, including exclusion by setting 0 ratio?

The first alternative is OK, but sometimes you don't want to have additional files generated.
For the second alternative, actually it is already implemented in MMAction2 (to apply it to AVA, a little modification to ava_dataset.py might be needed). Maybe we can discuss about which is the better design.

irvingzhang0512 · 2021-02-03T08:41:06Z

So an alternative implementation would be, writing a scirpt to fitler a subset of the annotation file, right?

An even better alternative is, implementing a sampling strategy on a class-basis, so that you can adjust the sampling ratio of any class to any propotions, including exclusion by setting 0 ratio?

There is a little difference between the first alternative and this pr. This pr modifies label id and num_classes. I was planning to train ava (or my own dataset) with softmax loss (single label instead of multi label) someday.

innerlee · 2021-02-05T01:02:45Z

I will post a discussion later today :)

irvingzhang0512 · 2021-02-20T03:24:09Z

kindly ping @innerlee

innerlee · 2021-02-20T04:29:45Z

I'm still trying to understand the motivation, so please be patient.

Q1: In the comparision tables, there are mAP(train 80 classes) and mAP (train custom classes only). Are they evaluated using the same amount of samples?
Q2: Does each row in the table correspond to a new model that is trained using the custom class in the row?
Q3: Maybe this has been asked before, what if you simply remove the records that do not belong to custom classes in the original ava_action_list_v2.1.pbtxt? Or rephrase as this: can this be implemented by simply creating a new dataset list?

irvingzhang0512 · 2021-02-20T06:24:18Z

Q1: In the comparision tables, there are mAP(train 80 classes) and mAP (train custom classes only). Are they evaluated using the same amount of samples?

Yes

Q2: Does each row in the table correspond to a new model that is trained using the custom class in the row?

Yes

Q3: Maybe this has been asked before, what if you simply remove the records that do not belong to custom classes in the original ava_action_list_v2.1.pbtxt? Or rephrase as this: can this be implemented by simply creating a new dataset list?

We should remove records that do not belong to custom classes AND change the label id of the samples that belong to custom classes. Sorry about my poor english, I may explain in chinese here.

为什么要对custom classes重新编号而不是使用原始编号？

如果使用ava的默认配置（multi-label任务，训练每个class使用sigmoid），那改变不改变标签编号对mAP并没有影响（因为使用了sigmoid），但inference time和model size能省一点算一点。
考虑到sit/stand/walk/lie这些是互斥的，后续可能会支持单标签训练（softmax），可能会得到更好的效果。使用softmax的时候，我们猜测num_classes会影响结果，假设custom_classes只有10类，AVA本身是80类，那最终的预测结果（softmax后）的shape应该是[batch size, 10]而不是[batch size, 80]，所以需要修改label id。当然这个需要以后试验验证。

另外也多说几句为什么要这个PR：

基本目标就是 训练输入数据集的一个子集，而不是整个输入数据集。
同时明确，我们的确可以通过写脚本修改输入配置文件来达到一样的功能。
训练数据集子集有两方面的考虑：一方面，节约训练时间（对于资源有限的用户，训练整个ava实在是太困难了）与提高模型精度；另一方面，是考虑联合训练AVA与自己的数据集，为了进行探索，我们可能需要尝试多种classes组合，写脚本生成新的training list恐怕管理起来更麻烦。

innerlee · 2021-02-21T03:39:05Z

Thanks for the explanation.

Considering that this change is fairly local (only affects ava dataset), and provides some convinent tool for real use cases, plus @kennymckormick , who will maintain this piece of code, has approved, I will merge it.

There are some alternative ways, and may leave to future,

Write easy-to-use script tool to transform (taking subsets, remapping labels, etc.) dataset annotations.
Implement a general purpose class-based dataset weighted-sampling functionality.

irvingzhang0512 added 3 commits January 21, 2021 09:04

add custom_classes in dataset and eval

e2749a0

add docs and sample config

e883f13

add unittest

43bcece

irvingzhang0512 mentioned this pull request Jan 21, 2021

Iteration Plan - Jan 2021 #531

Closed

6 tasks

typo

21cef09

update changelog

074c624

innerlee requested a review from kennymckormick January 21, 2021 02:29

kennymckormick reviewed Jan 21, 2021

View reviewed changes

configs/detection/ava/slowonly_kinetics_pretrained_r50_4x16x1_20e_ava_rgb_custom_classes.py Outdated Show resolved Hide resolved

mmaction/datasets/ava_dataset.py Show resolved Hide resolved

configs/detection/ava/README.md Show resolved Hide resolved

update config and add assert for custom_classes

088784d

fix unittest

593870d

irvingzhang0512 added 2 commits January 25, 2021 19:06

Merge branch 'master' into tp

de46d57

update ava readme and config

dfdf4b7

irvingzhang0512 added 4 commits February 1, 2021 09:37

Merge branch 'master' into ava-custom-classes

51696fe

add input arg topk and related tests

78b6465

update docs

1544177

update docs & fix tests

fa66570

update docs

4137ca6

add link

82a01ed

fix lint

6184fd5

kennymckormick approved these changes Feb 3, 2021

View reviewed changes

innerlee reviewed Feb 3, 2021

View reviewed changes

configs/detection/ava/README.md Outdated Show resolved Hide resolved

update docs

ff228e1

replace key_indicator with save_best

b94c253

irvingzhang0512 mentioned this pull request Feb 6, 2021

Take topk from config #603

Closed

innerlee merged commit d3d646a into open-mmlab:master Feb 21, 2021

irvingzhang0512 deleted the ava-custom-classes branch February 22, 2021 07:14

dreamerlin mentioned this pull request Feb 25, 2021

Iteration Plan - Feb 2021 #647

Closed

9 tasks

irvingzhang0512 mentioned this pull request Mar 3, 2021

RuntimeError while training slowfast for AVA custom classes #670

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Improvement] Training custom classes of ava dataset #555

[Improvement] Training custom classes of ava dataset #555

irvingzhang0512 commented Jan 21, 2021 •

edited by kennymckormick

Loading

codecov bot commented Jan 21, 2021 •

edited

Loading

kennymckormick commented Jan 21, 2021

irvingzhang0512 commented Jan 22, 2021 •

edited

Loading

kennymckormick commented Jan 22, 2021

irvingzhang0512 commented Jan 22, 2021

irvingzhang0512 commented Jan 25, 2021

kennymckormick commented Jan 25, 2021

irvingzhang0512 commented Jan 25, 2021

irvingzhang0512 commented Jan 29, 2021

kennymckormick commented Feb 1, 2021

irvingzhang0512 commented Feb 1, 2021

kennymckormick commented Feb 1, 2021

irvingzhang0512 commented Feb 1, 2021

kennymckormick commented Feb 1, 2021

irvingzhang0512 commented Feb 1, 2021

kennymckormick commented Feb 2, 2021

kennymckormick commented Feb 2, 2021

kennymckormick commented Feb 2, 2021

kennymckormick commented Feb 3, 2021

innerlee commented Feb 3, 2021

kennymckormick commented Feb 3, 2021

innerlee commented Feb 3, 2021

kennymckormick commented Feb 3, 2021

irvingzhang0512 commented Feb 3, 2021

innerlee commented Feb 5, 2021

irvingzhang0512 commented Feb 20, 2021

innerlee commented Feb 20, 2021 •

edited

Loading

irvingzhang0512 commented Feb 20, 2021 •

edited

Loading

innerlee commented Feb 21, 2021

[Improvement] Training custom classes of ava dataset #555

[Improvement] Training custom classes of ava dataset #555

Conversation

irvingzhang0512 commented Jan 21, 2021 • edited by kennymckormick Loading

Target

TODO

results

insights

codecov bot commented Jan 21, 2021 • edited Loading

Codecov Report

kennymckormick commented Jan 21, 2021

irvingzhang0512 commented Jan 22, 2021 • edited Loading

kennymckormick commented Jan 22, 2021

irvingzhang0512 commented Jan 22, 2021

irvingzhang0512 commented Jan 25, 2021

kennymckormick commented Jan 25, 2021

irvingzhang0512 commented Jan 25, 2021

irvingzhang0512 commented Jan 29, 2021

kennymckormick commented Feb 1, 2021

irvingzhang0512 commented Feb 1, 2021

kennymckormick commented Feb 1, 2021

irvingzhang0512 commented Feb 1, 2021

kennymckormick commented Feb 1, 2021

irvingzhang0512 commented Feb 1, 2021

kennymckormick commented Feb 2, 2021

kennymckormick commented Feb 2, 2021

kennymckormick commented Feb 2, 2021

kennymckormick commented Feb 3, 2021

innerlee commented Feb 3, 2021

kennymckormick commented Feb 3, 2021

innerlee commented Feb 3, 2021

kennymckormick commented Feb 3, 2021

irvingzhang0512 commented Feb 3, 2021

innerlee commented Feb 5, 2021

irvingzhang0512 commented Feb 20, 2021

innerlee commented Feb 20, 2021 • edited Loading

irvingzhang0512 commented Feb 20, 2021 • edited Loading

innerlee commented Feb 21, 2021

irvingzhang0512 commented Jan 21, 2021 •

edited by kennymckormick

Loading

codecov bot commented Jan 21, 2021 •

edited

Loading

irvingzhang0512 commented Jan 22, 2021 •

edited

Loading

innerlee commented Feb 20, 2021 •

edited

Loading

irvingzhang0512 commented Feb 20, 2021 •

edited

Loading