[Feature] Support TimeSformer #839

congee524 · 2021-04-28T13:14:42Z

kennymckormick

The current min version of mmcv is 1.3.1, is it sufficient for this PR?

configs/recognition/timesformer/timesformer_divST_8x32x1_15e_kinetics400_rgb.py

mmaction/models/backbones/timesformer.py

kennymckormick · 2021-05-26T04:30:46Z

The current min version of mmcv is 1.3.1, is it sufficient for this PR?

Oh, seems not. You need to update the version requirement as well as the CI. Besides, you can add some unittest to improve the CodeCov

congee524 · 2021-05-26T04:48:20Z

The current min version of mmcv is 1.3.1, is it sufficient for this PR?

Oh, seems not. You need to update the version requirement as well as the CI. Besides, you can add some unittest to improve the CodeCov

OK. The work may need to wait until mmcv 1.3.5 release.

kennymckormick · 2021-05-26T07:03:44Z

The current min version of mmcv is 1.3.1, is it sufficient for this PR?

Oh, seems not. You need to update the version requirement as well as the CI. Besides, you can add some unittest to improve the CodeCov

OK. The work may need to wait until mmcv 1.3.5 release.

You can ping me after this PR is ready.

… timesformer

kennymckormick · 2021-06-16T13:49:20Z

mmaction/models/common/transformer.py

+        self.num_heads = num_heads
+        self.num_frames = num_frames
+        self.norm = build_norm_layer(norm_cfg, self.embed_dims)[1]
+        self.attn = nn.MultiheadAttention(embed_dims, num_heads, attn_drop,


Perhaps you need to use MultiheadAttention from mmcv since it takes the arg batch_first

Yes, it should be turned into mmcv MultiheadAttention, pytorch 1.9 includes arg but previous versions not.

kennymckormick · 2021-06-16T13:49:49Z

mmaction/models/common/transformer.py

+        self.num_heads = num_heads
+        self.num_frames = num_frames
+        self.norm = build_norm_layer(norm_cfg, self.embed_dims)[1]
+        self.attn = nn.MultiheadAttention(embed_dims, num_heads, attn_drop,


CLAassistant · 2021-06-17T06:53:33Z

All committers have signed the CLA.

shvdiwnkozbw · 2021-06-24T07:53:12Z

It seems MMCV minimum version 1.3.6 is required.

kennymckormick · 2021-06-24T08:37:38Z

@shvdiwnkozbw The old checkpoints also need to be updated. You can send me the converted checkpoint.

codecov · 2021-06-24T09:37:12Z

Codecov Report

Merging #839 (2e445ba) into master (79fba89) will decrease coverage by 0.38%.
The diff coverage is 88.17%.

@@            Coverage Diff             @@
##           master     #839      +/-   ##
==========================================
- Coverage   83.73%   83.34%   -0.39%     
==========================================
  Files         132      135       +3     
  Lines        9982    10311     +329     
  Branches     1718     1764      +46     
==========================================
+ Hits         8358     8594     +236     
- Misses       1210     1285      +75     
- Partials      414      432      +18

Flag	Coverage Δ
unittests	`83.34% <88.17%> (-0.39%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
mmaction/models/backbones/timesformer.py	`78.00% <78.00%> (ø)`
mmaction/models/common/transformer.py	`97.56% <97.56%> (ø)`
mmaction/__init__.py	`100.00% <100.00%> (ø)`
mmaction/models/__init__.py	`100.00% <100.00%> (ø)`
mmaction/models/backbones/__init__.py	`100.00% <100.00%> (ø)`
mmaction/models/common/__init__.py	`100.00% <100.00%> (ø)`
mmaction/models/heads/__init__.py	`100.00% <100.00%> (ø)`
mmaction/models/heads/timesformer_head.py	`100.00% <100.00%> (ø)`
mmaction/models/localizers/base.py	`45.88% <0.00%> (-10.99%)`	⬇️
mmaction/datasets/pipelines/compose.py	`90.32% <0.00%> (-9.68%)`	⬇️
... and 12 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 79fba89...2e445ba. Read the comment docs.

shvdiwnkozbw · 2021-06-30T08:48:44Z

@shvdiwnkozbw The old checkpoints also need to be updated. You can send me the converted checkpoint.

This is the link for converted checkpoint
https://drive.google.com/drive/folders/1E3uwhfDZlArzI8wsvGGD26EoNbrxsUp7?usp=sharing

kennymckormick · 2021-06-30T11:25:24Z

@shvdiwnkozbw The old checkpoints also need to be updated. You can send me the converted checkpoint.

This is the link for converted checkpoint
https://drive.google.com/drive/folders/1E3uwhfDZlArzI8wsvGGD26EoNbrxsUp7?usp=sharing

Have you tested the converted checkpoints? It seems that the testing accuracy doesn't align with the reported numbers, e.g., I got 61.8% Top-1 when I test the divST model (jointST failed).

shvdiwnkozbw · 2021-06-30T12:09:56Z

Have you tested the converted checkpoints? It seems that the testing accuracy doesn't align with the reported numbers, e.g., I got 61.8% Top-1 when I test the divST model (jointST failed).

I have not tested the accuracy since my account has been cancelled, are there any missing parameters when loading the converted checkpoint?

kennymckormick · 2021-06-30T13:17:17Z

Have you tested the converted checkpoints? It seems that the testing accuracy doesn't align with the reported numbers, e.g., I got 61.8% Top-1 when I test the divST model (jointST failed).

I have not tested the accuracy since my account has been cancelled, are there any missing parameters when loading the converted checkpoint?

For divST, the parameter names aligned perfectly but the accuracy is about 10% inferior. For jointST, the checkpoint can not be loaded properly.

congee524 changed the title ~~[Feature] Support TimeSformer~~ [Feature] Support TimeSformer (BC) Apr 28, 2021

congee524 force-pushed the timesformer branch from fbb4bca to 6e5364c Compare May 8, 2021 07:23

congee524 added BC-breaking WIP work in progress labels May 13, 2021

congee524 changed the title ~~[Feature] Support TimeSformer (BC)~~ [Feature] Support TimeSformer May 13, 2021

congee524 added BLOCKED BY MMCV and removed BC-breaking labels May 21, 2021

congee524 force-pushed the timesformer branch from b3d153a to bb3c600 Compare May 21, 2021 08:59

congee524 marked this pull request as ready for review May 21, 2021 09:00

congee524 removed the WIP work in progress label May 24, 2021

congee524 requested review from kennymckormick, innerlee and SuX97 May 24, 2021 08:05

congee524 force-pushed the timesformer branch from 97dc3dc to a71b3b4 Compare May 24, 2021 08:18

kennymckormick reviewed May 26, 2021

View reviewed changes

congee524 added 12 commits May 27, 2021 18:02

add file

6b363d1

add attention

0065805

timesformer code

250f1a7

timesformer head code

e44d1c3

refactor timesformer according to new version of mmcv transformer

a07c460

add timesformer config

0a1834a

add trunc normalize

2f9ce2a

add stochastic depth decay rule

4ebb746

add spaceonly & joint space time configs

2b2bac6

refine timesformer backbone code

69be25c

reduce batch_size off jointST

06c5de1

add comments for timesformerhead

0612849

congee524 added 4 commits May 27, 2021 18:02

add README

5e6977f

use trunc_norm in mmcv

0633ddd

add comments

89d570c

add unittest

b2e4a7e

congee524 force-pushed the timesformer branch from ea0eb4c to b2e4a7e Compare May 27, 2021 13:31

update ckpt url

5abcab8

kennymckormick force-pushed the timesformer branch from ae1c5d1 to 5abcab8 Compare June 16, 2021 13:15

Merge branch 'master' of https://github.com/open-mmlab/mmaction2 into…

3094844

… timesformer

kennymckormick reviewed Jun 16, 2021

View reviewed changes

fix load state_dict

8e2920c

shvdiwnkozbw and others added 5 commits June 17, 2021 14:54

fix multiheadattention

aaa71c3

fix mmcv version

50f5229

Merge branch 'master' into timesformer

cb5ea0c

fix batch_first

ddf42d2

fix mmcv minimum version

f73ead2

fix mmcv installation in CI

4983cfb

kennymckormick added 2 commits July 1, 2021 16:15

revert changes

043c81d

check_version

2e445ba

kennymckormick removed the BLOCKED BY MMCV label Jul 1, 2021

kennymckormick approved these changes Jul 1, 2021

View reviewed changes

kennymckormick merged commit 1c2d9c6 into open-mmlab:master Jul 1, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Feature] Support TimeSformer #839

[Feature] Support TimeSformer #839

congee524 commented Apr 28, 2021 •

edited

Loading

kennymckormick left a comment

kennymckormick commented May 26, 2021

congee524 commented May 26, 2021

kennymckormick commented May 26, 2021

kennymckormick Jun 16, 2021

shvdiwnkozbw Jun 16, 2021

kennymckormick Jun 16, 2021

CLAassistant commented Jun 17, 2021 •

edited

Loading

shvdiwnkozbw commented Jun 24, 2021

kennymckormick commented Jun 24, 2021

codecov bot commented Jun 24, 2021 •

edited

Loading

shvdiwnkozbw commented Jun 30, 2021

kennymckormick commented Jun 30, 2021 •

edited

Loading

shvdiwnkozbw commented Jun 30, 2021

kennymckormick commented Jun 30, 2021

[Feature] Support TimeSformer #839

[Feature] Support TimeSformer #839

Conversation

congee524 commented Apr 28, 2021 • edited Loading

kennymckormick left a comment

Choose a reason for hiding this comment

kennymckormick commented May 26, 2021

congee524 commented May 26, 2021

kennymckormick commented May 26, 2021

kennymckormick Jun 16, 2021

Choose a reason for hiding this comment

shvdiwnkozbw Jun 16, 2021

Choose a reason for hiding this comment

kennymckormick Jun 16, 2021

Choose a reason for hiding this comment

CLAassistant commented Jun 17, 2021 • edited Loading

shvdiwnkozbw commented Jun 24, 2021

kennymckormick commented Jun 24, 2021

codecov bot commented Jun 24, 2021 • edited Loading

Codecov Report

shvdiwnkozbw commented Jun 30, 2021

kennymckormick commented Jun 30, 2021 • edited Loading

shvdiwnkozbw commented Jun 30, 2021

kennymckormick commented Jun 30, 2021

congee524 commented Apr 28, 2021 •

edited

Loading

CLAassistant commented Jun 17, 2021 •

edited

Loading

codecov bot commented Jun 24, 2021 •

edited

Loading

kennymckormick commented Jun 30, 2021 •

edited

Loading