take over video swin checkpoints

[innat]

Keras-team

Could you please take over the video swin checkponts and upload it to kaggle in order to make it usable in kaggle platform?

1. I have posted regarding weights. Add Video Swin Transformer #2369 (comment)
2. Weight verification. Add Video Swin Transformer #2369 (comment)
3. Also could you please answer this query? Add Video Swin Transformer #2369 (comment)

[divyashreepathihalli]

Hi Innat!! will do
Thanks!!

[divyashreepathihalli]

@innat the verification notebooks are not accessible. can you please double check the permissions?

• what is the difference between the base and the backbone weights?
• do you have weights for the video classifier task model that you added?
• Also can you please use the keras_model.save_to_preset(checkpoint_name) to save to preset, this will generate a config, metadata and weights file that we can then upload to Kaggle
• Can you also please add your checkpoint conversion file here - keras_cv/src/tools/checkpoint_conversion. We have added some in .ipynb format but this is a good example to follow - https://github.com/keras-team/keras-nlp/blob/master/tools/checkpoint_conversion/convert_pali_gemma_checkpoints.py

[innat]

@divyashreepathihalli

what is the difference between the base and the backbone weights?
do you have weights for the video classifier task model that you added?

Ignore else, you can only look at here.

- videoswin_tiny_kinetics400.weights.h5 <- for backbone
- videoswin_tiny_kinetics400_classifier.weights.h5 <- for classifier

- videoswin_small_kinetics400.weights.h5 <- for backbone
- videoswin_small_kinetics400_classifier.weights.h5 <- for classifier

- videoswin_base_kinetics400.weights.h5 <- for backbone
- videoswin_base_kinetics400_classifier.weights.h5 <- for classifier

Some other variations of this model. (also check this comment)

- videoswin_base_kinetics400_imagenet22k.weights.h5 <- for backbone
- videoswin_base_kinetics400_imagenet22k_classifier.weights.h5 <- for classifier

- videoswin_base_kinetics600_imagenet22k.weights.h5 <- for backbone
- videoswin_base_kinetics600_imagenet22k_classifier.weights.h5 <- for classifier

- videoswin_base_something_something_v2.weights.h5 <- for backbone
- videoswin_base_something_something_v2_classifier.weights.h5 <- for classifier

FYI, the tiny, small, base are refer to the variants of the model. The kinetics400 and kinetics600 refer the kinetics dataset with 400 and 600 classes. The _imagenet22k term refers the pre-trained weight which were incorporated from 2D swin image model to initialize the video swin model. You can also check the official repo for more clarification.

Also can you please use the keras_model.save_to_preset(checkpoint_name) to save to preset, this will generate a config, metadata and weights file that we can then upload to Kaggle

I see, new API. Let me check.

Can you also please add your checkpoint conversion file here - keras_cv/src/tools/checkpoint_conversion.

I have to clean lots of messy code. How about the following two files

• for bacbone - keras-cv VS torch-vision
• for classifier - keras-cv VS torch-vision
• and more.

[divyashreepathihalli]

I will wait for your generated preset files. Also I cannot still access the verification files.

[innat]

@divyashreepathihalli Do you have a kaggle id? If so, could you please share?

[divyashreepathihalli]

@innat what do you mean by Kaggle ID?

• are you talking about the Kaggle handle that needs to be added to the presets file?
• or an authentication key that you need from Kaggle account?
• or login id for a kaggle account? - If this is for uploading the preset, the team will have to be doing that for now.

[innat]

@divyashreepathihalli
Uh, sorry for the confusion. Actually, I prepared some kaggle notebooks (currently its private) which will help you to evaluate or verify the model and weights. You can also run them out of the box on kaggle env (plug-n-play). If you share your kaggle id, I can add you as collaborator.

Here are the notebooks on kaggle. (currently in private.). I don't like to make these notebook public (as I already did something before with my own code). The following notebooks load the keras-cv.video-swin from its latest release. After the take over process (checking, verifying, saving of presets or weights) is done (by you), I will remove these notebooks from my end. Hope its clear now.

1. k400-logit-matching-torch-vs-keras-cv (classifier)
2. k400-logit-matching-torch-vs-keras-cv-backbone
3. k600-ssv2-logit-matching-torch-vs-keras-cv (classifier)
4. k600-ssv2-logit-matching-torch-vs-keras-cv-backbon
5. generate presets for keras-cv with save_to_preset
6. All the above notebooks, wget the model weights from here.

(Note. In number 1 and 2, we load the official video swin model from torchvision (with their API), and in number 3 and 4, we load the official vidoe swin model in raw pytorch code. )

[divyashreepathihalli]

Hi Innat!! here is my kaggle id - divyasss.