ybacm的博客

该文章就是在 ViT 的基础上做了一些改进 1）mask 了大部分的 patches，以减少图像的冗余信息 2）加入了基于 Transformer 架构的 Decoder 来还原像素 3）加入了 ViT 后续工作的一些训练技巧，来提升模型的鲁棒性。

An Image is Worth 16x16 Words: Transformers for Image Recognition at ScaleAuthor Unit: Google Brain, Google ResearchAuthors: Alexey Dosovitskiy∗,†, Lucas Beyer∗, Alexander Kolesnikov∗, Dirk Weissenborn∗, Xiaohua Zhai∗, Thomas Unterthiner, Mostafa Dehghan

Attention is all you needAuthor Unit: Google Brain, Google Research, University of TorontoAuthors: Ashish Vaswani∗^*∗, Noam Shazeer*, Niki Parmar*, Jakob Uszkoreit*, Llion Jones*, Aidan N. Gomez*†^††, Łukasz Kaiser∗^∗∗, Illia Polosukhin∗‡^{∗ ‡}∗‡Code: h

BERT: Pre-training of Deep Bidirectional Transformers for Language UnderstandingAuthor Unit: Google AI LanguageAuthors: Jacob Devlin, Ming-Wei Chang, Kenton Lee, Kristina ToutanovaCode: https://github.com/ google-research/bertConference: arXiv:1810.048