Transformer顶会最新进展 | 含IJCAI,CVPR,AAAI2023等会议论文

Transformer 模型是自然语言处理领域中的一种最先进的模型，它使用注意力机制来处理任意长度的输入文本，并且可以学习语言模式和规律。近年来，随着计算能力的不断提高，Transformer 模型已经被广泛应用于机器翻译、文本生成、文本分类等任务中。同时，Transformer 模型也被应用于代码生成任务中，通过训练模型来生成高质量的代码。

Transformer作为提升NLP效率的黑魔法，在自然语言处理领域发生了巨大的影响，如当下火热的GPT-3 和 lang-8 等，就是基于Transformer架构构建的大语言模型。近年来，研究者们在注意力机制方面进行了广泛的研究。其中，一些研究者提出了基于注意力机制的新方法，如自适应的注意力机制和全局注意力机制。这些新方法为 Transformer 模型的改进提供了新的思路。同时，为了更好地利用计算资源，研究者们提出了许多模型压缩方法，如剪枝、量化和蒸馏等。这些方法可以帮助研究者们更好地利用已有的 Transformer 模型，并提高模型的性能和效率。

Transformer 模型的最新进展表明，它已经成为自然语言处理领域中最重要的模型之一，并且在许多任务中取得了最先进的性能。

关于Transformer模型的顶会论文列表如下（由于篇幅关系，本篇只展现部分顶会论文，可复制文末链接，直达顶会会议列表，查看所有论文）

1.Singularformer: Learning to Decompose Self-Attention to Linearize the Complexity of Transformer

2.Towards Long-delayed Sparsity: Learning a Better Transformer through Reward Redistribution

3.HDFormer: High-order Directed Transformer for 3D Human Pose Estimation

4.CiT-Net: Convolutional Neural Networks Hand in Hand with Vision Transformers for Medical Image Segmentation

5.Learning Attention from Attention: Efficient Self-refinement Transformer for Face Super-resolution

6.FedET: A Communication-Efficient Federated Class-Incremental Learning Framework Based on Enhanced Transformer

7.Towards Incremental NER Data Augmentation via Syntactic-aware Insertion Transformer

8.Neighborhood Attention Transformer

9.EfficientViT: Memory Efficient Vision Transformer with Cascaded Group Attention

10.RGB no more: Minimally-decoded JPEG Vision Transformers

11.BiFormer: Vision Transformer with Bi-Level Routing Attention

12.Mask3D: Pre-training 2D Vision Transformers by Learning Masked 3D Priors

13.DSVT: Dynamic Sparse Voxel Transformer with Rotated Sets

14.OneFormer: One Transformer to Rule Universal Image Segmentation

15.Graph Transformer GANs for Graph-Constrained House Generation

16.DeepVecFont-v2: Exploiting Transformers to Synthesize Vector Fonts with Higher Quality

17.Vision Transformers are Parameter-Efficient Audio-Visual Learners

18.In-context Reinforcement Learning with Algorithm Distillation

19.Language Modelling with Pixels

20.A Time Series is Worth 64 Words: Long-term Forecasting with Transformers

21.Relational Attention: Generalizing Transformers for Graph-Structured Tasks

22.Encoding Recurrence into Transformers

23.Specformer: Spectral Graph Neural Networks Meet Transformers

24.MaskViT: Masked Visual Pre-Training for Video Prediction

25.Efficient Attention via Control Variates

26.What Do Self-Supervised Vision Transformers Learn?

27.Are More Layers Beneficial to Graph Transformers?

28.User Retention-oriented Recommendation with Decision Transformer

29.Tracing Knowledge Instead of Patterns: Stable Knowledge Tracing with Diagnostic Transformer

30.MetaTroll: Few-shot Detection of State-Sponsored Trolls with Transformer Adapters

31.Compact Transformer Tracker with Correlative Masked Modeling

32.Learning Progressive Modality-shared Transformers for Effective Visible-Infrared Person Re-identification

33.An Empirical Study of End-to-End Video-Language Transformers with Masked Visual Modeling

34.Vision Transformers Are Good Mask Auto-Labelers

35.Castling-ViT: Compressing Self-Attention via Switching Towards Linear-Angular Attention During Vision Transformer Inference

36.Transferable Adversarial Attacks on Vision Transformers with Token Gradient Regularization

37.Lite-Mono: A Lightweight CNN and Transformer Architecture for Self-Supervised Monocular Depth Estimation

38.Burstormer: Burst Image Restoration and Enhancement Transformer

39.Sparsifiner: Learning Sparse Instance-Dependent Attention for Efficient Vision Transformers

40.Recurrent Vision Transformers for Object Detection with Event Cameras

41.Q-DETR: An Efficient Low-Bit Quantized Detection Transformer

42.Feature Shrinkage Pyramid for Camouflaged Object Detection with Transformers

43.A Light Touch Approach to Teaching Transformers Multi-view Geometry

44.DeepSolo: Let Transformer Decoder with Explicit Points Solo for Text Spotting

45.Slide-Transformer: Hierarchical Vision Transformer with Local Self-Attention

46.CompletionFormer: Depth Completion with Convolutions and Vision Transformers

47.Devil is in the Queries: Advancing Mask Transformers for Real-world Medical Image Segmentation and Out-of-Distribution Localization

48.Vision Transformer with Super Token Sampling

49.POTTER: Pooling Attention Transformer for Efficient Human Mesh Recovery

50.Supervised Masked Knowledge Distillation for Few-Shot Transformers

点击链接，直达顶会页面：https://www.aminer.cn/conf

————————————————————————————————

如何使用ChatPaper？

为了让更多科研人更高效的获取文献知识，AMiner基于GLM-130B大模型能力，开发了Chatpaper，帮助科研人快速提高检索、阅读论文效率，获取最新领域研究动态，让科研工作更加游刃有余。

图片
ChatPaper是一款集检索、阅读、知识问答于一体的对话式私有知识库，AMiner希望通过技术的力量，让大家更加高效地获取知识。

ChatPaper：https://www.aminer.cn/chat/g