当前位置:网站首页>This week's reading list: from neuroips 2020 to emnlp 2020

This week's reading list: from neuroips 2020 to emnlp 2020

2020-12-06 10:53:59 osc_ zd5cuok7

In the era of fragmented reading full of eyeballs , Less and less people will pay attention to the exploration and thinking behind each paper . In this column , You'll be quick get The highlights and pain points of each selected paper , Keep up AI Cutting edge results . If you also want your research results to be seen by more people , Welcome to reply backstage 「 Paper recommendation 」.

A quick overview of the paper

This paper introduces

He Kaiming is unsupervised and says he studies the latest job

Reinforcement learning in natural language processing

be based on Transformer The ranking model of

The new paradigm of large-scale pre training model

Contrastive learning based on antagonistic learning samples

Pre training language model robust fine tuning

be based on Kornia The method of differentiable data augmentation based on

Using question and answer model to solve natural language understanding task

01

Unsupervised means learning

Paper title Exploring Simple Siamese Representation Learning

Author of the paper :Xinlei Chen / Kaiming He

Thesis link http://www.paperweekly.site/papers/4652

This paper is a new work of he Kaiming on unsupervised expressive learning , It's worth reading . This paper focuses on the twin network which is widely used (Siamese Network) Analyze , Take the very popular comparative learning as an example , Twin networks use the same network to process two different representations of the same input , By narrowing two positive pair It means , Pull two away negative pair Between the expression of , So we can learn the invariance in input , So as to learn the representation of input better . Through the experimental analysis, this paper concludes that the structure of twin network plays the most important role , Other methods don't work that much .

besides , The author also puts forward a kind of “stop-gradient” The algorithm of , The algorithm is mainly applied to the model loss When giving feedback , Through the mechanism of gradient termination , To update only one of them encoder, The collapse solution in the twin network is realized (collapsing) It's good to avoid . And this simple structure can be used in ImageNet And the downstream mission achieved very good results . In order to prove the effectiveness of this algorithm , The author has done a lot of experiments , The superiority of the algorithm is fully proved . In addition, the author also discusses in depth where the algorithm is in the optimization model . Simple method , The effect is effective , A great work worth reading carefully .

02

NLP Intensive learning in

Paper title :Learning from Human Feedback: Challenges for Real-World Reinforcement Learning in NLP

Author of the paper :Julia Kreutzer / Stefan Riezler / Carolin Lawrence

Source of the paper :NeurIPS 2020

Thesis link :http://www.paperweekly.site/papers/4626

This article is published by Google and Heidelberg University in NeurIPS 2020 The job of . This is an exploratory article , Mainly aimed at NLP Intensive learning in , Analyzes the use of real world log Information to assist reinforcement learning (sequence to sequence learning) The problem is . The author thinks that in the real world NLP The system collects a lot of log information of interaction with users , For example, in automatic translation , Users can give feedback on the quality of the translation , At the same time, some simple operations are used to improve the quality of translation . But considering some of the requirements and limitations of the online system , There are some problems in using these feedback online to update reinforcement learning systems , Therefore, this paper focuses on NLP Feedback for reinforcement learning is used offline in .

The author sorts out several current challenges in using these interactive log feedback to improve system performance , The main thing is deterministic logging and reliable data problem . The former is mainly analyzed in order not to provide users with poor exploratory results ,RL The system tends to provide the most likely result , Limit RL The exploration and performance of . The latter focuses on the credibility and availability of data . Not all feedback data are valid data , So how to determine the quality of data is also a big challenge . Specific details and specific challenges can be read in the original text . This article can be seen as the future use of real interaction log RL Some possible research directions are provided .

03

be based on Transformer The ranking model of

Paper title :Modularized Transfomer-based Ranking Framework

Author of the paper Luyu Gao / Zhuyun Dai / Jamie Callan

Source of the paper :EMNLP 2020

Thesis link :http://www.paperweekly.site/papers/4662

This article is about CMU Published in EMNLP 2020 The job of . be based on Transformer The latest innovation of the ranking model has promoted the latest development of information retrieval . however , these transformer It's expensive to calculate , And their opaque hidden state makes it difficult to understand the ranking process .

In this work, the authors will Transformer ranker Modularization into individual modules , For text representation and interaction . The authors will show how the design uses offline precomputed representations and lightweight online interactions to significantly speed up rankings . Modular design is also easier to explain , And for Transformer The ranking process in the rankings provides insights . The author's experiments on a large supervised ranking dataset show that MORES The effectiveness and efficiency of . It's with the most advanced BERT The ranking machine works just as well , And the highest ranking speed can improve 120 times .

04

The new paradigm of large-scale pre training model

Paper title :Train No Evil: Selective Masking for Task-Guided Pre-Training

Author of the paper :Yuxian Gu / Zhengyan Zhang / Xiaozhi Wang / Zhiyuan Liu / Maosong Sun

Source of the paper :EMNLP 2020

Thesis link :http://www.paperweekly.site/papers/4631

Code link :https://github.com/thunlp/SelectiveMasking

This article is published by Professor Liu Zhiyuan of Tsinghua University in EMNLP 2020 The job of , This article shows once again that Do Not Stop Pre-training Importance . In the past, we used the pre training model according to pre-train+fine-tune A two-step paradigm , however fine-tune Because of the lack of labeled data, we can't make full use of it pre-train The full performance of the model .

Therefore, this paper puts forward a new method in which pre-train and fine-tune Add a Selective Masking The pre training phase of . seeing the name of a thing one thinks of its function , In this new pre training phase , The model predicts the words that are important to the model . At this stage, we use in-domain The data of , It can make the pre training model better adapt to the downstream tasks . The experimental results on two sentence analysis tasks show that , This method can be used in less than the amount of calculation 50% The performance of the original model is equal to or even better than that of the original model , It shows that the method in this paper is effective .

05

Contrastive learning based on antagonistic learning samples

Paper title :Contrastive Learning with Adversarial Examples

Author of the paper :Chih-Hui Ho / Nuno Vasconcelos

Source of the paper :NeurIPS 2020

Thesis link :http://www.paperweekly.site/papers/4619

This article is about UCSD Published in NeurIPS 2020 The job of . This paper is about comparative learning . Comparative learning (CL) It is a popular visual representation self supervised learning (SSL) technology . It uses the augmentation of unlabeled training sample pairs to define a classification task .

Although a lot of work has been done in the enhancement process , But the previous work did not solve how to choose a challenging negative pair , Because the images in the sampling batch are processed independently . In this paper, we introduce a series of new counter learning samples to solve this problem , And use these examples to define a new SSL Antagonistic training algorithm CLAE.CLAE And many in the literature CL Method compatible . Experiments show that , This method improves the existing multiple CL Baseline performance over multiple datasets .

06

Pre training language model

Paper title :InfoBERT: Improving Robustness of Language Models from An Information Theoretic Perspective

Author of the paper :Boxin Wang / Shuohang Wang / Yu Cheng / Zhe Gan / Ruoxi Jia / Bo Li / Jingjing Liu

Thesis link :http://www.paperweekly.site/papers/4644

This article is from UIUC And Microsoft . In recent years, studies have shown that ,BERT and RoBERTa Such large-scale pre training language models are easy to be written word-level Against the attack . This paper aims to solve this problem from the perspective of information theory , And to put forward InfoBERT This new learning framework , Used to fine tune the pre training language model . 

InfoBERT It contains two mutual information based regularizers for model training :1) Information Bottleneck Regularizer , Used to suppress noisy mutual information between input and feature representation ;2)Anchored Feature adjuster , It can increase the mutual information between local stability features and global features . A lot of experiments show that ,InfoBERT Reasoning in natural language (NLI) And questions and answers (QA) The latest robustness is achieved on multiple adversarial datasets of the task .

07

be based on Kornia The data can be widened in the future

Paper title :Differentiable Data Augmentation with Kornia

Author of the paper :Jian Shi / Edgar Riba / Dmytro Mishkin / Francesc Moreno / Anguelos Nicolaou

Source of the paper :NeurIPS 2020

Thesis link :http://www.paperweekly.site/papers/4643

This article is published by the Chinese university of Hong Kong NeurIPS 2020 The job of . In this paper, based on the Kornia And integrate it into PyTorch In workflow . This paper focuses on how to efficiently implement differentiable data augmentation and the ease of use of this method .

08

natural language understanding

Paper title :Language Model is All You Need: Natural Language Understanding as Question Answering

Author of the paper :Mahdi Namazifar / Alexandros Papangelis / Gokhan Tur / Dilek Hakkani-Tür

Thesis link :http://www.paperweekly.site/papers/4606

This article is from Amazon AI, The topic begins with attention is all you need similar , It's fascinating . To be specific , This paper studies transfer learning, Many tasks of natural language understanding can be solved by question answering model .

This idea is a bit similar to the use of the pre training model . First, we train a question answering model in the source domain , The source domain here is a question and answer dataset , And then for the task in the target domain , In this article, the author mainly considers two tasks : The first is slot detection, A question and answer for a specific attribute , The second is intent detection, By adding... At the beginning of a sentence Yes perhaps No Turn it into a question and answer question , such NLU It's converted to QA problem . And then we'll train the good ones QA The model is here transfer Then fine tune the target area , So as to improve the effect , In addition, the author also found that the fine-tuning model , The performance of Q & A will also be improved .

besides , The author also proposes a serialization of transfer learning , That is, the task of the target domain will be continuously transfer To the question and answer of the source domain , And fine tune it step by step QA Model .

Read more


????

Now? , stay 「 You know 」 We can also be found

Go to Zhihu home page and search 「PaperWeekly」

Click on 「 Focus on 」 Subscribe to our column

About PaperWeekly

PaperWeekly It's a recommendation 、 Reading 、 Discuss 、 An academic platform for reporting the achievements of the frontier papers on artificial intelligence . If you study or engage in AI field , Welcome to clicking on the official account 「 Communication group 」, The little assistant will take you into PaperWeekly In the communication group .

版权声明
本文为[osc_ zd5cuok7]所创,转载请带上原文链接,感谢
https://chowdera.com/2020/12/20201206105253480x.html