This brief summarizes the paper “Learning Transferable Visual Models From Natural Language Supervision,” which trains vision models by predicting which caption matches which image on 400 million internet (image, text) pairs, then uses natural language to enable zero-shot transfer to many downstream tasks.
This paper presents two gradient-based techniques for visualizing image classification models learned with deep convolutional networks: class score maximization for class “prototypes,” and per-image, per-class saliency maps that can be used for weakly supervised object segmentation, with a stated connection to deconvolutional networks.
This paper reports a convolutional neural network trained with a variant of Q-learning that learns control policies directly from raw Atari 2600 pixels by outputting a value function over future rewards, and it presents results on seven Arcade Learning Environment games using the same architecture and learning algorithm across games.
EfficientNet studies how to scale convolutional neural networks when more compute is available, rather than designing models only for a fixed resource budget. The paper reports that carefully balancing network depth, width, and resolution yields better performance, and it introduces a compound coefficient that uniformly scales these three dimensions.
This paper introduces Paragraph Vector, an unsupervised method that learns fixed-length vector representations for variable-length text such as sentences, paragraphs, and documents by training a dense document vector to predict words in the document.
This paper advocates training deep networks directly from decentralized, privacy-sensitive mobile data by leaving data on devices and aggregating locally computed updates into a shared model. It names this decentralized approach Federated Learning and presents a practical method based on iterative model averaging, together with an extensive empirical evaluation across five model architectures and four datasets.
This paper proposes a unified neural network architecture and learning algorithm for multiple NLP tasks, aiming to avoid task-specific feature engineering by learning internal representations from vast amounts of mostly unlabeled data.
The paper presents ADADELTA as a novel per-dimension learning rate method for gradient descent that dynamically adapts over time using only first order information. The paper also states that ADADELTA has minimal computational overhead beyond vanilla stochastic gradient descent, requires no manual tuning of a learning rate, and appears robust to several kinds of variation and noise. The snippet reports promising results compared to other methods on MNIST using a single machine and on a large scale voice dataset in a distributed cluster environment.
This paper proposes prototypical networks for few-shot classification, where a classifier must generalize to new classes not seen in the training set given only a small number of examples of each new class. Prototypical networks learn a metric space where classification is performed by computing distances to prototype representations of each class, and the paper reports excellent results from this approach.
The paper reports a method for learning image representations from scratch by predicting which caption matches which image on 400 million internet-collected (image, text) pairs, and then using natural language to enable zero-shot transfer to downstream tasks.
This paper investigates discriminatively trained deep convolutional network architectures for video action recognition, focusing on complementary appearance information from still frames and motion information between frames via a two-stream spatial–temporal design, multi-frame dense optical flow, and multi-task learning across two action datasets to increase training data and improve performance.
This paper reports two counter-intuitive properties of deep neural networks, connecting them to the models’ expressiveness and learned solutions. It describes (1) a lack of distinction between individual high-level units and random linear combinations of them, and (2) fairly discontinuous learned input-output mappings that allow misclassification via a certain imperceptible perturbation.
This paper describes an automatic hyperparameter-tuning approach using Bayesian optimization, where a learning algorithm’s generalization performance is modeled as a sample from a Gaussian process and the resulting posterior distribution guides which settings to try next.
This paper presents GNMT, Google’s Neural Machine Translation (NMT) system, motivated by the need for practical translation that balances accuracy and speed in deployments and services. The paper describes NMT as an end-to-end approach with potential to overcome weaknesses of conventional phrase-based translation systems, while also noting known challenges such as training and inference cost and rare-word difficulty.
This paper presents Fashion-MNIST, a dataset of 28×28 grayscale images of 70,000 fashion products from 10 categories, with a 60,000/10,000 train/test split, intended as a direct drop-in replacement for MNIST for benchmarking machine learning algorithms.
This paper presents Model-Agnostic Meta-Learning (MAML), a meta-learning algorithm that trains model parameters so that a small number of gradient steps using a small amount of data from a new task yields good generalization on that task.
YOLOv3 reports a set of incremental design changes and a newly trained network that is slightly larger than the prior version, more accurate, and still fast, with concrete speed–accuracy numbers at 320×320 and comparisons to SSD, RetinaNet, and mAP@50 timing on a Titan X.
SqueezeNet proposes a small deep neural network architecture that reaches AlexNet-level ImageNet accuracy while using far fewer parameters, and it reports additional size reductions using model-compression techniques.
This paper formulates precipitation nowcasting as spatiotemporal sequence forecasting and introduces ConvLSTM by adding convolutional structure to LSTM transitions, reporting better capture of spatiotemporal correlations and stronger results than FC-LSTM and the operational ROVER system.
This paper reports that large feedforward neural networks trained on small training sets often perform poorly on held-out test data, and it presents random “dropout,” which omits half of the feature detectors on each training case, as a method that greatly reduces overfitting and improves benchmark results in tasks including speech and object recognition.
This paper presents SimCLR as “a simple framework for contrastive learning of visual representations,” and it reports a systematic study of major framework components that affect contrastive prediction tasks. The paper reports three findings about augmentations, a learnable nonlinear transformation before the contrastive loss, and scaling with batch size and training steps.
PointNet++ extends PointNet by building a hierarchical network over nested partitions of a point set, using metric-space distances to learn local features at increasing contextual scales and handling non-uniform sampling densities with multi-scale feature aggregation.
This paper adapts ideas underlying the success of Deep Q-Learning to the continuous action domain and presents an actor-critic, model-free algorithm based on the deterministic policy gradient that can operate over continuous action spaces. The paper reports that, with the same learning algorithm, network architecture, and hyper-parameters, the method robustly solves more than 20 simulated physics tasks and can learn some tasks end-to-end from raw pixel inputs.
This paper revisits atrous convolution for semantic image segmentation and presents the DeepLabv3 system with multi-scale modules and an augmented ASPP design that adds image-level features for global context.
This paper introduces an attention-based neural model that learns to generate image descriptions, supports both deterministic and stochastic training, and visualizes where the model focuses while producing words.
This paper presents SHAP (SHapley Additive exPlanations), a unified framework for interpreting predictions from complex machine learning models by assigning each feature an importance value for a particular prediction.
This paper introduces deep convolutional generative adversarial networks (DCGANs), a class of convolutional neural networks with specific architectural constraints, and reports evidence that the generator and discriminator learn hierarchical visual representations that transfer to novel tasks as general image features.
This paper analyzes when L2 regularization matches weight decay and reports that the equivalence breaks for adaptive optimizers like Adam, motivating a decoupled weight decay update that the paper reports improves generalization and tuning behavior.
The paper proposes a family of policy gradient methods for reinforcement learning that alternates between sampling data by interacting with an environment and optimizing a surrogate objective with stochastic gradient ascent. The paper calls the methods proximal policy optimization (PPO) and reports that PPO is simpler to implement than TRPO while performing well on benchmark tasks such as simulated robotic locomotion and Atari game playing.
Graph Attention Networks (GATs) are neural network architectures for graph-structured data that use stacked masked self-attentional layers so nodes can attend over neighborhood features and assign different weights to different neighbors without costly matrix operations like inversion.
Denoising Diffusion Probabilistic Models (DDPM) presents diffusion probabilistic models for high-quality image synthesis, training them with a weighted variational bound linked to denoising score matching with Langevin dynamics, and reporting strong results on CIFAR-10 and LSUN.
The paper proposes the Transformer, a sequence transduction architecture built solely on attention mechanisms and designed to remove recurrence and convolutions from encoder-decoder models. It reports superior machine translation quality with improved parallelizability and substantially reduced training time, and it also reports successful application to English constituency parsing.