This paper presents a scalable approach for semi-supervised learning on graph-structured data using an efficient variant of convolutional neural networks that operate directly on graphs. The authors motivate their convolutional architecture using a localized first-order approximation of spectral graph convolutions. The paper reports linear scaling in the number of graph edges and hidden representations that encode local graph structure and node features. Experiments on citation networks and a knowledge graph dataset show the approach outperforming related methods by a significant margin.
This brief summarizes a paper that trains image models from natural language supervision by predicting which caption matches which image at internet scale, then uses language to enable zero-shot transfer to many downstream vision tasks.
This paper analyzes why many machine learning models, including neural networks, misclassify adversarial examples created by small worst-case perturbations, and it presents a fast method to generate such examples for adversarial training.
This brief summarizes the paper “Learning Transferable Visual Models From Natural Language Supervision,” which trains vision models by predicting which caption matches which image on 400 million internet (image, text) pairs, then uses natural language to enable zero-shot transfer to many downstream tasks.
This paper presents two gradient-based techniques for visualizing image classification models learned with deep convolutional networks: class score maximization for class “prototypes,” and per-image, per-class saliency maps that can be used for weakly supervised object segmentation, with a stated connection to deconvolutional networks.
This paper reports a convolutional neural network trained with a variant of Q-learning that learns control policies directly from raw Atari 2600 pixels by outputting a value function over future rewards, and it presents results on seven Arcade Learning Environment games using the same architecture and learning algorithm across games.
EfficientNet studies how to scale convolutional neural networks when more compute is available, rather than designing models only for a fixed resource budget. The paper reports that carefully balancing network depth, width, and resolution yields better performance, and it introduces a compound coefficient that uniformly scales these three dimensions.
This paper introduces Paragraph Vector, an unsupervised method that learns fixed-length vector representations for variable-length text such as sentences, paragraphs, and documents by training a dense document vector to predict words in the document.
This paper advocates training deep networks directly from decentralized, privacy-sensitive mobile data by leaving data on devices and aggregating locally computed updates into a shared model. It names this decentralized approach Federated Learning and presents a practical method based on iterative model averaging, together with an extensive empirical evaluation across five model architectures and four datasets.
This paper proposes a unified neural network architecture and learning algorithm for multiple NLP tasks, aiming to avoid task-specific feature engineering by learning internal representations from vast amounts of mostly unlabeled data.
The paper presents ADADELTA as a novel per-dimension learning rate method for gradient descent that dynamically adapts over time using only first order information. The paper also states that ADADELTA has minimal computational overhead beyond vanilla stochastic gradient descent, requires no manual tuning of a learning rate, and appears robust to several kinds of variation and noise. The snippet reports promising results compared to other methods on MNIST using a single machine and on a large scale voice dataset in a distributed cluster environment.
This paper proposes prototypical networks for few-shot classification, where a classifier must generalize to new classes not seen in the training set given only a small number of examples of each new class. Prototypical networks learn a metric space where classification is performed by computing distances to prototype representations of each class, and the paper reports excellent results from this approach.
The paper reports a method for learning image representations from scratch by predicting which caption matches which image on 400 million internet-collected (image, text) pairs, and then using natural language to enable zero-shot transfer to downstream tasks.
This paper investigates discriminatively trained deep convolutional network architectures for video action recognition, focusing on complementary appearance information from still frames and motion information between frames via a two-stream spatial–temporal design, multi-frame dense optical flow, and multi-task learning across two action datasets to increase training data and improve performance.
This paper reports two counter-intuitive properties of deep neural networks, connecting them to the models’ expressiveness and learned solutions. It describes (1) a lack of distinction between individual high-level units and random linear combinations of them, and (2) fairly discontinuous learned input-output mappings that allow misclassification via a certain imperceptible perturbation.
This paper describes an automatic hyperparameter-tuning approach using Bayesian optimization, where a learning algorithm’s generalization performance is modeled as a sample from a Gaussian process and the resulting posterior distribution guides which settings to try next.
This paper presents GNMT, Google’s Neural Machine Translation (NMT) system, motivated by the need for practical translation that balances accuracy and speed in deployments and services. The paper describes NMT as an end-to-end approach with potential to overcome weaknesses of conventional phrase-based translation systems, while also noting known challenges such as training and inference cost and rare-word difficulty.
This paper presents Fashion-MNIST, a dataset of 28×28 grayscale images of 70,000 fashion products from 10 categories, with a 60,000/10,000 train/test split, intended as a direct drop-in replacement for MNIST for benchmarking machine learning algorithms.
This paper presents Model-Agnostic Meta-Learning (MAML), a meta-learning algorithm that trains model parameters so that a small number of gradient steps using a small amount of data from a new task yields good generalization on that task.
YOLOv3 reports a set of incremental design changes and a newly trained network that is slightly larger than the prior version, more accurate, and still fast, with concrete speed–accuracy numbers at 320×320 and comparisons to SSD, RetinaNet, and mAP@50 timing on a Titan X.
SqueezeNet proposes a small deep neural network architecture that reaches AlexNet-level ImageNet accuracy while using far fewer parameters, and it reports additional size reductions using model-compression techniques.
This paper formulates precipitation nowcasting as spatiotemporal sequence forecasting and introduces ConvLSTM by adding convolutional structure to LSTM transitions, reporting better capture of spatiotemporal correlations and stronger results than FC-LSTM and the operational ROVER system.
This paper reports that large feedforward neural networks trained on small training sets often perform poorly on held-out test data, and it presents random “dropout,” which omits half of the feature detectors on each training case, as a method that greatly reduces overfitting and improves benchmark results in tasks including speech and object recognition.
This paper presents SimCLR as “a simple framework for contrastive learning of visual representations,” and it reports a systematic study of major framework components that affect contrastive prediction tasks. The paper reports three findings about augmentations, a learnable nonlinear transformation before the contrastive loss, and scaling with batch size and training steps.
PointNet++ extends PointNet by building a hierarchical network over nested partitions of a point set, using metric-space distances to learn local features at increasing contextual scales and handling non-uniform sampling densities with multi-scale feature aggregation.
This paper adapts ideas underlying the success of Deep Q-Learning to the continuous action domain and presents an actor-critic, model-free algorithm based on the deterministic policy gradient that can operate over continuous action spaces. The paper reports that, with the same learning algorithm, network architecture, and hyper-parameters, the method robustly solves more than 20 simulated physics tasks and can learn some tasks end-to-end from raw pixel inputs.
This paper revisits atrous convolution for semantic image segmentation and presents the DeepLabv3 system with multi-scale modules and an augmented ASPP design that adds image-level features for global context.
This paper introduces an attention-based neural model that learns to generate image descriptions, supports both deterministic and stochastic training, and visualizes where the model focuses while producing words.
This paper presents SHAP (SHapley Additive exPlanations), a unified framework for interpreting predictions from complex machine learning models by assigning each feature an importance value for a particular prediction.
This paper introduces deep convolutional generative adversarial networks (DCGANs), a class of convolutional neural networks with specific architectural constraints, and reports evidence that the generator and discriminator learn hierarchical visual representations that transfer to novel tasks as general image features.
This paper introduces conditional generative adversarial nets (cGANs) by feeding a conditioning variable y to both the generator and discriminator, and reports demonstrations on MNIST class-conditional digit generation plus preliminary examples for multimodal modeling and image tagging.
This paper proposes extending an encoder–decoder neural machine translation model by letting the model soft-search the source sentence for the parts most relevant to predicting each target word, addressing a conjectured bottleneck from encoding the entire source sentence into a single fixed-length vector.
TensorFlow is presented as a machine learning system that operates at large scale and in heterogeneous environments. The paper describes TensorFlow as using dataflow graphs to represent computation, shared state, and the operations that mutate that state.
This paper studies transfer learning for NLP through a single text-to-text framework, comparing pre-training objectives, architectures, data, and transfer approaches across many tasks, and reporting state-of-the-art results on multiple benchmarks using scale and the Colossal Clean Crawled Corpus.
A brief summary of arXiv:1412.3555, which compares recurrent units in RNNs and reports that gated units such as LSTM and GRU outperform traditional tanh units on polyphonic music and speech signal modeling tasks, with GRU comparable to LSTM.
MobileNets introduces an efficient CNN family for mobile and embedded vision that uses depth-wise separable convolutions and two global hyper-parameters to trade off latency and accuracy across tasks such as ImageNet classification and object detection.
This brief summarizes the TensorFlow paper (arXiv:1603.04467), focusing on what it claims about expressing machine learning computations and executing them across heterogeneous devices from mobile hardware to distributed clusters.
YOLOv4 studies which CNN features and training techniques reliably improve object detection, and it reports combining selected components such as CSP, CmBN, SAT, Mish, Mosaic augmentation, DropBlock, and CIoU loss to reach state-of-the-art results.
This paper describes knowledge distillation as a way to compress the predictive behavior of an expensive ensemble into a single model that is easier to deploy, and it reports results on MNIST and an acoustic model used in a commercial system.
This paper presents an end-to-end approach to mapping one sequence to another using multilayered LSTMs in an encoder–decoder setup, and it reports results on WMT’14 English-to-French translation with a BLEU score of 34.8 under an out-of-vocabulary penalty.
This brief summarizes the PyTorch paper (arXiv:1912.01703), which argues that usability and speed can be compatible in a deep learning framework through an imperative, Pythonic design that remains efficient and supports accelerators like GPUs.
This paper proposes two new model architectures for learning continuous vector representations of words from very large datasets, and it reports improved accuracy on word similarity evaluations at substantially lower computational cost, including training high-quality vectors from 1.6 billion words in less than a day.
This brief summarizes arXiv:1310.4546, which extends the continuous Skip-gram model to improve vector quality and training speed, introduces subsampling of frequent words and negative sampling, and discusses phrase learning to address word-order and idiom limitations.
Faster R-CNN (arXiv:1506.01497) introduces a Region Proposal Network that shares full-image convolutional features with a detection network to enable nearly cost-free region proposals, and it reports a merged design that shares convolutional features between the RPN and Fast R-CNN.
This paper presents scikit-learn as a Python module and package that integrates a wide range of state-of-the-art machine learning algorithms aimed at medium-scale supervised and unsupervised problems. It states a focus on bringing machine learning to non-specialists using a general-purpose high-level language, with emphasis on ease of use, performance, documentation, and API consistency. It also reports minimal dependencies, simplified BSD licensing intended to encourage academic and commercial use, and public downloads of code, binaries, and documentation via scikit-learn.org.
This paper introduces Vision Transformer (ViT), a “pure transformer” approach that treats an image as a sequence of patches and applies a Transformer directly for image classification. The authors report strong transfer results after large-scale pre-training and state that ViT can match or outperform state-of-the-art convolutional networks while using substantially fewer computational resources to train.
Batch Normalization introduces per-mini-batch normalization of layer inputs as part of the network architecture to reduce “internal covariate shift,” accelerate training, enable higher learning rates, relax initialization sensitivity, and sometimes reduce the need for Dropout.
This paper evaluates how increasing convolutional network depth affects accuracy for large-scale image recognition using an architecture built from very small 3×3 convolution filters, reporting significant improvements by pushing depth to 16–19 weight layers and describing results connected to an ImageNet Challenge 2014 submission and transfer to other datasets.
This paper analyzes when L2 regularization matches weight decay and reports that the equivalence breaks for adaptive optimizers like Adam, motivating a decoupled weight decay update that the paper reports improves generalization and tuning behavior.
The paper proposes a family of policy gradient methods for reinforcement learning that alternates between sampling data by interacting with an environment and optimizing a surrogate objective with stochastic gradient ascent. The paper calls the methods proximal policy optimization (PPO) and reports that PPO is simpler to implement than TRPO while performing well on benchmark tasks such as simulated robotic locomotion and Atari game playing.
Graph Attention Networks (GATs) are neural network architectures for graph-structured data that use stacked masked self-attentional layers so nodes can attend over neighborhood features and assign different weights to different neighbors without costly matrix operations like inversion.
Denoising Diffusion Probabilistic Models (DDPM) presents diffusion probabilistic models for high-quality image synthesis, training them with a weighted variational bound linked to denoising score matching with Langevin dynamics, and reporting strong results on CIFAR-10 and LSUN.
The paper proposes the Transformer, a sequence transduction architecture built solely on attention mechanisms and designed to remove recurrence and convolutions from encoder-decoder models. It reports superior machine translation quality with improved parallelizability and substantially reduced training time, and it also reports successful application to English constituency parsing.