This brief summarizes the TensorFlow paper (arXiv:1603.04467), focusing on what it claims about expressing machine learning computations and executing them across heterogeneous devices from mobile hardware to distributed clusters.
YOLOv4 studies which CNN features and training techniques reliably improve object detection, and it reports combining selected components such as CSP, CmBN, SAT, Mish, Mosaic augmentation, DropBlock, and CIoU loss to reach state-of-the-art results.
This paper describes knowledge distillation as a way to compress the predictive behavior of an expensive ensemble into a single model that is easier to deploy, and it reports results on MNIST and an acoustic model used in a commercial system.
This paper presents an end-to-end approach to mapping one sequence to another using multilayered LSTMs in an encoder–decoder setup, and it reports results on WMT’14 English-to-French translation with a BLEU score of 34.8 under an out-of-vocabulary penalty.
This brief summarizes the PyTorch paper (arXiv:1912.01703), which argues that usability and speed can be compatible in a deep learning framework through an imperative, Pythonic design that remains efficient and supports accelerators like GPUs.
This paper proposes two new model architectures for learning continuous vector representations of words from very large datasets, and it reports improved accuracy on word similarity evaluations at substantially lower computational cost, including training high-quality vectors from 1.6 billion words in less than a day.
This brief summarizes arXiv:1310.4546, which extends the continuous Skip-gram model to improve vector quality and training speed, introduces subsampling of frequent words and negative sampling, and discusses phrase learning to address word-order and idiom limitations.
Faster R-CNN (arXiv:1506.01497) introduces a Region Proposal Network that shares full-image convolutional features with a detection network to enable nearly cost-free region proposals, and it reports a merged design that shares convolutional features between the RPN and Fast R-CNN.
This paper presents scikit-learn as a Python module and package that integrates a wide range of state-of-the-art machine learning algorithms aimed at medium-scale supervised and unsupervised problems. It states a focus on bringing machine learning to non-specialists using a general-purpose high-level language, with emphasis on ease of use, performance, documentation, and API consistency. It also reports minimal dependencies, simplified BSD licensing intended to encourage academic and commercial use, and public downloads of code, binaries, and documentation via scikit-learn.org.
This paper introduces Vision Transformer (ViT), a “pure transformer” approach that treats an image as a sequence of patches and applies a Transformer directly for image classification. The authors report strong transfer results after large-scale pre-training and state that ViT can match or outperform state-of-the-art convolutional networks while using substantially fewer computational resources to train.
Batch Normalization introduces per-mini-batch normalization of layer inputs as part of the network architecture to reduce “internal covariate shift,” accelerate training, enable higher learning rates, relax initialization sensitivity, and sometimes reduce the need for Dropout.
This paper evaluates how increasing convolutional network depth affects accuracy for large-scale image recognition using an architecture built from very small 3×3 convolution filters, reporting significant improvements by pushing depth to 16–19 weight layers and describing results connected to an ImageNet Challenge 2014 submission and transfer to other datasets.
This paper analyzes when L2 regularization matches weight decay and reports that the equivalence breaks for adaptive optimizers like Adam, motivating a decoupled weight decay update that the paper reports improves generalization and tuning behavior.
The paper proposes a family of policy gradient methods for reinforcement learning that alternates between sampling data by interacting with an environment and optimizing a surrogate objective with stochastic gradient ascent. The paper calls the methods proximal policy optimization (PPO) and reports that PPO is simpler to implement than TRPO while performing well on benchmark tasks such as simulated robotic locomotion and Atari game playing.
Graph Attention Networks (GATs) are neural network architectures for graph-structured data that use stacked masked self-attentional layers so nodes can attend over neighborhood features and assign different weights to different neighbors without costly matrix operations like inversion.
Denoising Diffusion Probabilistic Models (DDPM) presents diffusion probabilistic models for high-quality image synthesis, training them with a weighted variational bound linked to denoising score matching with Langevin dynamics, and reporting strong results on CIFAR-10 and LSUN.
The paper proposes the Transformer, a sequence transduction architecture built solely on attention mechanisms and designed to remove recurrence and convolutions from encoder-decoder models. It reports superior machine translation quality with improved parallelizability and substantially reduced training time, and it also reports successful application to English constituency parsing.