ICLR 2019
International Conference on Learning Representations
New Orleans, Louisiana, United States May 6 - May 9, 2019 https://iclr.cc/Conferences/2019
Questions or Concerns
Please contact the OpenReview support team at<EMAIL_ADDRESS>with any questions or concerns about the
OpenReview platform.
Please contact the ICLR 2019 Program Chairs at<EMAIL_ADDRESS>with
any questions or concerns about conference administration or policy.
-
BA-Net: Dense Bundle Adjustment Networks
Show details- Abstract: This paper introduces a network architecture to solve the structure-from-motion (SfM) problem via feature-metric bundle adjustment (BA), which explicitly enforces multi-view geometry constraints in the form of feature-metric error. The whole pipeline is differentiable, so that the network can learn suitable features that make the BA problem more tractable. Furthermore, this work introduces a novel depth parameterization to recover dense per-pixel depth. The network first generates several basis depth maps according to the input image, and optimizes the final depth as a linear combination of these basis depth maps via feature-metric BA. The basis depth maps generator is also learned via end-to-end training. The whole system nicely combines domain knowledge (i.e. hard-coded multi-view geometry constraints) and deep learning (i.e. feature learning and basis depth maps learning) to address the challenging dense SfM problem. Experiments on large scale real data prove the success of the proposed method.
- Keywords: Structure-from-Motion, Bundle Adjustment, Dense Depth Estimation
- TL;DR: This paper introduces a network architecture to solve the structure-from-motion (SfM) problem via feature bundle adjustment (BA)
-
Deterministic Variational Inference for Robust Bayesian Neural Networks
Show details- Abstract: Bayesian neural networks (BNNs) hold great promise as a flexible and principled solution to deal with uncertainty when learning from finite data. Among approaches to realize probabilistic inference in deep neural networks, variational Bayes (VB) is theoretically grounded, generally applicable, and computationally efficient. With wide recognition of potential advantages, why is it that variational Bayes has seen very limited practical use for BNNs in real applications? We argue that variational inference in neural networks is fragile: successful implementations require careful initialization and tuning of prior variances, as well as controlling the variance of Monte Carlo gradient estimates. We provide two innovations that aim to turn VB into a robust inference tool for Bayesian neural networks: first, we introduce a novel deterministic method to approximate moments in neural networks, eliminating gradient variance; second, we introduce a hierarchical prior for parameters and a novel Empirical Bayes procedure for automatically selecting prior variances. Combining these two innovations, the resulting method is highly efficient and robust. On the application of heteroscedastic regression we demonstrate good predictive performance over alternative approaches.
- Keywords: Bayesian neural network, variational inference, variational bayes, variance reduction, empirical bayes
- TL;DR: A method for eliminating gradient variance and automatically tuning priors for effective training of bayesian neural networks
-
Ordered Neurons: Integrating Tree Structures into Recurrent Neural Networks
Show details- Abstract: Natural language is hierarchically structured: smaller units (e.g., phrases) are nested within larger units (e.g., clauses). When a larger constituent ends, all of the smaller constituents that are nested within it must also be closed. While the standard LSTM architecture allows different neurons to track information at different time scales, it does not have an explicit bias towards modeling a hierarchy of constituents. This paper proposes to add such inductive bias by ordering the neurons; a vector of master input and forget gates ensures that when a given neuron is updated, all the neurons that follow it in the ordering are also updated. Our novel recurrent architecture, ordered neurons LSTM (ON-LSTM), achieves good performance on four different tasks: language modeling, unsupervised parsing, targeted syntactic evaluation, and logical inference.
- Keywords: Deep Learning, Natural Language Processing, Recurrent Neural Networks, Language Modeling
- TL;DR: We introduce a new inductive bias that integrates tree structures in recurrent neural networks.
-
Large Scale GAN Training for High Fidelity Natural Image Synthesis
Show details- Abstract: Despite recent progress in generative image modeling, successfully generating high-resolution, diverse samples from complex datasets such as ImageNet remains an elusive goal. To this end, we train Generative Adversarial Networks at the largest scale yet attempted, and study the instabilities specific to such scale. We find that applying orthogonal regularization to the generator renders it amenable to a simple "truncation trick", allowing fine control over the trade-off between sample fidelity and variety by reducing the variance of the Generator's input. Our modifications lead to models which set the new state of the art in class-conditional image synthesis. When trained on ImageNet at 128x128 resolution, our models (BigGANs) achieve an Inception Score (IS) of 166.3 and Frechet Inception Distance (FID) of 9.6, improving over the previous best IS of 52.52 and FID of 18.65.
- Keywords: GANs, Generative Models, Large Scale Training, Deep Learning
- TL;DR: GANs benefit from scaling up.
-
Learning deep representations by mutual information estimation and maximization
Show details- Abstract: This work investigates unsupervised learning of representations by maximizing mutual information between an input and the output of a deep neural network encoder. Importantly, we show that structure matters: incorporating knowledge about locality in the input into the objective can significantly improve a representation's suitability for downstream tasks. We further control characteristics of the representation by matching to a prior distribution adversarially. Our method, which we call Deep InfoMax (DIM), outperforms a number of popular unsupervised learning methods and compares favorably with fully-supervised learning on several classification tasks in with some standard architectures. DIM opens new avenues for unsupervised learning of representations and is an important step towards flexible formulations of representation learning objectives for specific end-goals.
- Keywords: representation learning, unsupervised learning, deep learning
- TL;DR: We learn deep representation by maximizing mutual information, leveraging structure in the objective, and are able to compute with fully supervised classifiers with comparable architectures
-
KnockoffGAN: Generating Knockoffs for Feature Selection using Generative Adversarial Networks
Show details- Abstract: Feature selection is a pervasive problem. The discovery of relevant features can be as important for performing a particular task (such as to avoid overfitting in prediction) as it can be for understanding the underlying processes governing the true label (such as discovering relevant genetic factors for a disease). Machine learning driven feature selection can enable discovery from large, high-dimensional, non-linear observational datasets by creating a subset of features for experts to focus on. In order to use expert time most efficiently, we need a principled methodology capable of controlling the False Discovery Rate. In this work, we build on the promising Knockoff framework by developing a flexible knockoff generation model. We adapt the Generative Adversarial Networks framework to allow us to generate knockoffs with no assumptions on the feature distribution. Our model consists of 4 networks, a generator, a discriminator, a stability network and a power network. We demonstrate the capability of our model to perform feature selection, showing that it performs as well as the originally proposed knockoff generation model in the Gaussian setting and that it outperforms the original model in non-Gaussian settings, including on a real-world dataset.
- Keywords: Knockoff model, Feature selection, False discovery rate control, Generative Adversarial networks
-
Learning Protein Structure with a Differentiable Simulator
Show details- Abstract: The Boltzmann distribution is a natural model for many systems, from brains to materials and biomolecules, but is often of limited utility for fitting data because Monte Carlo algorithms are unable to simulate it in available time. This gap between the expressive capabilities and sampling practicalities of energy-based models is exemplified by the protein folding problem, since energy landscapes underlie contemporary knowledge of protein biophysics but computer simulations are challenged to fold all but the smallest proteins from first principles. In this work we aim to bridge the gap between the expressive capacity of energy functions and the practical capabilities of their simulators by using an unrolled Monte Carlo simulation as a model for data. We compose a neural energy function with a novel and efficient simulator based on Langevin dynamics to build an end-to-end-differentiable model of atomic protein structure given amino acid sequence information. We introduce techniques for stabilizing backpropagation under long roll-outs and demonstrate the model's capacity to make multimodal predictions and to, in some cases, generalize to unobserved protein fold types when trained on a large corpus of protein structures.
- Keywords: generative models, simulators, molecular modeling, proteins, structured prediction
- TL;DR: We use an unrolled simulator as an end-to-end differentiable model of protein structure and show it can (sometimes) hierarchically generalize to unseen fold topologies.
-
ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness
Show details- Abstract: Convolutional Neural Networks (CNNs) are commonly thought to recognise objects by learning increasingly complex representations of object shapes. Some recent studies suggest a more important role of image textures. We here put these conflicting hypotheses to a quantitative test by evaluating CNNs and human observers on images with a texture-shape cue conflict. We show that ImageNet-trained CNNs are strongly biased towards recognising textures rather than shapes, which is in stark contrast to human behavioural evidence and reveals fundamentally different classification strategies. We then demonstrate that the same standard architecture (ResNet-50) that learns a texture-based representation on ImageNet is able to learn a shape-based representation instead when trained on 'Stylized-ImageNet', a stylized version of ImageNet. This provides a much better fit for human behavioural performance in our well-controlled psychophysical lab setting (nine experiments totalling 48,560 psychophysical trials across 97 observers) and comes with a number of unexpected emergent benefits such as improved object detection performance and previously unseen robustness towards a wide range of image distortions, highlighting advantages of a shape-based representation.
- Keywords: deep learning, psychophysics, representation learning, object recognition, robustness, neural networks, data augmentation
- TL;DR: ImageNet-trained CNNs are biased towards object texture (instead of shape like humans). Overcoming this major difference between human and machine vision yields improved detection performance and previously unseen robustness to image distortions.
-
Smoothing the Geometry of Probabilistic Box Embeddings
Show details- Abstract: There is growing interest in geometrically-inspired embeddings for learning hierarchies, partial orders, and lattice structures, with natural applications to transitive relational data such as entailment graphs. Recent work has extended these ideas beyond deterministic hierarchies to probabilistically calibrated models, which enable learning from uncertain supervision and inferring soft-inclusions among concepts, while maintaining the geometric inductive bias of hierarchical embedding models. We build on the Box Lattice model of Vilnis et al. (2018), which showed promising results in modeling soft-inclusions through an overlapping hierarchy of sets, parameterized as high-dimensional hyperrectangles (boxes). However, the hard edges of the boxes present difficulties for standard gradient based optimization; that work employed a special surrogate function for the disjoint case, but we find this method to be fragile. In this work, we present a novel hierarchical embedding model, inspired by a relaxation of box embeddings into parameterized density functions using Gaussian convolutions over the boxes. Our approach provides an alternative surrogate to the original lattice measure that improves the robustness of optimization in the disjoint case, while also preserving the desirable properties with respect to the original lattice. We demonstrate increased or matching performance on WordNet hypernymy prediction, Flickr caption entailment, and a MovieLens-based market basket dataset. We show especially marked improvements in the case of sparse data, where many conditional probabilities should be low, and thus boxes should be nearly disjoint.
- Keywords: embeddings, order embeddings, knowledge graph embedding, relational learning
- TL;DR: Improve hierarchical embedding models using kernel smoothing
-
On Random Deep Weight-Tied Autoencoders: Exact Asymptotic Analysis, Phase Transitions, and Implications to Training
Show details- Abstract: We study the behavior of weight-tied multilayer vanilla autoencoders under the assumption of random weights. Via an exact characterization in the limit of large dimensions, our analysis reveals interesting phase transition phenomena when the depth becomes large. This, in particular, provides quantitative answers and insights to three questions that were yet fully understood in the literature. Firstly, we provide a precise answer on how the random deep weight-tied autoencoder model performs “approximate inference” as posed by Scellier et al. (2018), and its connection to reversibility considered by several theoretical studies. Secondly, we show that deep autoencoders display a higher degree of sensitivity to perturbations in the parameters, distinct from the shallow counterparts. Thirdly, we obtain insights on pitfalls in training initialization practice, and demonstrate experimentally that it is possible to train a deep autoencoder, even with the tanh activation and a depth as large as 200 layers, without resorting to techniques such as layer-wise pre-training or batch normalization. Our analysis is not specific to any depths or any Lipschitz activations, and our analytical techniques may have broader applicability.
- Keywords: Random Deep Autoencoders, Exact Asymptotic Analysis, Phase Transitions
- TL;DR: We study the behavior of weight-tied multilayer vanilla autoencoders under the assumption of random weights. Via an exact characterization in the limit of large dimensions, our analysis reveals interesting phase transition phenomena.
-
Meta-Learning Update Rules for Unsupervised Representation Learning
Show details- Abstract: A major goal of unsupervised learning is to discover data representations that are useful for subsequent tasks, without access to supervised labels during training. Typically, this involves minimizing a surrogate objective, such as the negative log likelihood of a generative model, with the hope that representations useful for subsequent tasks will arise as a side effect. In this work, we propose instead to directly target later desired tasks by meta-learning an unsupervised learning rule which leads to representations useful for those tasks. Specifically, we target semi-supervised classification performance, and we meta-learn an algorithm -- an unsupervised weight update rule -- that produces representations useful for this task. Additionally, we constrain our unsupervised update rule to a be a biologically-motivated, neuron-local function, which enables it to generalize to different neural network architectures, datasets, and data modalities. We show that the meta-learned update rule produces useful features and sometimes outperforms existing unsupervised learning techniques. We further show that the meta-learned unsupervised update rule generalizes to train networks with different widths, depths, and nonlinearities. It also generalizes to train on data with randomly permuted input dimensions and even generalizes from image datasets to a text task.
- Keywords: Meta-learning, unsupervised learning, representation learning
- TL;DR: We learn an unsupervised learning algorithm that produces useful representations from a set of supervised tasks. At test-time, we apply this algorithm to new tasks without any supervision and show performance comparable to a VAE.
-
Transferring Knowledge across Learning Processes
Show details- Abstract: In complex transfer learning scenarios new tasks might not be tightly linked to previous tasks. Approaches that transfer information contained only in the final parameters of a source model will therefore struggle. Instead, transfer learning at at higher level of abstraction is needed. We propose Leap, a framework that achieves this by transferring knowledge across learning processes. We associate each task with a manifold on which the training process travels from initialization to final parameters and construct a meta-learning objective that minimizes the expected length of this path. Our framework leverages only information obtained during training and can be computed on the fly at negligible cost. We demonstrate that our framework outperforms competing methods, both in meta-learning and transfer learning, on a set of computer vision tasks. Finally, we demonstrate that Leap can transfer knowledge across learning processes in demanding reinforcement learning environments (Atari) that involve millions of gradient steps.
- Keywords: meta-learning, transfer learning
- TL;DR: We propose Leap, a framework that transfers knowledge across learning processes by minimizing the expected distance the training process travels on a task's loss surface.
-
GENERATING HIGH FIDELITY IMAGES WITH SUBSCALE PIXEL NETWORKS AND MULTIDIMENSIONAL UPSCALING
Show details- Abstract: The unconditional generation of high fidelity images is a longstanding benchmark for testing the performance of image decoders. Autoregressive image models have been able to generate small images unconditionally, but the extension of these methods to large images where fidelity can be more readily assessed has remained an open problem. Among the major challenges are the capacity to encode the vast previous context and the sheer difficulty of learning a distribution that preserves both global semantic coherence and exactness of detail. To address the former challenge, we propose the Subscale Pixel Network (SPN), a conditional decoder architecture that generates an image as a sequence of image slices of equal size. The SPN compactly captures image-wide spatial dependencies and requires a fraction of the memory and the computation. To address the latter challenge, we propose to use multidimensional upscaling to grow an image in both size and depth via intermediate stages corresponding to distinct SPNs. We evaluate SPNs on the unconditional generation of CelebAHQ of size 256 and of ImageNet from size 32 to 128. We achieve state-of-the-art likelihood results in multiple settings, set up new benchmark results in previously unexplored settings and are able to generate very high fidelity large scale samples on the basis of both datasets.
- TL;DR: We show that autoregressive models can generate high fidelity images.
-
Temporal Difference Variational Auto-Encoder
Show details- Abstract: To act and plan in complex environments, we posit that agents should have a mental simulator of the world with three characteristics: (a) it should build an abstract state representing the condition of the world; (b) it should form a belief which represents uncertainty on the world; (c) it should go beyond simple step-by-step simulation, and exhibit temporal abstraction. Motivated by the absence of a model satisfying all these requirements, we propose TD-VAE, a generative sequence model that learns representations containing explicit beliefs about states several steps into the future, and that can be rolled out directly without single-step transitions. TD-VAE is trained on pairs of temporally separated time points, using an analogue of temporal difference learning used in reinforcement learning.
- Keywords: generative models, variational auto-encoders, state space models, temporal difference learning
- TL;DR: Generative model of temporal data, that builds online belief state, operates in latent space, does jumpy predictions and rollouts of states.
-
A Unified Theory of Early Visual Representations from Retina to Cortex through Anatomically Constrained Deep CNNs
Show details- Abstract: The vertebrate visual system is hierarchically organized to process visual information in successive stages. Neural representations vary drastically across the first stages of visual processing: at the output of the retina, ganglion cell receptive fields (RFs) exhibit a clear antagonistic center-surround structure, whereas in the primary visual cortex (V1), typical RFs are sharply tuned to a precise orientation. There is currently no unified theory explaining these differences in representations across layers. Here, using a deep convolutional neural network trained on image recognition as a model of the visual system, we show that such differences in representation can emerge as a direct consequence of different neural resource constraints on the retinal and cortical networks, and for the first time we find a single model from which both geometries spontaneously emerge at the appropriate stages of visual processing. The key constraint is a reduced number of neurons at the retinal output, consistent with the anatomy of the optic nerve as a stringent bottleneck. Second, we find that, for simple downstream cortical networks, visual representations at the retinal output emerge as nonlinear and lossy feature detectors, whereas they emerge as linear and faithful encoders of the visual scene for more complex cortical networks. This result predicts that the retinas of small vertebrates (e.g. salamander, frog) should perform sophisticated nonlinear computations, extracting features directly relevant to behavior, whereas retinas of large animals such as primates should mostly encode the visual scene linearly and respond to a much broader range of stimuli. These predictions could reconcile the two seemingly incompatible views of the retina as either performing feature extraction or efficient coding of natural scenes, by suggesting that all vertebrates lie on a spectrum between these two objectives, depending on the degree of neural resources allocated to their visual system.
- Keywords: visual system, convolutional neural networks, efficient coding, retina
- TL;DR: We reproduced neural representations found in biological visual systems by simulating their neural resource constraints in a deep convolutional model.
-
Pay Less Attention with Lightweight and Dynamic Convolutions
Show details- Abstract: Self-attention is a useful mechanism to build generative models for language and images. It determines the importance of context elements by comparing each element to the current time step. In this paper, we show that a very lightweight convolution can perform competitively to the best reported self-attention results. Next, we introduce dynamic convolutions which are simpler and more efficient than self-attention. We predict separate convolution kernels based solely on the current time-step in order to determine the importance of context elements. The number of operations required by this approach scales linearly in the input length, whereas self-attention is quadratic. Experiments on large-scale machine translation, language modeling and abstractive summarization show that dynamic convolutions improve over strong self-attention models. On the WMT'14 English-German test set dynamic convolutions achieve a new state of the art of 29.7 BLEU.
- Keywords: Deep learning, sequence to sequence learning, convolutional neural networks, generative models
- TL;DR: Dynamic lightweight convolutions are competitive to self-attention on language tasks.
-
Enabling Factorized Piano Music Modeling and Generation with the MAESTRO Dataset
Show details- Abstract: Generating musical audio directly with neural networks is notoriously difficult because it requires coherently modeling structure at many different timescales. Fortunately, most music is also highly structured and can be represented as discrete note events played on musical instruments. Herein, we show that by using notes as an intermediate representation, we can train a suite of models capable of transcribing, composing, and synthesizing audio waveforms with coherent musical structure on timescales spanning six orders of magnitude (~0.1 ms to ~100 s), a process we call Wave2Midi2Wave. This large advance in the state of the art is enabled by our release of the new MAESTRO (MIDI and Audio Edited for Synchronous TRacks and Organization) dataset, composed of over 172 hours of virtuosic piano performances captured with fine alignment (~3 ms) between note labels and audio waveforms. The networks and the dataset together present a promising approach toward creating new expressive and interpretable neural models of music.
- Keywords: music, piano transcription, transformer, wavnet, audio synthesis, dataset, midi
- TL;DR: We train a suite of models capable of transcribing, composing, and synthesizing audio waveforms with coherent musical structure, enabled by the new MAESTRO dataset.
-
Learning to Remember More with Less Memorization
Show details- Abstract: Memory-augmented neural networks consisting of a neural controller and an external memory have shown potentials in long-term sequential learning. Current RAM-like memory models maintain memory accessing every timesteps, thus they do not effectively leverage the short-term memory held in the controller. We hypothesize that this scheme of writing is suboptimal in memory utilization and introduces redundant computation. To validate our hypothesis, we derive a theoretical bound on the amount of information stored in a RAM-like system and formulate an optimization problem that maximizes the bound. The proposed solution dubbed Uniform Writing is proved to be optimal under the assumption of equal timestep contributions. To relax this assumption, we introduce modifications to the original solution, resulting in a solution termed Cached Uniform Writing. This method aims to balance between maximizing memorization and forgetting via overwriting mechanisms. Through an extensive set of experiments, we empirically demonstrate the advantages of our solutions over other recurrent architectures, claiming the state-of-the-arts in various sequential modeling tasks.
- Keywords: memory-augmented neural networks, writing optimization
-
Learning Robust Representations by Projecting Superficial Statistics Out
Show details- Abstract: Despite impressive performance as evaluated on i.i.d. holdout data, deep neural networks depend heavily on superficial statistics of the training data and are liable to break under distribution shift. For example, subtle changes to the background or texture of an image can break a seemingly powerful classifier. Building on previous work on domain generalization, we hope to produce a classifier that will generalize to previously unseen domains, even when domain identifiers are not available during training. This setting is challenging because the model may extract many distribution-specific (superficial) signals together with distribution-agnostic (semantic) signals. To overcome this challenge, we incorporate the gray-level co-occurrence matrix (GLCM) to extract patterns that our prior knowledge suggests are superficial: they are sensitive to the texture but unable to capture the gestalt of an image. Then we introduce two techniques for improving our networks' out-of-sample performance. The first method is built on the reverse gradient method that pushes our model to learn representations from which the GLCM representation is not predictable. The second method is built on the independence introduced by projecting the model's representation onto the subspace orthogonal to GLCM representation's. We test our method on the battery of standard domain generalization data sets and, interestingly, achieve comparable or better performance as compared to other domain generalization methods that explicitly require samples from the target distribution for training.
- Keywords: domain generalization, robustness
- TL;DR: Building on previous work on domain generalization, we hope to produce a classifier that will generalize to previously unseen domains, even when domain identifiers are not available during training.
-
Slalom: Fast, Verifiable and Private Execution of Neural Networks in Trusted Hardware
Show details- Abstract: As Machine Learning (ML) gets applied to security-critical or sensitive domains, there is a growing need for integrity and privacy for outsourced ML computations. A pragmatic solution comes from Trusted Execution Environments (TEEs), which use hardware and software protections to isolate sensitive computations from the untrusted software stack. However, these isolation guarantees come at a price in performance, compared to untrusted alternatives. This paper initiates the study of high performance execution of Deep Neural Networks (DNNs) in TEEs by efficiently partitioning DNN computations between trusted and untrusted devices. Building upon an efficient outsourcing scheme for matrix multiplication, we propose Slalom, a framework that securely delegates execution of all linear layers in a DNN from a TEE (e.g., Intel SGX or Sanctum) to a faster, yet untrusted, co-located processor. We evaluate Slalom by running DNNs in an Intel SGX enclave, which selectively delegates work to an untrusted GPU. For canonical DNNs (VGG16, MobileNet and ResNet variants) we obtain 6x to 20x increases in throughput for verifiable inference, and 4x to 11x for verifiable and private inference.
- Keywords: Trusted hardware, integrity, privacy, secure inference, SGX
- TL;DR: We accelerate secure DNN inference in trusted execution environments (by a factor 4x-20x) by selectively outsourcing the computation of linear layers to a faster yet untrusted co-processor.
-
The Neuro-Symbolic Concept Learner: Interpreting Scenes, Words, and Sentences From Natural Supervision
Show details- Abstract: We propose the Neuro-Symbolic Concept Learner (NS-CL), a model that learns visual concepts, words, and semantic parsing of sentences without explicit supervision on any of them; instead, our model learns by simply looking at images and reading paired questions and answers. Our model builds an object-based scene representation and translates sentences into executable, symbolic programs. To bridge the learning of two modules, we use a neuro-symbolic reasoning module that executes these programs on the latent scene representation. Analogical to human concept learning, the perception module learns visual concepts based on the language description of the object being referred to. Meanwhile, the learned visual concepts facilitate learning new words and parsing new sentences. We use curriculum learning to guide the searching over the large compositional space of images and language. Extensive experiments demonstrate the accuracy and efficiency of our model on learning visual concepts, word representations, and semantic parsing of sentences. Further, our method allows easy generalization to new object attributes, compositions, language concepts, scenes and questions, and even new program domains. It also empowers applications including visual question answering and bidirectional image-text retrieval.
- Keywords: Neuro-Symbolic Representations, Concept Learning, Visual Reasoning
- TL;DR: We propose the Neuro-Symbolic Concept Learner (NS-CL), a model that learns visual concepts, words, and semantic parsing of sentences without explicit supervision on any of them.
-
The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks
Show details- Abstract: Neural network pruning techniques can reduce the parameter counts of trained networks by over 90%, decreasing storage requirements and improving computational performance of inference without compromising accuracy. However, contemporary experience is that the sparse architectures produced by pruning are difficult to train from the start, which would similarly improve training performance. We find that a standard pruning technique naturally uncovers subnetworks whose initializations made them capable of training effectively. Based on these results, we articulate the "lottery ticket hypothesis:" dense, randomly-initialized, feed-forward networks contain subnetworks ("winning tickets") that - when trained in isolation - reach test accuracy comparable to the original network in a similar number of iterations. The winning tickets we find have won the initialization lottery: their connections have initial weights that make training particularly effective. We present an algorithm to identify winning tickets and a series of experiments that support the lottery ticket hypothesis and the importance of these fortuitous initializations. We consistently find winning tickets that are less than 10-20% of the size of several fully-connected and convolutional feed-forward architectures for MNIST and CIFAR10. Above this size, the winning tickets that we find learn faster than the original network and reach higher test accuracy.
- Keywords: Neural networks, sparsity, pruning, compression, performance, architecture search
- TL;DR: Feedforward neural networks that can have weights pruned after training could have had the same weights pruned before training
-
FFJORD: Free-Form Continuous Dynamics for Scalable Reversible Generative Models
Show details- Abstract: A promising class of generative models maps points from a simple distribution to a complex distribution through an invertible neural network. Likelihood-based training of these models requires restricting their architectures to allow cheap computation of Jacobian determinants. Alternatively, the Jacobian trace can be used if the transformation is specified by an ordinary differential equation. In this paper, we use Hutchinson’s trace estimator to give a scalable unbiased estimate of the log-density. The result is a continuous-time invertible generative model with unbiased density estimation and one-pass sampling, while allowing unrestricted neural network architectures. We demonstrate our approach on high-dimensional density estimation, image generation, and variational inference, achieving the state-of-the-art among exact likelihood methods with efficient sampling.
- Keywords: generative models, density estimation, approximate inference, ordinary differential equations
- TL;DR: We use continuous time dynamics to define a generative model with exact likelihoods and efficient sampling that is parameterized by unrestricted neural networks.
-
How Powerful are Graph Neural Networks?
Show details- Abstract: Graph Neural Networks (GNNs) are an effective framework for representation learning of graphs. GNNs follow a neighborhood aggregation scheme, where the representation vector of a node is computed by recursively aggregating and transforming representation vectors of its neighboring nodes. Many GNN variants have been proposed and have achieved state-of-the-art results on both node and graph classification tasks. However, despite GNNs revolutionizing graph representation learning, there is limited understanding of their representational properties and limitations. Here, we present a theoretical framework for analyzing the expressive power of GNNs to capture different graph structures. Our results characterize the discriminative power of popular GNN variants, such as Graph Convolutional Networks and GraphSAGE, and show that they cannot learn to distinguish certain simple graph structures. We then develop a simple architecture that is provably the most expressive among the class of GNNs and is as powerful as the Weisfeiler-Lehman graph isomorphism test. We empirically validate our theoretical findings on a number of graph classification benchmarks, and demonstrate that our model achieves state-of-the-art performance.
- Keywords: graph neural networks, theory, deep learning, representational power, graph isomorphism, deep multisets
- TL;DR: We develop theoretical foundations for the expressive power of GNNs and design a provably most powerful GNN.


Website "https://bavarikon.de/"
Show language dependent values in
01.01.2000
Alert message.
| # | Label | Status | Start of publication | End of publication | In page navigation | last modified | created | Actions |
|---|


Started by the Swiss corporation New7Wonders Foundation to choose the New7Wonders of the World, the finalists were announced at the start of 2006, from a selection of 200 existing monuments.
Below are five of the seven wonders, as announced in July, 2007.
Sources: Wikipedia and Google Street View.


We began the rational-number implementation in section \ref{sec:rationals} by implementing the rational-number operations add_rat, sub_rat, and so on in terms of three unspecified functions: make_rat, numer, and denom. At that point, we could think of the operations as being defined in terms of data objects—numerators, denominators, and rational numbers—whose behavior was specified by the latter three functions.
But exactly what is meant by data? It is not enough to say
whatever is implemented by the given selectors and constructors.
Clearly, not every arbitrary set of three
functions
can serve as an
appropriate basis for the rational-number implementation. We need to
guarantee that,
if we construct a rational number x from a pair
of integers n and d, then extracting the numer and the
denom of x and dividing them should yield the same result
as dividing n by d.
In other words, make_rat,
numer, and denom must satisfy the condition that, for any
integer n and any non-zero integer d, if x is
make_rat(n,d), then
In fact, this is the only condition make_rat, numer, and denom must fulfill in order to form a suitable basis for a rational-number representation. In general, we can think of data as defined by some collection of selectors and constructors, together with specified conditions that these functions must fulfill in order to be a valid representation.[1]
This point of view can serve to define not only high-level
data
objects, such as rational numbers, but lower-level objects as well.
Consider the notion of a pair, which we used in order to define our
rational numbers. We never actually said what a pair was, only that
the language supplied
functions
pair, head, and tail
for operating on pairs. But the only thing we need to know about
these three operations
is that if we glue two objects together using
pair we can retrieve the objects using head and tail.
That is, the operations satisfy the condition that, for any objects
x and y, if z is pair(x, y) then head(z)
is x and tail(x) is y. Indeed, we mentioned that
these three
functions
are included as primitives in our language.
However, any triple of
functions
that satisfies the above condition
can be used as the basis for implementing pairs. This point is
illustrated strikingly by the fact that we could implement pair,
head, and tail without using any data structures at all but
only using
functions. Here are the definitions:
m === 0 ? x : m === 1 ? y \end{JavaScriptClickable} \end{lrbox} \begin{JavaScriptClickable} /*!\href{https://sourceacademy.nus.edu.sg/playground#chap=2&prgrm=GYVwdgxgLglg9mABABwIYwE4AoAeAaRATwEpEBvAKEWsQwFMoQMkBbRAXgD4qbea32gxAAZEPPrwD8iHOIm8AXIgFCAjGICQ86YTnzFiOhgxxsLAgCIAghgDmIFnTBREYOC9GnE6mEjSYLYgBuCgBfClBIWAREAAs6VAATLAAvUkpeekZmRBSsYWCwiPBoeCQodAAbVPS5LKYkPNVC8IoIBABnFxwOFHRsVTwAJkL4pNxgoA}{\usebox\lstbox}!*/ : error(m, "Argument not 0 or 1 in pair"); } function head(z) { return z(0); } function tail(z) { return z(1); } \end{JavaScriptClickable}
This use of functions corresponds to nothing like our intuitive notion of what data should be. Nevertheless, all we need to do to show that this is a valid way to represent pairs is to verify that these functions satisfy the condition given above.
The subtle point to notice is that the value returned by pair(x, y) is a
function—namely the internally defined
function
dispatch, which takes one argument and returns either x or y depending on whether the argument is 0 or 1. Correspondingly, head(z) is defined to apply z to 0. Hence, if z is the
function
formed by pair(x, y), then z applied to 0 will
yield x. Thus, we have shown that head(pair(x, y)) yields
x, as desired. Similarly, tail(pair(x, y)) applies the
function
returned by pair(x, y) to 1, which returns y.
Therefore, this
functional
implementation of pairs is a valid
implementation, and if we access pairs using only pair, head, and tail we cannot distinguish this implementation from one
that uses real
data structures.
The point of exhibiting the functional representation of pairs is not that our language works this way (we will be using arrays to represent pairs) but that it could work this way. The functional representation, although obscure, is a perfectly adequate way to represent pairs, since it fulfills the only conditions that pairs need to fulfill. This example also demonstrates that the ability to manipulate functions as objects automatically provides the ability to represent compound data. This may seem a curiosity now, but functional representations of data will play a central role in our programming repertoire. This style of programming is often called message passing, and we will be using it as a basic tool in chapter 3 when we address the issues of modeling and simulation.
m(x, y); } function head(z) { return z((p, q) => p); } \end{JavaScriptClickable} \end{lrbox} \href{https://sourceacademy.nus.edu.sg/playground#chap=2&prgrm=GYVwdgxgLglg9mABABwIYwE4AoAeAaRATwEpEBvAKEWsQwFMoQMkBbRAXgD5EXcCSA3BQC+FUJFgJEACzqoAJlgBepSjVoMmSJVizICAR1JcUxIaIoQEAZyiIcHFOmwBGPACYzFWQtxmgA}{\usebox\lstbox}
q); } \end{JavaScriptClickable} \end{lrbox} \href{https://sourceacademy.nus.edu.sg/playground#chap=2&prgrm=GYVwdgxgLglg9mABABwIYwE4AoAeAaRATwEpEBvAKEWsQwFMoQMkBbRAXgD5EXcCSA3BQC+FUJFgJEACzqoAJlgBepSjVoMmSJVizICAR1JcUxIaPHR4SKOgA2y1VRr1GzRDr14jHbkfMUFBAIAM5QiDgcKOjYAIx4AExmFLYwDjhmQA}{\usebox\lstbox}
x => x; function add_1(n) { return f => x => f(n(f)(x)); } \end{JavaScriptClickable} \end{lrbox} \href{https://sourceacademy.nus.edu.sg/playground#chap=2&prgrm=MYewdgzgLgBAXgUwE4hgXhgM3QPhgD1wIG4AoTAVzGCgEtwYBDAE2YH0BGACjAEoYA3qRgiYSBFApIwWIoTR5MPLpl5d8vXmQC+QA}{\usebox\lstbox}
x => f(x); const two = f => x => f(f(x)); function plus(n, m) { return f => x => n(f)(m(f)(x)); } // testing \end{JavaScriptClickable} \end{lrbox} \begin{JavaScriptClickable} /*!\href{https://sourceacademy.nus.edu.sg/playground#chap=2&prgrm=MYewdgzgLgBOCmMC8MBmyB8MAem0ApsBKAbgChRJYoB3EZNPXJLVfN40ss1AVzGBQAluBgAHADa8I+MABoYAWyIwA3mRiaYAJ3hRe2sIxY48YdkXyKLhIlwC+3APROYUeNCFgA5t0rQ3AAtdRBRJaXwEBVoQLh5+QREjYECDFIB9KBB0sF5FACN4bXxgFXUtHT0DZNkzGABqGABGSwAGBwpU7QysnLzC4qhg+HhSIA}{\usebox\lstbox}!*/ const three = plus(one, two); function church_to_number(c) { return c(n => n + 1)(0); } church_to_number(three); \end{JavaScriptClickable}
functions plus conditionsspecification as outlined in the rational-number example above. Note that the condition on the rational-number representation was stated in terms of facts about integers (equality and division). In general, abstract models define new kinds of data objects in terms of previously defined types of data objects. Assertions about data objects can therefore be checked by reducing them to assertions about previously defined data objects. Another approach, introduced by Zilles at MIT, by Goguen, Thatcher, Wagner, and Wright at IBM (see
functionsas elements of an abstract algebraic system whose behavior is specified by axioms that correspond to our
conditions,and uses the techniques of abstract algebra to check assertions about data objects. Both methods are surveyed in the paper by Liskov and Zilles (1975).


picture_as_pdf Computer Origanization
picture_as_pdf Discrete Mathematics
picture_as_pdf English
picture_as_pdf Hindi
picture_as_pdf Kannada
picture_as_pdf Problem Solving Techniques Using C
picture_as_pdf Sanskrit
picture_as_pdf Digital Electronics
picture_as_pdf Environment and Public Health
picture_as_pdf Data Structures
picture_as_pdf Database Management Systems
picture_as_pdf Numerical and Statistical Methods
picture_as_pdf Indian Constitution and Human Rights
picture_as_pdf Operating System
picture_as_pdf Object Oriented Programming Using C++
picture_as_pdf Financial Accounting and Management
picture_as_pdf Operations Research
picture_as_pdf Unix Shell Programming
picture_as_pdf Visual Programming
picture_as_pdf Personality Development
picture_as_pdf Computer Architecture
picture_as_pdf Data Communication and Networks
picture_as_pdf Java Programming
picture_as_pdf Microprocessor and Assembly Language
picture_as_pdf Software Engineerning
picture_as_pdf Banking and Finance
picture_as_pdf Software Engineering
picture_as_pdf Cryptography and Network Security
picture_as_pdf System Programming
picture_as_pdf Web Programming
picture_as_pdf Computer Applications and Information Technology
picture_as_pdf Theory of Computation


Intelligence 96% Complete
Attractiveness 95% Complete
Strength 98% Complete
Overall Awesomeness 100% Complete
Happy Birthday Squire! Hope all has been well my friend. Listened to "Fork" by 2 Chainz yesterday, wasn't the same lol
Happy Birthday man! Hope your having a great time celebrating! in honor of your birthday, I stold a squat rack from crossfit guy at the gym today while he as off doing other excersises (pull ups, jerking off, etc.)
Happy Birthday brother. Hope all is well for you out on the West Coast. We have to get together sometime soon and catch back up. Have a fantastic day man.
Good Morning. Happy Birthday!! I hope you have a great day!! Any special plans for your birthday? I think we have had better weather here lol
Happy Birthday to Jeremy.. wishing you plenty of good times ahead. Lots of love.
Wow. Jeremy, it was a great oppurtunity meeting someone like you during the US visit. I know you have a powerful future ahead, may you enjoy the day and live to blow 1,000 candles. Much love bro. Happy Birthday from a good old friend.
On a Saturday morning at 6:00am, Jeremy and Alexis got married in the rain. They were on top of Twin Peaks with only their parents and their siblings. Everyone was wet and there they were surronded by so many clouds you couldn't even see the bay or ocean, but it was absolutely perfect. Jeremy and Alexis wouldn't have it any other way.
Jeremy finally starts his new job, flying off to Dallas for training. He couldn't be more thrilled and was ecstatic to start this wonderful new career that he worked so hard to get.
Less than a week it took them to find the perfect apartment. They searched and searched and after many, many craigslist emails they found a young couple who were sent from God. The apartment had two bedrooms, hardwood floors, and all the love they could imagine.
After months and months of planning and deciding exactly how they would move to San Francisco they finally packed all of their belongings into a tiny cube and flew off to a knew destination where they knew noone and would have to start a life of their own.
After deciding to move out to San Francisco, California with Alexis, he decided that he would want to spend the rest of her life with her. Both Jeremy and Alexis knew that they would want to be with each other for the rest of their lives and nothing would change that so they both decided to get engaged.
Congratulations to Jeremy Ryan Fahling to graduating with a double major in management and accounting from the College of Business and Public Administration at Drake University in Des Moines, IA!
This is when Jeremy decided to accept his job at Ernst ;amp; Young as a Business Consultant in the Finance group. He also chose to move to San Francisco, California after a lot of thinking. San Francisco beat out Chicago, Texas, Florida, and New York as well as many other amazing cities in the United States.


Anthony's Portfolio
This is my portfolio.
I am a fullstack engineer studying at the University at Buffalo (SUNY). I have developed several websites, created Python applications, and created an Android App in my time as a student!
Connect with me on LinkedIn
Click here to see content created by server code.
Click here to get a random message:


function GetSample(X, Y: TFloat): TColor32; override;
GetSample provides a method for acquiring a color sample from the coordinate (X, Y). The sample may be generated by the sampler itself, or it may be acquired from a nested sampler. Descendants of TCustomSampler must always override at least one of GetSampleFixed or GetSampleFloat.
The different postfixes determines the precision of the input parameters (integer, fixed-point or floating-point).