Geek

CONTENTSMany of the research frontiers in deep learning involve building a probabilistic model of the input, pmodel (x)p_{\text {model }}(\boldsymbol{x})pmodel ​(x). Such a model can, in principle, use probabilistic inference to predict any of

CONTENTSOther ApplicationsIn this section we cover a few other types of applications of deep learning that are different from the standard object recognition, speech recognition and natural language processing tasks discussed above. Part III of this boo

CONTENTSNatural Language ProcessingNatural language processing (NLP) is the use of human languages, such as English or French, by a computer. Computer programs typically read and emit specialized languages designed to allow efficient and unambiguous pa

CONTENTSComputer VisionComputer vision has traditionally been one of the most active research areas for deep learning applications, because vision is a task that is effortless for humans and many animals but challenging for computers ( Ballard et al. ,

CONTENTSLarge-Scale Deep LearningDeep learning is based on the philosophy of connectionism: while an individual biological neuron or an individual feature in a machine learning model is not intelligent, a large population of these neurons or features ac

CONTENTSDebugging StrategiesWhen a machine learning system performs poorly, it is usually difficult to tell whether the poor performance is intrinsic to the algorithm itself or whether there is a bug in the implementation of the algorithm. Machine lear

CONTENTSSelecting HyperparametersMost deep learning algorithms come with many hyperparameters that control many aspects of the algorithm’s behavior. Some of these hyperparameters affect the time and memory cost of running the algorithm. Some of these h

CONTENTSSuccessfully applying deep learning techniques requires more than just a good knowledge of what algorithms exist and the principles that explain how they work. A good machine learning practitioner also needs to know how to choose an algorithm for

CONTENTSLeaky Units and Other Strategies for Multiple Time ScalesOne way to deal with long-term dependencies is to design a model that operates at multiple time scales, so that some parts of the model operate at fine-grained time scales and can handle s

CONTENTSRecurrent neural networks or RNNs (Rumelhart et al., 1986a) are a family of neural networks for processing sequential data. Much as a convolutional network is a neural network that is specialized for processing a grid of values XXX such as an im

CONTENTSRandom or Unsupervised FeaturesTypically, the most expensive part of convolutional network training is learning the features. The output layer is usually relatively inexpensive due to the small number of features provided as input to this layer

CONTENTSVariants of the Basic Convolution FunctionWhen discussing convolution in the context of neural networks, we usually do not refer exactly to the standard discrete convolution operation as it is usually understood in the mathematical literature.

CONTENTSConvolutional networks ( LeCun , 1989 ), also known as convolutional neural networks or CNNs, are a specialized kind of neural network for processing data that has a known, grid-like topology. Examples include time-series data, which can be thoug

CONTENTSApproximate Second-Order MethodsFor simplicity of exposition, the only objective function we examine is the empirical risk:J(θ)=Ex,y∼p^data (x,y)[L(f(x;θ),y)]=1m∑i=1mL(f(x(i);θ),y(i))J(\boldsymbol{\theta})=\mathbb{E}_{\mathbf{x}, \mathrm{

CONTENTSBasic AlgorithmsWe have previously introduced the gradient descent (section 4.3) algorithm that follows the gradient of an entire training set downhill. This may be accelerated considerably by using stochastic gradient descent to follow the gradi

CONTENTSHow Learning Differs from Pure OptimizationOptimization algorithms used for training of deep models differ from traditional optimization algorithms in several ways.Machine learning usually acts indirectly. In most machine learning scenarios,

CONTENTSParameter Tying and Parameter SharingThus far, in this chapter, when we have discussed adding constraints or penalties to the parameters, we have always done so with respect to a fixed region or point. For example, L2L^{2}L2 regularization (or

CONTENTSNoise RobustnessSection 7.4 has motivated the use of noise applied to the inputs as a dataset augmentation strategy. For some models, the addition of noise with infinitesimal variance at the input of the model is equivalent to imposing a penalt

CONTENTSA central problem in machine learning is how to make an algorithm that will perform well not just on the training data, but also on new inputs. Many strategies used in machine learning are explicitly designed to reduce the test error, possibly at

CONTENTSBack-Propagation and Other Differentiation AlgorithmsWhen we use a feedforward neural network to accept an input xxx and produce an output y^\hat{\boldsymbol{y}}y^​, information flows forward through the network. The inputs x\boldsymbol{x}x provi

CONTENTSHidden UnitsThe design of hidden units is an extremely active area of research and does not yet have many definitive guiding theoretical principles.Rectified linear units are an excellent default choice of hidden unit. Many other types of hi

CONTENTSDeep feedforward networks, also often called feedforward neural networks, or multilayer perceptrons (MLPs), are the quintessential deep learning models. The goal of a feedforward network is to approximate some function f∗f^{*}f∗. For example, fo

CONTENTSSupervised Learning AlgorithmsRoughly speaking, learning algorithms that learn to associate some input with some output, given a training set of examples of inputs xxx and outputs yyy. In many cases the outputs yyy may be difficult to collect au

CONTENTSCapacity, Overfitting and UnderfittingThe ability to perform well on previously unobserved inputs is called generalization.The field of statistical learning theory provides some answers. If the training and the test set are collected arbitra

CODE WORKSWork Here!CONTENTSLearning AlgorithmsA machine learning algorithm is an algorithm that is able to learn from data. But what do we mean by learning? Mitchell (1997) provides the definition "A computer program is said to learn from experience

Overflow and UnderflowThe fundamental difficulty in performing continuous math on a digital computer is that we need to represent infinitely many real numbers with a finite number of bit patterns.Rounding error（舍入误差） is problematic, especially when i

There are three possible sources of uncertainty：Inherent stochasticity in the system being modeledIncomplete observability（we cannot observe all of the variables that drive the behavior of the system）Incomplete modeling（ When we use a model that must d.

GOALS矩阵的对⻆化分解，以及⼀般矩阵的svd分解，以及应⽤pca算法推导逆矩阵以及伪逆举证，最⼩⼆乘估计，最⼩范数估计CODE WORKSWork Here!CONTENTSScalars: A scalar is just a single numbe.We write scalars in italics.We usually give scalars lower-case variable names.Vectors: A vector is an array of