Neural Networks and Deep Learning

FROM： http://neuralnetworksanddeeplearning.com/

Neural Networks and Deep Learning

Neural Networks and Deep Learning

What this book is about

On the exercises and problems

Using neural nets to recognize handwritten digits

• Perceptrons
• Sigmoid neurons
• The architecture of neural networks
• A simple network to classify handwritten digits
• Learning with gradient descent
• Implementing our network to classify digits
• Toward deep learning

How the backpropagation algorithm works

• Warm up: a fast matrix-based approach to computing the output from a neural network
• The two assumptions we need about the cost function
• The Hadamard product, $s \odot t$
• The four fundamental equations behind backpropagation
• Proof of the four fundamental equations (optional)
• The backpropagation algorithm
• The code for backpropagation
• In what sense is backpropagation a fast algorithm?
• Backpropagation: the big picture

Improving the way neural networks learn

• The cross-entropy cost function
• Overfitting and regularization
• Weight initialization
• Handwriting recognition revisited: the code
• How to choose a neural network's hyper-parameters?
• Other techniques

A visual proof that neural nets can compute any function

• Two caveats
• Universality with one input and one output
• Many input variables
• Extension beyond sigmoid neurons
• Fixing up the step functions
• Conclusion

Why are deep neural networks hard to train?

• The vanishing gradient problem
• What's causing the vanishing gradient problem? Unstable gradients in deep neural nets
• Unstable gradients in more complex networks
• Other obstacles to deep learning

Deep learning

• Convolutional neural networks
• Pretraining
• Recurrent neural networks, Boltzmann machines, and other models
• Is there a universal thinking algorithm?
• On the future of neural networks

Acknowledgements

Frequently Asked Questions

---

Sponsors

Thanks to all the supporters who made the book possible.Thanks also to all the contributors to the Bugfinder Hall of Fame.

The book is currently a beta release, and is stillunder active development. Please send error reports tomn@michaelnielsen.org. For other enquiries, please see the FAQ first.

---

Resources

Code repository

Mailing list for book announcements

Michael Nielsen's project announcement mailing list

---

By Michael Nielsen / Dec 2014

Neural Networks and Deep Learning is a free online book. Thebook will teach you about:

• Neural networks, a beautiful biologically-inspired programmingparadigm which enables a computer to learn from observational data
• Deep learning, a powerful set of techniques for learning in neuralnetworks

Neural networks and deep learning currently provide the best solutionsto many problems in image recognition, speech recognition, and naturallanguage processing. This book will teach you the core conceptsbehind neural networks and deep learning.

The book is currently an incomplete beta draft. More chapters will beadded over the coming months. For now, you can:

• Read Chapter 1, which explains howneural networks can learn to recognize handwriting
• Read Chapter 2, which explainsbackpropagation, the most important algorithm used to learn in neuralnetworks.
• Read Chapter 3, which explainsmany techniques which can be used to improve the performance ofbackpropagation.
• Read Chapter 4, which explains whyneural networks can compute any function.
• Learn more about the approach taken inthis book

In academic work,please cite this book as: Michael A. Nielsen, "Neural Networks andDeep Learning", Determination Press, 2015

This work is licensed under a Creative Commons Attribution-NonCommercial 3.0Unported License. This means you're free to copy, share, andbuild on this book, but not to sell it. If you're interested incommercial use, please contact me. Last update: Sun Dec 21 14:49:05 2014