https://docs.google.com/a/codeaudit.com/document/d/1ygFTDaRnVeXKkQqPTP8Oq92z2BtGLf7nkGE2njzb50k/edit?usp=sharing

Introduction

“Nature's imagination far surpasses our own.” - Richard Feynman

“Writing is perhaps the greatest of human inventions, binding together people who never knew each other, citizens of distant epochs. Books break the shackles of time. A book is proof that humans are capable of working magic.” - Carl Sagan

A solid theoretical foundation of why Deep Learning works is still a topic of much research. This leads to the question as to how we can create a Pattern Language without a solid foundation of the mechanisms that make it work? Can we begin with verified first principles and work our way towards a more solid understanding of the field?

Deep Learning is a rapid evolving approach to machine learning. What we have found is that there are several recurring patterns in practice and in the literature. By capturing these patterns and exploring their relationships, we may develop an improved understanding of the subject. Alexander's originated the idea of a pattern language that aided in the expression of the architectural concerns of buildings and municipalities. It is doubtful that he was able to create a fundamental theory on architecture. However, by exploring the various recurring patterns and their inter-relationships, Alexander was able to build a comprehensive compilation of of observed patterns. This book is a similar compilation that will allow readers to become exposed to the multitude of recurring patterns in Deep Learning. This will aid in improving the practitioners understanding of an otherwise extremely complex subject.

The approach of developing a pattern language to gain understanding is quite analogous to the core methodology of machine learning. Machine learning in general is concerned with algorithms that are able to learn a model based only on representative training data. Given sufficient exposure to training data, recurring and intrinsic patterns are identified and captured by a learning machine. The goal of the learning process is for a new model be attained such that previously unseen data can be also be predicted accurately. Pattern languages are a similarly distillation of a complex subject that humans can leverage to accelerate their own learning.

We may break up the many patterns we find in Deep Learning into several high level categories. However, considering the complexity of the subject, many of the patterns are bound to be entangled with other patterns that belong to the same or different categories. So there may be many patterns that may logically belong to more than one category. However, we don't see this as being problematic as the more entanglement that we can explicitly highlight, the better our understanding of the overall material.

We've broken up the patterns into seven categories, these are: Representation, Model, Composite, Learning, Exploration, Explanation and Serving. Representation patterns explores how neural network represent data. Model and Composite Model patterns cover different constructs that are able to map between representations. Learning and Exploration patterns explores the mechanisms that evolve a neural network from one model into a more accurate model. Explanation patterns covers different kinds of output from neural networks. Finally, Serving patterns provides additional techniques to address operational concerns when these learning machines are deployed.

There is plenty of new research being published every day on deep learning. This rapid advance of knowledge creates an impedance mismatch between what we see in research and what we find in practice. Furthermore, research papers tends to focus on more simplified models so as to bound the complexity to allow for analysis. Practice however is a lot messier where were the solutions are much more heterogeneous and as a consequence much more difficult to analyze. The importance of a patterns language is magnified in that it provides a framework for mixing and matching components into a total solution. You will be hard pressed to find this kind of discourse in research papers or in introductory texts.

Our collective understanding of DL is continuously and rapidly evolving. One concern about writing a book wherein the fundamentals are in flux is to be too ambitious in attempting to nail down a cohesive and comprehensive presentation. We will make a best attempt to conjure up a logical framework of how everything should fit together. We do hope however that the pattern language approach leads to new and better insight by the community.

History of Deep Learning

<insert history of deep learning>

Current Developments

<insert commercial developments here>

References

https://devblogs.nvidia.com/parallelforall/deep-learning-nutshell-history-training

http://www.andreykurenkov.com/writing/a-brief-history-of-neural-nets-and-deep-learning/

http://kennethfriedman.org/projects/escaping-local-min/

http://ufldl.stanford.edu/tutorial/

http://www.wired.co.uk/article/master-algorithm-pedro-domingos

http://www.jair.org/media/4992/live-4992-9623-jair.pdf A Primer on Neural Network Models for Natural Language Processing

https://arxiv.org/abs/1703.09039v1 Efficient Processing of Deep Neural Networks: A Tutorial and Survey

© 2016 Copyright - Carlos E. Perez