Differences

This shows you the differences between two versions of the page.

Link to this comparison view

introduction [2017/03/29 20:13]
introduction [2017/03/29 20:13] (current)
Line 1: Line 1:
 +https://​docs.google.com/​a/​codeaudit.com/​document/​d/​1ygFTDaRnVeXKkQqPTP8Oq92z2BtGLf7nkGE2njzb50k/​edit?​usp=sharing
 +
 +====== Introduction ======
 +"​Nature'​s imagination far surpasses our own."
 +- Richard Feynman ​
 +
 +"​Writing is perhaps the greatest of human inventions, binding together people who never knew each other, citizens of distant epochs. Books break the shackles of time. A book is proof that humans are capable of working magic."​
 +- Carl Sagan
 +
 +
 +
 +
 +
 +A solid theoretical foundation of why Deep Learning works is still a topic of much research. ​  This leads to the question as to how we can create a Pattern Language without a solid foundation of the mechanisms that make it work?  Can we begin with verified first principles and work our way towards a more solid understanding of the field?
 +
 +Deep Learning is a rapid evolving approach to machine learning. ​ What we have found is that there are several recurring patterns in practice and in the literature. ​ By capturing these patterns and exploring their relationships,​ we may develop an improved understanding of the subject. ​ Alexander'​s originated the idea of a pattern language that aided in the expression of the architectural concerns of buildings and municipalities. ​ It is doubtful that he was able to create a fundamental theory on architecture. ​ However, by exploring the various recurring patterns and their inter-relationships,​ Alexander was able to build a comprehensive compilation of of observed patterns. ​ This book is a similar compilation that will allow readers to become exposed to the multitude of recurring patterns in Deep Learning. ​ This will aid in improving the practitioners understanding of an otherwise extremely complex subject.
 +
 +The approach of developing a pattern language to gain understanding is quite analogous to the core methodology of machine learning. ​  ​Machine learning in general is concerned with algorithms that are able to learn a model based only on representative training data.  Given sufficient exposure to training data, recurring and intrinsic patterns are identified and captured by a learning machine. ​ The goal of the learning process is for a new model be attained such that previously unseen data can be also be predicted accurately. ​ Pattern languages are a similarly distillation of a complex subject that humans can leverage to accelerate their own learning.
 +
 +We may break up the many patterns we find in Deep Learning into several high level categories. ​ However, considering the complexity of the subject, many of the patterns are bound to be entangled with other patterns that belong to the same or different categories. ​ So there may be many patterns that may logically belong to more than one category. ​  ​However,​ we don't see this as being problematic as the more entanglement that we can explicitly highlight, the better our understanding of the overall material.
 +
 +We've broken up the patterns into seven categories, ​ these are: [[Representation]],​ [[Model]], [[Composite]],​ [[Learning]], ​ [[Exploration]],​ [[Explanation]] and [[Serving]]. ​  ​Representation patterns explores how neural network represent data.   Model and Composite Model patterns cover different constructs that are able to map between representations. ​ Learning and Exploration patterns explores the mechanisms that evolve a neural network from one model into a more accurate model. ​  ​Explanation patterns covers different kinds of output from neural networks. ​ Finally, Serving patterns provides additional techniques to address operational concerns when these learning machines are deployed.
 +
 +There is plenty of new research being published every day on deep learning. ​ This rapid advance of knowledge creates an impedance mismatch between what we see in research and what we find in practice. ​ Furthermore,​ research papers tends to focus on more simplified models so as to bound the complexity to allow for analysis. ​ Practice however is a lot messier where were the solutions are much more heterogeneous and as a consequence much more difficult to analyze. ​  The importance of a patterns language is magnified in that it provides a framework for mixing and matching components into a total solution. ​ You will be hard pressed to find this kind of discourse in research papers or in introductory texts. ​  
 +
 +
 +Our collective understanding of DL is continuously and rapidly evolving. One concern about writing a book wherein the fundamentals are in flux is to be too ambitious in attempting to nail down a cohesive and comprehensive presentation. We will make a best attempt to conjure up a logical framework of how everything should fit together. We do hope however that the pattern language approach leads to new and better insight by the community.
 +
 +
 +** History of Deep Learning **
 +
 +<insert history of deep learning>​
 +
 +** Current Developments **
 +
 +<insert commercial developments here>
 +
 +
 +
 +**References**
 +
 +https://​devblogs.nvidia.com/​parallelforall/​deep-learning-nutshell-history-training
 +
 +http://​www.andreykurenkov.com/​writing/​a-brief-history-of-neural-nets-and-deep-learning/​
 +
 +http://​kennethfriedman.org/​projects/​escaping-local-min/​
 +
 +http://​ufldl.stanford.edu/​tutorial/​
 +
 +http://​www.wired.co.uk/​article/​master-algorithm-pedro-domingos
 +
 +http://​www.jair.org/​media/​4992/​live-4992-9623-jair.pdf ​ A Primer on Neural Network Models
 +for Natural Language Processing
 +
 +https://​arxiv.org/​abs/​1703.09039v1 Efficient Processing of Deep Neural Networks: A Tutorial and Survey
 +
 +
 +
 +© 2016 Copyright - Carlos E. Perez
 +