Edit: https://docs.google.com/document/d/1dxTVD6Dj1jJurr9cKcfeLaadwEpdHROvsJgXpgw5ZqA/edit


In layman terms, what is this book about?

Let’s say it is the first time you watch an american football game. Let’s say you watch it live in the stands. You see players playing with a ball that isn’t round, they sometimes throw the ball, sometimes carry the ball and sometimes kick the ball. Nothing really makes sense until you learn the rules of the game. So you ask you buddy to explain it to you. Then you start to notice the chain gang on the side! You can now predict when a team has to kick the ball instead of throwing it. You are now finally able to follow the game.

The next time you watch an american football game, you watch it on TV. There are commentators who are describing the game as it goes on. They begin to speak using words that again don’t make sense. They talk about “ground yardage”, “laterals”, “hail marys”, “blitzing”, etc. You are again at a loss. But as you watch the game long enough, you begin to see what they mean by all these words. You begin to get a glimpse of strategies and tactics that are happening on the field.

However, you notice that there are two commentators, one who does the play-by-play (or color guy) and another one who is the analyst. The play-by-play guys describes the game as it goes on, however the analyst interjects only at certain times. However, when the analyst speaks, he begins to speak in yet another kind of vocabulary. He speaks in more detail about the timing of a move, the mechanics of a throw, the kind of decision that was made, different tradeoffs and even quoting statistics.

This book is about Deep Learning and it is a book of talks about Deep Learning in the way the analyst talks about football. To understand the analyst you need to understand the rules of the game, a vocabulary that describes plays and an understanding of the reasoning behind a play. With this knowledge, you are become someone that is capable of “monday morning quarterbacking.”

What is Deep Learning?

Artificial Neural Networks (ANN) were proposed over half a century ago, however the concept has only gained substantial application in the last five years. Mindboggling results have been achieved with the advancement of semiconductor technology, particularly the Graphic Processing Unit (GPU), the availability of Big Data and newly discovered algorithms. These multilayered ANNs were coined as Deep Learning to highlight its newly discovered capabilities. Deep Learning systems are able to perform machine inference at a level not previously achieved. For the past 5 years Deep Learning systems have bested every other known algorithm in machine learning contests. Deep Learning system have exhibited super-human capabilities in areas of image and speech recognition as well as besting the world champion in the game of Go. It is therefore important for computer professionals to rapidly gain an understanding of this disruptive technology.

Deep Learning is one of the most important new technologies in the study of Artificial Intelligence. Many of the bleeding edge technology companies are retrofitting their infrastructure to support Deep Learning technologies. Google’s CEO Pichai remarked that Google was “re-thinking” all of its products to include more of Deep Learning.

What are Design Patterns?

Design patterns capture reusable elements in a design. For instance, software design patterns are recurring ways to architect software systems. The author employs a pattern language to best explain the concepts underpinning Deep Learning. Patterns do not exist in isolation, the author shows how other patterns are related and how they fit into an overall Deep Learning solution. Included in each pattern in the motivation of each concept, the problems they solve and how they are used in practice. The patterns in this book comprise a cookbook of useful concepts and insights to guide the developer into building creative and innovative applications using Deep Learning.

Why does Deep Learning need Design Patterns?

Deep Learning is not only a fast growing field, but a complex one. Furthermore, literature about state of the art methods are fragmented. By taking a Design Pattern approach we can consolidate all the latest concepts and techniques under a common conceptual framework. In doing so, we make it more convenient for practitioners to build DL solutions.

Deep Learning is a complex and rapidly evolving technology. Dozens of academic papers are published every other week on the subject and just keeping updated is an effort in itself. Furthermore, an overarching conceptual framework is difficult to find and knowledge is diffused in too many places. This innovative book recognizes the need of practitioners for a book that distills the concepts of Deep Learning is an easily accessible and consistent form. Pattern Languages have been adopted by the software community as an effective form of communication of complex subject matters. This book presents a bleeding edge technology in a manner that is accessible to the average software developer or data scientist.

OK, given a simple feed-forward network, what design patterns are involved?

The simplest neural network is a single neuron which performs logistic regression. The most fundamental or canonical patterns in deep learning are: Similarity, Classifier and Collection. The single neuron performs similarity by taking the inner product of the inputs against its weights and classification by activating a sigmoidal squashing function. The single output is then ensembled into a signal for the next layer through the merge operator.

How can Deep Learning design patterns benefit me?

Often times when designing a network architecture for deep learning, one is faced with a myriad of options that may be dependent on the data and problem itself. This in many ways is similar to how software engineers may have several options to solve a problem. Design patterns offer a principled approach based on theory and empirical success (papers and implementations) to suggest the most relevant approaches to problem solving. This book further helps guide the reader with a graph of pattern relationships illustrating how various design choices may affect the overall solution.. Each node in the graph is clearly explained with details on the patterns involved and the pros/cons and tradeoffs of the proposed approach.

I've already architected an approach and it works well, why bother using DL patterns?

Although your approach may currently work well, it may be difficult to understand how we can improve the DL implementation. Deep Learning patterns consolidates a collection of patterns that can give your DL solutions an extra edge. This is achieved through not only a more intuitive understanding of DL mechanisms but with a more comprehensive collection of solutions to various design problems.

Is there a rhyme or reason for the patterns or names?

Like other patterns books, these patterns have long existed, and the author and editors have provided many references detailing the popularity of these methods. When possible, we have chosen names that are: (1) well-recognized, (2) appropriate, (3) not conflicted with other terms in the literature. Some papers/books may define terms differently, and we offer clarification when necessary and also list aliases, synonyms and differentiate similar ideas.

Why should I read this book?

The target audience for this book includes practitioners and theorists alike. We assume a rudimentary knowledge of neural network basics. Just know what they are and how they work intuitively.

This book truly shines when you've had the chance to tinker with several network architectures and are wondering how to improve your network or understand why it works. We offer canonical advice about how to structure such networks and in which directions improvements may lie.

What's the best way to read this book? This book is structured in the following way. First, we introduce and review basic concepts of neural networks. Once the foundation has been laid, we introduce canonical patterns that underlie most other patterns. Once the reader is familiar with these building blocks, one can freely visit the remaining chapters in any order desired to reference the relevant concepts.

Neural networks are all about connections, and so are design patterns. Deep learning patterns in isolation don't do much; groups of patterns are necessary to accomplish great things. As such, we've organized the book with clear graphical maps showing the relationships between patterns and various options that one can choose from in designing their networks.

If you're a practitioner and just want to get a “quick start” on using Deep Learning Patterns, we recommend browing one of the many “cheat sheets” in this book to see examples that are clearly worked out, and offer several options to deploy depending on your problem and data.

I'm a researcher, how can I help?

This book is written as a collaborative wiki and you can submit comments or propose edits to help add new references, clarify the patterns or suggest new content. Due to the fast-paced nature of publishing in the machine learning circles, we also appreciate your help with tagging new papers to identify which patterns they may be related to. To that end, we plan to have support for integrating with Arxiv-Sanity to add annotations for patterns and comments on how the papers relate to patterns. If you're writing a paper, we'd also like to invite you to use our pattern vocabulary to help readers find relevant papers more easily.

I'm a practitioner, how can I help?

Since many of the patterns were developed in lab settings, they need some burn-in on the battleground to prove their worth. We'd appreciate your help in reporting which ideas work and under what conditions. Through this effort, we hope to identify the real-world performance of certain designs.

Design patterns are usually consider post-facto in software, and benefit most when deployment software is refactored. Does early use of a pattern stifle creativity or limit me? Why use design patterns at the outset?

Design patterns help engineers architect solutions and consider alternatives. Often times when designing a solution, one searches the space rapidly for something that works, then work on refinement and improvement begins through polishing and refactoring.

The use of design patterns in this book serves several purposes. The first is that tried and true patterns are presented clearly, and that gives readers an edge in finding that initial solution. Secondly, we present clear alternatives to design choices made so that the reader can find better ways to adapt their design to the problem at hand. Third, we offer ways to explain why a design works in depth using solid theory, rather than through trial and error. Finally, we offer solid theoretical ways to test the networls to ensure that they behave as expected and are aren't easily fooled. We offer solid tests to ensure that networks do not regress with small changes to the data or network architecture.

My software, e.g. Tensorflow, Keras, already includes the latest algorithms and plenty of examples that I can adapt for my project. Why do I need to read this book?

Although it is true that many deep learning frameworks come with excellent working examples, applying domain-specific knowledge to excel on a new dataset is still largely a matter of experimentation. We recommend that you read the chapters on feature patterns to see if your own data fits a particular pattern. Then look at the explanatory patterns to see if the problem you are trying to solve fits one of the patterns. For these two starting points, you can begin to craft a solution that best fits your problem.

If you've already built a model, you may still want to review alternative Models that may fit your problem better. New techniques are always cropping up, and our playbooks are revised to reflect these. Updating your existing model to the new best practices can lead to a better solution.