A Gentle Introduction to Differentiable Programming
Learn how AI tools can solve non-AI problems
At a high level, differentiable programming is the paradigm of writing software that can be automatically differentiated. In plain English, we can quickly inspect how changes in the program inputs affect the program outputs. For instance, say we code an automatically differentiable car simulator. Using this system, we can simulate a car and study how changing some of its variables (e.g., tire radius, suspension, etc.) relate to changes in its outputs (e.g., fuel consumption, noise, etc.).
The beauty of such a simulator is that it enables us to improve our simulated car continuously. For example, if our goal is to reduce fuel consumption, we can iteratively run our simulation, look for variables that negatively affect its consumption, and try new settings. Likewise, a differentiable factory simulator could tune its operations to reduce waste or increase yield.
Differentiation is not a novel concept. The true innovation with Differentiable Programming is doing it automatically, without custom routines or solvers. This tiny change carries considerable implications, as optimization through differentiation is a powerful but prohibitively expensive framework.
So far, solving problems using derivatives has been a niche topic reserved for more mathematically inclined fields, such as AI and operational research. However, with automatic differentiation, much of the mathematical groundwork needed to get these systems up and running is lifted, exposing these methods to a much wider audience.
If you are still unsure if this concept is relevant to you, consider the amount of progress made in deep learning. With automatic differentiation, researchers started to ask these systems how to improve their networks to perform better on specific datasets, a process known as backpropagation. By doing so effortlessly, the field began to explore different networks and approaches more easily, culminating in the great success we see today, with models such as GPT-3 and StyleGAN.
Throughout this article, I elaborate on what differentiation is and what it can accomplish, followed by an overview of how you can use TensorFlow for general-purpose differentiable…
