Differentiable Programming: A Deep Dive from First Principles

This article provides a comprehensive explanation of differentiable programming, starting from the definition of derivatives in calculus and progressing to concepts like gradients, directional derivatives, and Jacobians. It details three differentiation methods: numerical differentiation, symbolic differentiation, and automatic differentiation (forward and reverse modes), comparing their strengths and weaknesses. Finally, it demonstrates how reverse-mode automatic differentiation, combined with gradient descent, can solve real-world optimization problems using an image de-blurring example.