Build Neural Network in C/CUDA (Part 1): Deep Learning Foundations for our Implementation

Code (we will implement this in the next video):
https://github.com/IaroslavElistratov/ml-systems-course

Implementing a minimal deep learning library in C (learning exercise, no ai assistance)
includes:
– 24 naive CUDA/CPU ops
– autodiff engine
– python API (ops bindings/layers/models)
– tensor abstraction
– strides, complex indexing (multi-dim slices like numpy)
– computation-graph visualizer
– saving/loading params
– primitive garbage collection
– autogenerated tests
– optimizers, weight initializers
– etc.

🔗 https://www.linkedin.com/in/iaroslav-elistratov
🔗 https://x.com/iaro_e
🔗 https://github.com/IaroslavElistratov

timestamps:
00:00 Overview
03:46 Why Deep Learning?
19:31 Loss function (part 1)
27:04 What is a “model”?
31:58 Naive approach to training a model
40:10 Recap so far
41:41 Motivation for backpropagation
53:26 Backpropagation intuition
01:05:57 Computational graph
01:07:54 Loss function (part 2)
01:17:08 Limit definition of derivative
01:22:39 Forward Pass
01:27:08 Overview of backward pass
01:29:14 Derivative of the absolute value function (abs)
01:37:38 Chaining derivatives
01:42:47 Derivative of subtraction
01:56:42 Derivative of addition
02:04:14 Derivative of multiplication
02:13:50 Adjusting weights
02:19:31 Interpreting gradients with respect to the parameters (weights)

source