I recently came across Andrej Karpathy's article about backpropagation. Andrej argues that anyone interested in building neural networks should understand the implementation details of techniques like backpropagation. These details help developers become more effective at building neural networks. Andrej is makes the case that backpropagation is a leaky abstraction because the implementations for it make it seems like a simple concept. Like many other students early in their machine learning journey, I initially avoided diving into implementations of functions like gradient descent or backpropagation, viewing them as tedious blockers on the path to building working models. However, as I've delved deeper into machine learning, I find myself returning to understand these fundamental concepts at a deeper level.
Andrej’s article gave me the idea to implement a neural network with backpropgation and optimization in java from scratch. Why did I choose Java? Partly because it was the first programming language I learned and I haven’t been able to use it extensively since in a while. My computer science program in college was mostly taught in Java but I don’t count those because they were simple console applications. I also wanted to take a small break from python as I use it for professional and other personal work. Additionally, Java does not have a numpy-esq package so many mathematical operations had to be done at a very rudimentary level. Turns out when you have to implement the dot product between two 2-dimensionals array or write your own softmax function, you learn more about how those operations are done.
Full disclosure, I based this project on Neural Networks From Scratch in Python by Harrison Kinsley (aka sentdex). This book excellently teaches the fundamentals of neural networks using pure Python without additional libraries, combining practical code implementation with essential theoretical concepts.
Code
I won't cover all the code in this blog post. You can find the complete repository here. Let's begin with the basic building blocks of a neural network
Vanilla Neural Network
Dense Layer
public class DenseLayer {
private int inputSize;
private int numberOfNeurons;
private double[][] weights;
private double[] biases;
public DenseLayer(int inputSize, int numberOfNeurons) {
this.inputSize = inputSize;
this.numberOfNeurons = numberOfNeurons;
if (inputSize < 1 || numberOfNeurons < 1) {
throw new IllegalArgumentException("Input size and number of neurons must be greater than 0.");
}
// Initialize weights and biases here
this.weights = new double[inputSize][numberOfNeurons];
this.biases = new double[numberOfNeurons];
this.weights = initializeWeights(inputSize, numberOfNeurons);
this.biases = initializeBiases(numberOfNeurons);
}
/**
* Forward pass through the layer.
* * @param input 2D array of inputs (batch size x input size).
*/
public void forward(double[][] input) {
// Store the inputs for backpropagation
this.inputs = input;
// Calculate the output values from inputs, weights, and biases
int batchSize = input.length;
int numberOfNeurons = this.numberOfNeurons;
// Initialize the output array
this.output = new double[batchSize][numberOfNeurons];
// Perform the matrix multiplication (dot product) and add biases
for (int i = 0; i < batchSize; i++) {
for (int j = 0; j < numberOfNeurons; j++) {
this.output[i][j] = biases[j];
for (int k = 0; k < inputSize; k++) {
this.output[i][j] += input[i][k] * weights[k][j];
}
}
}
}
}
Activation Function
public class ActivationCategoricalCrossentropySoftmaxLoss {
private ActivationSoftmax activation;
private LossCategoricalCrossentropy loss;
private double[][] output;
public ActivationCategoricalCrossentropySoftmaxLoss() {
// Softmax activation
this.activation = new ActivationSoftmax();
// Categorical crossentropy loss
this.loss = new LossCategoricalCrossentropy();
}
/**
* * Forward pass through the softmax activation and calculate the loss using
* categorical crossentropy.
*
* @param inputs The input data (2D array)
* @param yTrue The true labels (2D array)
* @return The loss value calculated using categorical crossentropy.
*/
public double forward(double[][] inputs, Object yTrue) {
// Forward pass through softmax activation
this.activation.forward(inputs);
// Get the output (predicted values) from softmax activation
this.output = this.activation.getOutput();
// Calculate the loss using categorical crossentropy
return this.loss.calculate(this.output, yTrue);
}
...
}
Softmax Function
public class ActivationSoftmax {
private double[][] output;
public ActivationSoftmax() {
}
/**
* Forward pass through the softmax layer.
*
* @param inputs 2D array of inputs (batch size x number of classes).
*/
public void forward(double[][] inputs) {
// Unnormalized probabilities
int rows = inputs.length;
int cols = inputs[0].length;
double[][] expValues = new double[rows][cols];
for (int i = 0; i < rows; i++) {
// Find the maximum value in the row to prevent overflow
double rowMax = inputs[i][0];
for (int j = 1; j < cols; j++) {
if (inputs[i][j] > rowMax) {
rowMax = inputs[i][j];
}
}
// Subtract the maximum value from each element in the row and calculate the
// exponentials
for (int j = 0; j < cols; j++) {
expValues[i][j] = Math.exp(inputs[i][j] - rowMax);
}
}
// Normalize the probabilities
double[][] probabilities = new double[rows][cols];
for (int i = 0; i < rows; i++) {
double sum = 0.0;
for (int j = 0; j < cols; j++) {
sum += expValues[i][j];
}
for (int j = 0; j < cols; j++) {
probabilities[i][j] = expValues[i][j] / sum;
}
}
this.output = probabilities;
}
...
}
Categorical Cross-Entropy
public class LossCategoricalCrossentropy {
private double MIN = 1e-7;
private double MAX = 1 - 1e-7;
public LossCategoricalCrossentropy() {
}
/**
* * Forward pass through the categorical crossentropy loss layer.
* This method calculates the categorical crossentropy loss for each sample in
* the batch.
*
* @param yPred 2D array of predicted probabilities (batch size x number of
* classes).
* @param yTrue 1D array of true class labels (for binary classification) or 2D
* array of one-hot encoded labels (for categorical
* classification).
* @return 1D array of categorical crossentropy losses for each sample in the
* batch.
*/
public double[] forward(double[][] yPred, Object yTrue) {
// Calculate the categorical crossentropy loss
int rows = yPred.length;
int cols = yPred[0].length;
// Clip data to prevent division by 0 (log(0))
double[][] clipped = new double[rows][cols];
int samples = clipped.length;
for (int i = 0; i < rows; i++) {
for (int j = 0; j < cols; j++) {
clipped[i][j] = Math.max(this.MIN, Math.min(this.MAX, yPred[i][j])); // Clip values between 1e-15 and 1
// - 1e-15
}
}
// Probabilities for target values
double[] correctConfidences = new double[samples];
// Binary classification
if (yTrue instanceof int[]) {
correctConfidences = correctConfidences(clipped, (int[]) yTrue);
} else if (yTrue instanceof int[][]) {
correctConfidences = correctConfidences(clipped, (int[][]) yTrue);
} else {
throw new IllegalArgumentException("Invalid type for yTrue. Expected int[] or double[][]");
}
// Calculate the loss
double[] losses = negativeLogLikelihoods(correctConfidences);
return losses;
}
...
}
Combined Categorical Cross-Entropy Softmax Loss
For simplicity during the backpropagation phase, we'll combine our softmax activation and categorical cross-entropy loss function into one class.
public class ActivationCategoricalCrossentropySoftmaxLoss {
private ActivationSoftmax activation;
private LossCategoricalCrossentropy loss;
private double[][] output;
public ActivationCategoricalCrossentropySoftmaxLoss() {
// Softmax activation
this.activation = new ActivationSoftmax();
// Categorical crossentropy loss
this.loss = new LossCategoricalCrossentropy();
}
/**
* * Forward pass through the softmax activation and calculate the loss using
* categorical crossentropy.
*
* @param inputs The input data (2D array)
* @param yTrue The true labels (2D array)
* @return The loss value calculated using categorical crossentropy.
*/
public double forward(double[][] inputs, Object yTrue) {
// Forward pass through softmax activation
this.activation.forward(inputs);
// Get the output (predicted values) from softmax activation
this.output = this.activation.getOutput();
// Calculate the loss using categorical crossentropy
return this.loss.calculate(this.output, yTrue);
}
...
}
Main Application
Now that we have our core components. Lets make a forward pass.
import models.Data;
public class App {
public static void main(String[] args) throws Exception {
// Create dataset
// 100 samples, 3 classes
Data dataset = SpiralDataGenerator.createDataset(100, 3);
double[][] X = dataset.getX();
int[] y = dataset.getY();
// Create a DenseLayer with 2 input features and 3 output neurons
// First DenseLayer with 2 inputs and 3 neurons
DenseLayer denseLayer1 = new DenseLayer(2, 3);
// ReLU activation layer
ActivationReLU activation1 = new ActivationReLU();
// Second DenseLayer with 3 input features (from the previous layer) and 3
// output neurons
DenseLayer denseLayer2 = new DenseLayer(3, 3);
// Softmax classifier's combined loss and activation layer
ActivationCategoricalCrossentropySoftmaxLoss lossActivation = new ActivationCategoricalCrossentropySoftmaxLoss();
// Perfomance of forward pass through the first layer
// Forward pass through the first layer
denseLayer1.forward(X);
// Forward pass through ReLU activation
activation1.forward(denseLayer1.getOutput());
// Perform a forward pass through the second layer
// Forward pass through the second layer
denseLayer2.forward(activation1.getOutput());
// Forward pass through the loss layer
double loss = lossActivation.forward(denseLayer2.getOutput(), y);
// Get the predicted class labels
int[] predictions = argMax(lossActivation.getOutput());
// Calculate accuracy
double accuracy = calculateAccuracy(predictions, y
}
}
We have built up our base layers and can now focus on backpropagation.
Backpropagation
Combined Categorical Loss Backward Pass
Let's begin with the combined softmax and cross-entropy implementation. The critical part of this process is calculating and normalizing the gradient.
public class ActivationCategoricalCrossentropySoftmaxLoss {
double[][] dinputs;
....
/**
* Backward pass through the loss layer.
* This method calculates the gradient of the loss with respect to the inputs.
*
* @param dvalues The gradient of the loss with respect to the output of the
* softmax activation (2D array).
* @param yTrue The true labels (1D array).
*/
public void backward(double[][] dvalues, Object yTrue) {
// Initialize dinputs
this.dinputs = new double[dvalues.length][dvalues[0].length];
int samples = dvalues.length;
int numClasses = dvalues[0].length;
int[] yTrueint = null;
// Check if labels are binary or categorical
if (yTrue instanceof int[]) {
yTrueint = (int[]) yTrue;
} else if (yTrue instanceof int[][]) {
// not implementated
System.out.println("yTrue is int[][]");
} else {
throw new IllegalArgumentException("yTrue must be an int[] or int[][]");
}
// Copy to modify the original array
for (int i = 0; i < samples; i++) {
for (int j = 0; j < numClasses; j++) {
this.dinputs[i][j] = dvalues[i][j];
}
}
// Subtract the true class from the output
for (int i = 0; i < samples; i++) {
this.dinputs[i][yTrueint[i]] -= 1;
}
// Normalize the gradient by the number of samples
for (int i = 0; i < samples; i++) {
for (int j = 0; j < numClasses; j++) {
this.dinputs[i][j] /= samples;
}
}
}
/**
* Returns the gradient of the loss with respect to the inputs.
*
* @return The gradient of the loss with respect to the inputs (2D array).
*/
public double[][] getDInputs() {
return this.dinputs;
}
}
Dense Layer Backward Pass
In this dense layer backpropagation implementation, we transpose the inputs, calculate the dot product, compute the biases, and determine the gradients on the inputs. While this process is straightforward, it's amusing how verbose the Java implementation becomes compared to Python—though I brought this on myself by choosing Java.
/**
* DenseLayer.java
* Represents a dense layer in a neural network.
*/
public class DenseLayer {
private double[][] dWeights;
private double[] dBiases;
private double[][] dInputs;
...
/**
* Backward pass through the layer.
*
* @param dValues Gradient of the loss with respect to the output of this layer
* (2D array).
*/
public void backward(double[][] dValues) {
// Gradient on parameters, dot product of inputs and gradient on output
this.dWeights = new double[inputSize][numberOfNeurons];
// Gradient on biases, sum of gradient on output
this.dBiases = new double[dValues[0].length];
double[][] inputsT = transpose(this.inputs);
this.dWeights = dot(inputsT, dValues);
for (int i = 0; i < dValues.length; i++) {
for (int j = 0; j < dValues[0].length; j++) {
this.dBiases[j] += dValues[i][j];
}
}
// Gradient on inputs
double[][] weightsT = transpose(this.weights);
this.dInputs = dot(dValues, weightsT);
}
/**
* Transpose a matrix.
*
* @param matrix 2D array to be transposed.
* @return Transposed matrix (2D array).
*/
private double[][] transpose(double[][] matrix) {
int rows = matrix.length;
int cols = matrix[0].length;
double[][] transposed = new double[cols][rows];
for (int i = 0; i < rows; i++) {
for (int j = 0; j < cols; j++) {
transposed[j][i] = matrix[i][j];
}
}
return transposed;
}
/**
* Matrix multiplication (dot product) of two matrices.
*
* @param a
* @param b
* @return Result of the dot product of a and b.
*/
private double[][] dot(double[][] a, double[][] b) {
int rowsA = a.length;
int colsA = a[0].length;
int rowsB = b.length;
int colsB = b[0].length;
if (colsA != rowsB) {
throw new IllegalArgumentException("Incompatible dimensions for matrix multiplication.");
}
double[][] result = new double[rowsA][colsB];
for (int i = 0; i < rowsA; i++) {
for (int j = 0; j < colsB; j++) {
for (int k = 0; k < rowsB; k++) {
result[i][j] += a[i][k] * b[k][j];
}
}
}
return result;
}
}
Activation Function Backward Pass
Simple backpropagation of ReLU, not as much code as the dense layer implementation.
/**
* ActivationReLU.java
* Represents the ReLU (Rectified Linear Unit) activation function.
* This class provides methods for the forward and backward pass of the ReLU
* activation function.
*/
public class ActivationReLU {
private double[][] dInputs;
...
/**
* Backward pass through the ReLU activation function.
* @param dValues Gradient of the loss with respect to the output of this layer
* (2D array).
*/
public void backward(double[][] dValues) {
// Copy original since we will need to modify it
dInputs = new double[dValues.length][dValues[0].length];
for (int i = 0; i < dValues.length; i++) {
for (int j = 0; j < dValues[0].length; j++) {
dInputs[i][j] = dValues[i][j];
}
}
// Zero gradient for negative inputs (could merge with the above loop)
for (int i = 0; i < dValues.length; i++) {
for (int j = 0; j < dValues[0].length; j++) {
if (inputs[i][j] <= 0) {
dInputs[i][j] = 0;
}
}
}
}
/**
* Get the gradient of the inputs.
*
* @return The gradient of the inputs (2D array)
*/
public double[][] getDInputs() {
return dInputs;
}
}
Adding Backward Passes
public class App {
public static void main(String[] args) throws Exception {
// Backward pass
lossActivation.backward(lossActivation.getOutput(), y);
denseLayer2.backward(lossActivation.getDInputs());
activation1.backward(denseLayer2.getDInputs());
denseLayer1.backward(activation1.getDInputs());
}
}
Conclusion
There are other things that I did in this project that we’re out of scope for this blog. You can find more details about the code in the repository. I’ll be honest with and say that I didn’t fully understood the details of backpropagation before implementing several times. It’s not a difficult concept but more so, there are many moving parts to keep track off. The math looks intimidating but it is just derivatives.
Overall, this project alone won't make me better at building neural networks compared to working with actual datasets and real-world problems. However, it has given me the confidence to dive deep into technical challenges whenever necessary.