Gradient flow describes the movement of gradients during the training of a machine learning model. A gradient is a value (vector or scalar) that shows the direction and magnitude of the steepest increase of a function, indicating how a function changes between consecutive points. The cumulative movement of these gradients across all steps is referred to as gradient flow.
This concept is important not only for learning but also for optimizing neural networks after training. Many neurons in a trained network may contribute very little to its performance while still consuming memory. This can be determined by looking at their activations. In neural networks, the activation of a neuron is its output value after applying an activation function to its weighted input. For a simple linear component, this is the result of the equation mx+b.
Neurons with consistently low activations across multiple test inferences have minimal impact on the network’s output. These neurons can be safely pruned to reduce the network’s size and improve efficiency without significantly affecting its performance.