Pruning a PyTorch model might seem like just stripping away weights, but it’s actually a sophisticated technique that can fundamentally alter a model’s computational graph to achieve significant speedups, not just size reduction.
Let’s see what this actually looks like. Imagine a simple linear layer:
import torch
import torch.nn as nn
class SimpleModel(nn.Module):
def __init__(self):
super().__init__()
self.linear1 = nn.Linear(100, 50)
self.relu = nn.ReLU()
self.linear2 = nn.Linear(50, 10)
def forward(self, x):
x = self.linear1(x)
x = self.relu(x)
x = self.linear2(x)
return x
model = SimpleModel()
Now, let’s say we want to prune 50% of the weights in linear1 based on magnitude. PyTorch’s torch.nn.utils.prune module is our tool.
import torch.nn.utils.prune as prune
# Prune 50% of the weights in linear1 by magnitude
prune.l1_unstructured(model.linear1, name="weight", amount=0.5)
# Now, let's look at the weight tensor.
# It's not actually zeroed out yet, but masked.
print(model.linear1.weight)
The prune.l1_unstructured function doesn’t immediately remove the weights or change the tensor shape. Instead, it attaches a weight_mask buffer and a weight_orig parameter to the module. The forward pass is then re-written to use weight_orig * weight_mask.
# Inspecting the module after pruning
print(hasattr(model.linear1, 'weight_mask')) # True
print(hasattr(model.linear1, 'weight_orig')) # True
print(model.linear1.weight) # This will show the masked weights
The magic happens when you remove the pruning reparametrization. This is when the model’s structure is actually altered.
# Make the pruning permanent
prune.remove(model.linear1, 'weight')
# Now, weight_orig and weight_mask are gone.
# The weight tensor itself is modified, with zeros where weights were pruned.
print(hasattr(model.linear1, 'weight_mask')) # False
print(hasattr(model.linear1, 'weight_orig')) # False
print(model.linear1.weight) # This shows the actual modified weight tensor
print(model.linear1.weight.shape) # Shape remains (50, 100)
Notice that the shape of the weight tensor (50, 100) remains the same. Pruning, in this common unstructured form, doesn’t change the tensor dimensions. It simply zeros out specific weights. For inference speedups, especially on hardware that can leverage sparse matrix operations (which is still relatively niche outside of specialized libraries and hardware), the primary benefit comes from reducing the number of non-zero operations.
The real power for inference acceleration, beyond just size reduction, comes from structured pruning. This is where you remove entire neurons, channels, or even layers.
Consider a Convolutional Neural Network (CNN). Pruning individual weights in a convolutional layer (nn.Conv2d) has a similar masking effect as shown above. However, pruning an entire output channel of a convolutional layer is structured pruning.
import torch
import torch.nn as nn
import torch.nn.utils.prune as prune
class SimpleCNN(nn.Module):
def __init__(self):
super().__init__()
self.conv1 = nn.Conv2d(3, 16, kernel_size=3, padding=1) # Output channels: 16
self.relu = nn.ReLU()
self.conv2 = nn.Conv2d(16, 32, kernel_size=3, padding=1) # Output channels: 32
self.pool = nn.AdaptiveAvgPool2d((1, 1))
self.fc = nn.Linear(32, 10)
def forward(self, x):
x = self.conv1(x)
x = self.relu(x)
x = self.conv2(x)
x = self.relu(x)
x = self.pool(x)
x = torch.flatten(x, 1)
x = self.fc(x)
return x
cnn_model = SimpleCNN()
# Let's prune 50% of the *output channels* of conv1.
# This is structured pruning.
# We need to specify pruning on the 'weight' and 'bias' if it exists,
# and importantly, the 'dim' argument tells it which dimension corresponds to output channels.
# For Conv2d, the weight shape is (out_channels, in_channels, kernel_height, kernel_width).
# So, dimension 0 is out_channels.
prune.ln_structured(cnn_model.conv1, name="weight", amount=0.5, n=2, dim=0)
# If bias exists, prune it too. Bias shape is (out_channels,)
if cnn_model.conv1.bias is not None:
prune.ln_structured(cnn_model.conv1, name="bias", amount=0.5, n=2, dim=0)
# After applying structured pruning, the 'weight' and 'bias' are masked.
# The actual removal requires prune.remove.
# When prune.remove is called on structured pruning, the tensor dimensions *change*.
prune.remove(cnn_model.conv1, 'weight')
if cnn_model.conv1.bias is not None:
prune.remove(cnn_model.conv1, 'bias')
print(f"Original conv1 weight shape: (16, 3, 3, 3)")
print(f"Pruned conv1 weight shape: {cnn_model.conv1.weight.shape}")
# The output channels dimension (dim=0) will be reduced.
# If we pruned 50% of 16 channels, we'd expect 8 output channels.
# The shape will become (8, 3, 3, 3).
This structural change is what enables significant inference speedups on standard hardware. A convolutional layer with fewer output channels requires fewer computations in subsequent layers that consume its output. This isn’t just about skipping multiplications; it’s about reducing the overall FLOPs and memory bandwidth requirements by shrinking the tensors that flow through the network.
The process typically involves:
- Pre-training or Fine-tuning: Train a dense model to a satisfactory accuracy.
- Pruning: Apply a pruning strategy (e.g., magnitude-based unstructured, structured channel pruning, sparsity-inducing regularization).
- Fine-tuning (Iterative Pruning): Retrain the pruned model for a few epochs to recover accuracy lost during pruning. This step is crucial.
- Pruning: Repeat steps 2 and 3 iteratively until the desired sparsity or compression level is reached.
- Final Fine-tuning: A last round of fine-tuning on the highly sparse model.
- Export/Removal: Use
prune.removeto make the pruning permanent and reduce the model’s actual size and computational graph.
The most common misconception is that unstructured pruning alone will yield significant speedups on general-purpose hardware. While it reduces the number of non-zero weights, modern hardware (CPUs, GPUs) is optimized for dense matrix operations. Unless you’re using specialized sparse computation libraries or hardware, unstructured pruning primarily reduces model size and memory footprint, but not necessarily inference latency. Structured pruning, by reducing dimensions, directly impacts the computational load and is far more effective for acceleration.
The next hurdle after achieving a compressed model is deploying it efficiently, which often involves converting it to formats like ONNX or TorchScript for optimized inference engines.