Reference

This complementary section explains the derivation of the formulas used by moduleprofiler to estimate the complexity of some neural network modules. It is intended to be used both as a quick reference and for educational purposes.

Note

It is important to consider that all formulas derived in this section are purely based on the mathematical relationship between the input and output of each module. In practice, there could be additional optimizations performed by linear algebra libraries or hardware-specific capabilities that are used to accelerate or minimize the calculations required by a specific module type to compute the corresponding output. For this reason, estimating complexity using moduleprofiler may result in bigger numbers compared to other packages used for similar purposes.

List of reference pages:

• Conv1d (torch.nn.Conv1d)
• Conv2d (torch.nn.Conv2d)
• ConvTranspose1d (torch.nn.ConvTranspose1d)
• ConvTranspose2d (torch.nn.ConvTranspose2d)
• GRUCell (torch.nn.GRUCell)
• GRU (torch.nn.GRU)
• LayerNorm (torch.nn.LayerNorm)
• Linear (torch.nn.Linear)
• LSTMCell (torch.nn.LSTMCell)
• LSTM (torch.nn.LSTM)
• ReLU (torch.nn.ReLU)
• Sigmoid (torch.nn.Sigmoid)
• Softmax (torch.nn.Softmax)
• Tanh (torch.nn.Tanh)