LayerNorm vs BatchNorm: why Transformers normalize per token, not per batch

BatchNorm normalizes features across the entire mini-batch, which fails for sequence models with variable lengths or small batches. LayerNorm normalizes features within each sample, making it suitable for sequence models. This difference affects training and inference for BatchNorm, but not for LayerNorm. To use LayerNorm, normalize per token, not per batch.