Techniques for Optimizing PyTorch Models

• Forward Automatic Differentiation -- For cases when $N_k$ is larger than $N_0$ in the computational graph
• Quantization -- Technique to reduce computational load and fasten inferencing
• Automatic Mixed Precision -- To match appopriate datatype for each operation and reduce runtime
• Knowledge Distillation -- Method to transfer knowledge from larger to small network
• Profiling -- To assess time and memory consumption for a model's operation
• Pruning -- Method to reduce model parameters without affecting much of the performance