1. Import common initialization methods
from torch.nn.init import xavier_uniform_, xavier_normal_ from torch.nn.init
import kaiming_uniform_, kaiming_normal_
2. Analysis of various initialization methods

* xavier_uniform_(tensor, gain=1.0)
Note: Initializes the input with evenly distributed values tensor. Methods based on 《Understanding the difficulty of training deep
feedforward neural networks - Glorot, X. & Bengio, Y. (2010)》 Paper implementation . What we got in the end Tesor Values are sampled at
U(−a,a) ,

among : \

parameter :

            gain: Scaling factor (optional)

* xavier_normal_(tensor, gain=1.0)
Note: Initializes the input with a value of a positive distribution tensor.  Methods based on 《Understanding the difficulty of training deep
feedforward neural networks - Glorot, X. & Bengio, Y. (2010)》 Paper implementation . What we got in the end Tesor Values are sampled at
,

among : 

* kaiming_uniform_(tensor, a=0, mode='fan_in', nonlinearity='leaky_relu')
Note:  Initializes the input with evenly distributed values tensor. Methods based on 《Delving deep into rectifiers: Surpassing
human-level performance on ImageNet classification - He, K. et al.
(2015)》 Paper implementation . What we got in the end Tesor Values are sampled at U(−bound,bound) ,

among : 

parameter :a: 

         mode: "fan_in" or "fan_out". choice “fan_in" Save the magnitude of the weight variance in forward propagation , ”fan_out"
Preserving amplitude in backward propagation .

        nonlinearity: Nonlinear function . recommend "relu" or "leaky_relu".

* kaiming_normal_(tensor, a=0, mode='fan_in', nonlinearity='leaky_relu')
Note: Initializes the input with a value of a positive distribution tensor. Methods based on 《Delving deep into rectifiers: Surpassing
human-level performance on ImageNet classification - He, K. et al.
(2015)》 Paper implementation . What we got in the end Tesor Values are sampled at ,

among : 

 

          

Technology