No account yet ?

In deep learning, the goal is to find the optimum weights of the model to get the desired output.

In transfer learning, the network is initialized using the best pre-trained weights.

The question is how do you initialize the weights for a non-pretrained model?

Training algorithms for deep learning models are usually iterative in nature and thus require the user to specify some initial point from which to begin the iterations.

Moreover, training deep models is a sufficiently difficult task that most algorithms are strongly affected by the choice of initialization.

All 0s or All 1s

One of the ways is to initialize all weights to 0s.

As all the weights are same, the activations in all hidden units are also the same.

This makes the gradient w.r.t to each weight be same.

Thus, the problem arises as to which weight the network should update and by how much i.e. backpropagation finds it difficult to minimize the loss.

The same problem occurs if all weights are initialized as 1s.

Uniform distribution

The other way is to initialize weights randomly from a uniform distribution.

Every number in uniform distribution has equal probability to be picked.

General rule

Choosing high values of weights is not the best for the model as it brings problems of exploding and vanishing gradients.

The general way to initialize weights is to select small random values, which are close to 0.

Normal distribution

Another way is to initialize weights randomly from a normal distribution.

As most values are concentrated towards the mean, most of the random values selected have higher probability to be closer to mean (say μ = 0).

Source : https://kharshit.github.io/blog/2019/02/08/weight-initialization-in-neural-nets