Dimensions of this neural network i.e (4 inputs, 2 hidden layer with X neurons each, etc)

Question

I was looking at tensorflow examples by Aymeric Damien (https://github.com/aymericdamien/TensorFlow-Examples/blob/master/examples/3_NeuralNetworks/multilayer_perceptron.py) and in multilayer_perceptron.py he uses a neural net to classify MNIST digits. I think he is using a neural network with 784inputs, with 2 hidden layers with 256 neurons each, and 10 outputs. Am I correct? How do matrix dimensions in weights and biases in multilayer_perceptron.py correspond with ANN "dimensions" (#inputs, #hidden layers, #output, #neurons in each hidden layer, etc. Thank you!

Answer 1

This is a 3-layer neural network (2 hidden layers and an output layer).

The connection between the inputs to the first hidden layer has 784 x 256 weights with 256 biases. This configuration is due to the fact that each of the 784 inputs is fully connected to the 256 hidden layer nodes, and each hidden layer node has 1 bias.

The connection between that first hidden layer to the second hidden layer has 256 x 256 weights due to full connectivity between the layers. The second layer's 256 nodes each has 1 bias.

The connection between the second hidden layer and the output layer is similar. There are 256 x 10 weights (for the second hidden layer's 256 nodes and the output layer's 10 nodes), and each output node has 1 bias.

There are thus 785*256 + 256*256 + 256*10 = 269,056 weights and 256 + 256 + 10 = 522 biases.

The figure below should explain it fully.