A REVIEW AND PERFORMANCE ANALYSIS OF NON-LINEAR ACTIVATION FUNCTIONS IN DEEP NEURAL NETWORKS









Abstract

Activation Functions are mathematical equations which perform a nonlinear transformation on the input such that the neuron can be used to learn and perform complex tasks efficiently. A weighted sum of the input is calculated and a bias is added to it to produce an output. The decision to fire a neuron is made after applying the activation function to the output. An analysis of activation functions used in ANN is presented. Our paper explores the efficiency of activation functions on the MNIST and Fashion MNIST datasets. The convergence speed during the backpropagation after applying the activation functions is also shown in this work. Keywords: Activation Function, Neural Network, Hidden Unit, Backpropagation, Deep Neural Networks.


Modules


Algorithms


Software And Hardware

• Hardware: Processor: i3 ,i5 RAM: 4GB Hard disk: 16 GB • Software: operating System : Windws2000/XP/7/8/10 Anaconda,jupyter,spyder,flask Frontend :-python Backend:- MYSQL