When building your Deep Learning model, activation functions are an important choice to make. In this article, we’ll review the main activation functions, their implementations in Python, and advantages/disadvantages of each.

# Linear Activation

Linear activation is the simplest form of activation. In that case, is just the identity. If you use a linear activation function the wrong way, your whole Neural Network ends up being a regression :

Linear activations are only needed when you’re considering a regression problem, as a last layer. The whole idea behind the other activation functions is to create non-linearity, to be able to model highly non-linear data that cannot be solved by a simple regression !

# ReLU

ReLU stands for **Rectified Linear Unit**. It is a widely used activation function. The formula is simply the maximum between and 0 :

To implement this in Python, you might simply use :

```
def relu(x) :
return max(x, 0)
```

The derivative of the ReLU is :

- if is greater than 0
- if is smaller or equal to 0

```
def der_relu(x):
if x <= 0 :
return 0
if x > 0 :
return 1
```

Let’s simulate some data and plot them to illustrate this activation function :

```
import numpy as np
import matplotlib.pyplot as plt
# Data which will go through activations
x = np.linspace(-10,10,100)
plt.figure(figsize=(12,8))
plt.plot(x, list(map(lambda x: relu(x),x)), label="relu")
plt.plot(x, list(map(lambda x: der_relu(x),x)), label="derivative")
plt.title("ReLU")
plt.legend()
plt.show()
```

Advantage | Easy to implement and quick to compute |

Disadvantage | Problematic when we have lots of negative values, since the outcome is always 0 and leads to the death of the neuron |

# Leaky-ReLU

Leaky-ReLU is an improvement of the main default of the ReLU, in the sense that it can handle the negative values pretty well, but still brings non-linearity.

The derivative is also simple to compute :

- if
- else

```
def leaky_relu(x):
return max(0.01*x,x)
def der_leaky_relu(x):
if x < 0 :
return 0.01
if x >= 0 :
return 1
```

And we can plot the result of this activation function :

```
plt.figure(figsize=(12,8))
plt.plot(x, list(map(lambda x: leaky_relu(x),x)), label="leaky-relu")
plt.plot(x, list(map(lambda x: der_leaky_relu(x),x)), label="derivative")
plt.title("Leaky-ReLU")
plt.legend()
plt.show()
```

Advantage | Leaky-ReLU overcomes the problem of the death of the neuron linked to a zero-slope. |

Disadvantage | The factor 0.01 is arbitraty, and can be tuned (PReLU for parametric ReLU) |

# ELU

Exponential Linear Units (ELU) try to make the mean activations closer to zero, which speeds up learning.

if , and otherwise, where is a positive constant.

The derivative is :

- if
- otherwise

If we set to 1 :

```
def elu(x):
if x > 0 :
return x
else :
return (np.exp(x)-1)
def der_elu(x):
if x > 0 :
return 1
else :
return np.exp(x)
```

```
plt.figure(figsize=(12,8))
plt.plot(x, list(map(lambda x: elu(x),x)), label="elu")
plt.plot(x, list(map(lambda x: der_elu(x),x)), label="derivative")
plt.title("ELU")
plt.legend()
plt.show()
```

Advantage | Can achieve higher accuracy than ReLU |

Disadvantage | Same as ReLU, and a needs to be tuned |

# Softplus

The Softplus function is a continuous approximation of ReLU. It is given by :

The derivative of the softplus function is :

You can implement them in Python :

```
def softplus(x):
return np.log(1+np.exp(x))
def der_softplus(x):
return 1/(1+np.exp(x))*np.exp(x)
```

```
plt.figure(figsize=(12,8))
plt.plot(x, list(map(lambda x: softplus(x),x)), label="softplus")
plt.plot(x, list(map(lambda x: der_softplus(x),x)), label="derivative")
plt.title("Softplus")
plt.legend()
plt.show()
```

Advantage | Softplus is continuous and might have good properties in terms of derivability. It is interesting to use it when the values are between 0 and 1. |

Disadvantage | As ReLU, problematic when we have lots of negative values, since the outcome gets really close to 0 and might lead to the death of the neuron |

# Sigmoid

Sigmoid is one of the most common activation functions in litterature these days. The sigmoid function has the following form :

The derivative of the sigmoid is :

In Python :

```
def softmax(x):
return 1/(1+np.exp(-x))
def der_softmax(x):
return 1/(1+np.exp(-x)) * (1-1/(1+np.exp(-x)))
```

```
plt.figure(figsize=(12,8))
plt.plot(x, list(map(lambda x: softmax(x),x)), label="softmax")
plt.plot(x, list(map(lambda x: der_softmax(x),x)), label="derivative")
plt.title("Softmax")
plt.legend()
plt.show()
```

It might not be obvious when considering data between and only, but the sigmoid is subject to the vanishing gradient problem. This means that the gradient will tend to vanish as takes large values.

Since the gradient is the sigmoid times 1 minus the sigmoid, the gradient can be efficiently computed. If we keep track of the sigmoids, we can compute the gradient really quickly.

Advantage | Interpretability of the output mapped between 0 and 1, compute gradient quickly |

Disadvantage | Vanishing Gradient |

# Hyperbolic Tangent

Hyperbolic tangent is quite similar to the sigmoid function, expect is maps the input between -1 and 1.

The derivative is computed in the following way :

Implement this in Python :

```
def hyperb(x):
return (np.exp(x)-np.exp(-x)) / (np.exp(x) + np.exp(-x))
def der_hyperb(x):
return 1 - ((np.exp(x)-np.exp(-x)) / (np.exp(x) + np.exp(-x)))**2
```

And plot it :

```
plt.figure(figsize=(12,8))
plt.plot(x, list(map(lambda x: hyperb(x),x)), label="hyperbolic")
plt.plot(x, list(map(lambda x: der_hyperb(x),x)), label="derivative")
plt.title("Hyperbolic")
plt.legend()
plt.show()
```

Advantage | Efficient since it has mean 0 in the middle layers between -1 and 1 |

Disadvantage | Vanishing gradient too |

# Arctan

This activation function maps the input values in the range . It is the inverse of a hyperbolic tangent function.

And the derivative :

In Python :

```
def arctan(x):
return np.arctan(x)
def der_arctan(x):
return 1 / (1+x**2)
```

And plot it :

```
plt.figure(figsize=(12,8))
plt.plot(x, list(map(lambda x: arctan(x),x)), label="arctan")
plt.plot(x, list(map(lambda x: der_arctan(x),x)), label="derivative")
plt.title("Arctan")
plt.legend()
plt.show()
```

Advantage | Simple derivative |

Disadvantage | Vanishing gradient |

Like it? Buy me a coffee

## Leave a comment