Activation Functions

Activation Functions offer ways to introduce non-linearity. It takes a raw input value (often resulting from a linear combination of variables), and maps it to a specific output format. There are many activation functions. We list a popular few here.

Function	Graph (Desmos)
Sigmoid / Logistic Function $σ (x) = \frac{1}{1 + e^{- x}}$ Domain: $(- \infty, \infty)$ Range: $(0, 1)$
Softmax $σ (z)_{i} = \frac{e^{z_{i}}}{\sum_{j = 1}^{K} e^{z_{j}}}$ Extends Sigmoid to multiple dimensions. Image is for two dimensions.
ReLU (Rectified Linear Unit) $f (x) = max (0, x)$ Domain: $(- \infty, \infty)$ Range: $[0, \infty)$
Leaky ReLU $f (x) = {\begin{cases} x & if x > 0 \\ α x & otherwise \end{cases}$ Where $0 < α < 1$ is a small constant, typically 0.01. Domain: $(- \infty, \infty)$ Range: $(- \infty, \infty)$
Heaviside (Step Function) $H (x) = {\begin{cases} 0 & if x < 0 \\ 1 & if x \geq 0 \end{cases}$ Domain: $(- \infty, \infty)$ Range: ${0, 1}$