7.7
2 Module rml-neural/activation.
This module defines a set of activation functions, or method
that may be used to determine the sensitivity of neurons in a network layer.
To support both forward and backward propagation each method contains the
activation function and it’s derivative. This
Wikipedia
page has a good overview of a number of activation functions.
2.1 Activation Function Structure
Contracts that encapsulate the pattern data-type or false.
Contracts used to define the procedures used in the structures below. Both
activation and derivative functions are represented as a procedure that take
a single, and return a single,
real? or
flonum?. They are
equivalent to the following contract values.
See also Parallelism with Futures
in The Racket Guide In general it
is preferable to use the flonum-activator? structure and the
corresponding flonum-activation/c form as this reduces the numeric
conversions and allows optimization such as futures to work efficiently.
This structure provides the activator function, it’s derivative, and an optional
expectation value for a given method.
f is the activation function, \phi(v_i)
df is the activation derivative function, \phi^\prime(v_i)
– sometimes shown as \phi^{-1}(v_i)
α is an optional stochastic variable sampled from a uniform
distribution at training time and fixed to the expectation value
of the distribution at test time
2.2 Activation Functions
Each of the activator? structures below will be defined by it’s
activation function (the derivative is not shown). A sample plot shows the
shape of the activation function in red and it’s derivative in turquoise.
\phi(v_i) =
\begin{cases}
0 & \text{for } v_i < 0\\
1 & \text{for } v_i \geq 0
\end{cases}
\phi(v_i) = \frac{1}{1+e^{-v_i}}
\phi(v_i) = \operatorname{atan}^{-1}(v_i)
\phi(v_i) = \frac{v_i}{1+\left|v_i\right|}
\phi(v_i) = \frac{v_i}{\sqrt{1+\alpha v_{i}^2}}
\phi(v_i) =
\begin{cases}
\frac{v_i}{\sqrt{1+\alpha v_{i}^2}} & \text{for } v_i < 0\\
v_i & \text{for } v_i \geq 0
\end{cases}
\phi(v_i) =
\begin{cases}
0 & \text{for } v_i < 0\\
v_i & \text{for } v_i \geq 0
\end{cases}
\phi(v_i) =
\begin{cases}
\delta v_i & \text{for } v_i < 0\\
v_i & \text{for } v_i \geq 0
\end{cases}
Note that the fixed form of this activator uses a delta value \delta=0.01.
\phi(v_i) = \ln\left( 1 + e^{v_i} \right)
\phi(v_i) = \frac{\sqrt{v_{i}^2+1}-1}{2}+v_i
\phi(v_i) =
\begin{cases}
1 & \text{for } v_i = 1\\
\frac{\sin(v_i)}{v_i} & \text{for } v_i \neq 0
\end{cases}
