Change the threshold value of the keras RELU activation function

Question

I am trying to change the threshold value of the activation function Relu while building my neural network.

So, the initial code was the one written below where the default value of the relu threshold is 0.

model = Sequential([
    Dense(n_inputs, input_shape=(n_inputs, ), activation = 'relu'),
    Dense(32, activation = 'relu'),
    Dense(2, activation='softmax')
])

However, Keras provides a function implementation of the same which can be reffered to here and adding a screenshot as well.

So, I changed my code to the following to pass a custom function only to get the following error.

from keras.activations import relu
model = Sequential([
    Dense(n_inputs, input_shape=(n_inputs, ), activation = relu(threshold = 2)), 
    Dense(32, activation = relu(threshold = 2)),
    Dense(2, activation='softmax')
])

Error: TypeError: relu() missing 1 required positional argument: 'x'

I understand the error is that I am not using x in the relu function but there is no way for me to pass something like that. The syntax expects me to write model.add(layers.Activation(activations.relu)) but then I won't be able to change the threshold. This is where I need a workaround or solution.

I then used the Layer implementation of the ReLU function which worked for me as written below but I want to find out if there is a way I can make the activation function implementation work because the layer is not always convenient to add and I want to make more modifications inside the Dense function.

Code which worked for me:-

from keras.layers import ReLU
model = Sequential([
    Dense(n_inputs, input_shape=(n_inputs, )),
    ReLU(threshold=4), 
    Dense(32),
    ReLU(threshold=4),
    Dense(2, activation='softmax')
])

Answer 1

The error you're facing is reasonable. However, you can use the following trick on the relu function for your work. In this way, you define a function that takes necessary arguments e.g alpha, threshold etc, and in the function body, you define another function that calculates relu activations with these parameters, and the end returns to the upper function.

# help(tf.keras.backend.relu)
from tensorflow.keras import backend as K
def relu_advanced(alpha=0.0, max_value=None, threshold=0):        
    def relu_plus(x):
        return K.relu(x, 
                      alpha = tf.cast(alpha, tf.float32), 
                      max_value = max_value,
                      threshold= tf.cast(threshold, tf.float32))
    return relu_plus

Samples:

foo = tf.constant([-10, -5, 0.0, 5, 10], dtype = tf.float32)
tf.keras.activations.relu(foo).numpy()
array([ 0.,  0.,  0.,  5., 10.], dtype=float32)

x = relu_advanced(threshold=1)
x(foo).numpy()
array([-0., -0.,  0.,  5., 10.], dtype=float32)

For your case, simply use as follows:

model = Sequential([
    Dense(64, input_shape=(32, ), activation = relu_advanced(threshold=2)), 
    Dense(32, activation = relu_advanced(threshold=2)),
    Dense(2, activation='softmax')
])