Tensorflow. call() method to create a baseline model which averages the "inputs" with type framework.ops.Tensor for timeseries data

Question

I have been following the "Time series forecasting" tutorial and I have some troubles making an own multi-step baseline model that can average the input data in the time axis, and repeat it as the target values in the model’s prediction.

In the link above, the authors of the tutorial use simple baselines such as the code below, to take the last time element of the input and tile (or repeat) it OUT_STEPS times.

class MultiStepLastBaseline(tf.keras.Model):
  def call(self, inputs):
    return tf.tile(inputs[:, -1:, :], [1, OUT_STEPS, 1])

Or a baseline that just repeats the same values of the data in the past, according to the number of steps stated in the window of data (input_width) of the model.

class RepeatBaseline(tf.keras.Model):
  def call(self, inputs):
    return inputs

I would like to take all the elements of the inputs and apply a simple average to them before tf.tile them. Basically apply an average to RepeatBaselineand repeat the average as inMultiStepLastBaseline`.

The minimal code to produce a tf.Dataset that can be replicated with the code in the tutorial (https://www.tensorflow.org/tutorials/structured_data/time_series) is the following:

import numpy as np
import pandas as pd
import tensorflow as tf

n =  8
rng = np.random.default_rng(seed=0)
df = pd.DataFrame(np.around(rng.random((n, 4)), 1), columns=['a','b','c','y'])

In the tutorial, the data is managed by a window of consecutive samples from the data, which can be emulated with:

# Settings of what a window object would have
OUT_STEPS = 2  
input_width = 2   # Take two rows (or time steps) of all columns as input
label_width = OUT_STEPS  # Size of the prediction (output)
shift = OUT_STEPS  # Time (or rows) offset between input and output
total_window_size = input_width + shift
batch_size = 1
label_index = None #In the future will be the index of 'y'

# Just a conversion of the df to an tf._.EagerTensor
data = np.array(df.values, dtype=np.float32)
def stack_data(data, total_window_size):
    batches = []
    start = 0
    end = total_window_size
    for start in range(data.shape[0]-1):
        batch = data[start:end]
        start = start + total_window_size + 1
        end = start
        if batch.shape[0] == total_window_size:
            batches.append(batch)
    return tf.stack(batches)

stacked_data = stack_data(data, total_window_size)

Further, the data is manipulated and transformed into a dataset. The minimal code is

input_slice = slice(0, input_width)
label_slice = slice(total_window_size-label_width, None)
def split_stacked_data(stacked_data):
    """
    Split dataset into inputs and labels (or targets)
    https://www.tensorflow.org/tutorials/structured_data/time_series#2_split
    """
    inputs = stacked_data[:,input_slice, :] 
    labels = stacked_data[:,label_slice,label_index:] 
    inputs.set_shape([None, input_width, None])
    labels.set_shape([None, label_width, None])
    return inputs, labels
# inputs, labels = split_stacked_data(stacked_data)

input_dataset = tf.keras.utils.timeseries_dataset_from_array(
        data=data, targets=None, sequence_length=total_window_size,  
        sequence_stride=1, shuffle=False, batch_size=batch_size)
input_dataset = input_dataset.map(split_stacked_data)

So far, in the MyAverageBaseline model, I have succeeded to pass the inputs through a Dense model and make an Average of the data. Since the df has four features, a similar number of units are declared in de Dense. However, the results are not what I expect to get.

class MyAverageBaseline(tf.keras.Model):
    def __init__(self, out_steps, label_index=None):
        super().__init__()
        self.label_index = label_index
        self.out_steps = out_steps
        self.a_model = tf.keras.layers.Dense(4, activation=tf.nn.relu, trainable=False)

    def call(self, inputs): 
        # type(inputs): <class 'tensorflow.python.framework.ops.Tensor'>
        if self.label_index is None:
            # How can I grab each input & average the values along the time?

            # Working but not delivering the results
            x = self.a_model(inputs)
            result = tf.keras.layers.Average()([x])
            
            # The **pseudocode** where each inputs is averaged in time dimension
            # average_time_dim = inputs[:, np.mean(:, axis=0), :] # SyntaxError
            # possible shape of average_time_dim: (4,)
            # average_reshaped = average_time_dim[tf.newaxis, tf.newaxis, :]
            # return tf.tile(average_reshaped, [1, self.out_steps, 1])

            return result

        # TODO. I would just have to pick a part of the resul
        # result = inputs[:, :, self.label_index]
        # return result[:, :, tf.newaxis]

In the pseudocode, def call(): would calculate the average of the inputs along the time axis and then tf.tile the average OUT_STEPS times.

I also have tried with tf.mean_reduce() and tf.mean() but I guess I am new in Tensorflow to make a calculation working inside an iterator.

The desired results should be such that once I compile and evaluate the baseline model:

baseline_model = MyAverageBaseline()
baseline_model.compile(loss=tf.keras.losses.MeanSquaredError(),
                metrics=[tf.keras.metrics.MeanAbsoluteError()])
evaluation = baseline_model.evaluate(input_dataset, verbose=2)
predictions = baseline_model.predict(input_dataset)

The predicted values would be the same as the results below, taking into account that seed=0 to generate the random df.

>>> print(predictions) 
>>> array([[[0.7, 0.6, 0.3, 0.35],
        [0.7, 0.6, 0.3, 0.35]],

        [[0.65, 0.9, 0.7, 0.35],
        [0.65, 0.9, 0.7, 0.35]],

        ...  More data here  ]])

Could somebody help me to figure out the code that the def call(self, inputs): method should include in the class MyAverageBaseline(tf.keras.Model):?

Answer 1

After a couple of days, I could manage to solve the issue with the call () method. Using tf.reduce_mean was the good way, but I had to fix the dimensionality of the results. The extra code added to the baseline model was.

out_steps = inputs. Shape[1]
averages = tf.reduce_mean(inputs, axis=1)
averages = averages[:,tf.newaxis,:]
result = tf.tile(averages, [1, out_steps, 1])

if self.label_index is None:
    return result