Explaining LSTM keras with Eli5 library

Question

I'm trying to use Eli5 for explaining an LSTM keras model for time series prediction. The keras model receives as input an array with shape (nsamples, timesteps, nfeatures).

This is my code:

def baseline_model():
    model = Sequential()
    model.add(LSTM(32, input_shape=(X_train.shape[1], X_train.shape[2])))
    model.add(Dropout(0.2))
    model.add(Dense(1))
    model.compile(loss='logcosh', optimizer='adam')
return model

from keras.wrappers.scikit_learn import KerasClassifier, KerasRegressor
import eli5
from eli5.sklearn import PermutationImportance

my_model = KerasRegressor(build_fn= baseline_model, nb_epoch= 30,   batch_size= 32, verbose= False)

history = my_model.fit(X_train, y_train)

So far, everything is ok. The problem is when I execute the following line that launchs an error:

# X_train has a shape equal to (nsamples, timesteps, nfeatures) and y_train has a shape (nsamples)

perm = PermutationImportance(my_model, random_state=1).fit(X_train, y_train)

Error:

ValueError Traceback (most recent call last)
    in ()
    2 d2_train_dataset = X_train.reshape((nsamples, timesteps * features))
    3
    ----> 4 perm = PermutationImportance(my_model, random_state=1).fit(X_train, y_train)
    5 #eli5.show_weights(perm, feature_names = X.columns.tolist())

    ~/anaconda3/lib/python3.6/site-packages/eli5/sklearn/permutation_importance.py in fit(self, X, y, groups, **fit_params)
    183 self.estimator_.fit(X, y, **fit_params)
    184
    --> 185 X = check_array(X)
    186
    187 if self.cv not in (None, "prefit"):

    ~/anaconda3/lib/python3.6/site-packages/sklearn/utils/validation.py in check_array(array, accept_sparse, accept_large_sparse, dtype, order, copy, force_all_finite, ensure_2d, allow_nd, ensure_min_samples, ensure_min_features, warn_on_dtype, estimator)
    568 if not allow_nd and array.ndim >= 3:
    569 raise ValueError("Found array with dim %d. %s expected <= 2."
    --> 570 % (array.ndim, estimator_name))
    571 if force_all_finite:
    572 _assert_all_finite(array,

    ValueError: Found array with dim 3. Estimator expected <= 2.

What can I do to fix this error? How can I use eli5 with my LSTM Keras Model?