Having trouble training multidimensional data (4d numpy matrix) in sklearn and pybrain

Question

I have am training an emotion recognition system that detects emotions through facial movement and I have a 4 dimensional matrix that I am trying to train. However, sklearn is not allowing me to train the matrix. What I am doing is that I am trying to classify each emotion by creating a set of videos

Features that makes up the 4D matrix:
Number of videos (and each video will be assigned emotion label)
Number of frames per video
Direction of the facial landmarks per frame
Speed of the facial landmarks per frame

The actual feature matrix I am trying to train:
The left side is the speed (hypotenuse between same facial landmark each frame)
The right side is direction (arctan of the x and y values of the same facial landmark each frame)

>> main.shape
(2, 17, 68, 2)
# 2 videos, 17 frames per video, 68 facial landmarks, 2 features (direction and speed)
>> main
array([[[[ 0.        ,  0.        ],
         [ 0.        ,  0.        ],
         [ 0.        ,  0.        ],
         ..., 
         [ 0.        ,  0.        ],
         [ 0.        ,  0.        ],
         [ 0.        ,  0.        ]],

        [[-1.        , -0.        ],
         [-1.        , -0.        ],
         [-1.        , -0.        ],
         ..., 
         [ 0.        ,  0.        ],
         [ 1.41421356,  0.78539816],
         [ 1.        ,  0.        ]],

        [[ 3.16227766, -1.24904577],
         [ 2.23606798, -1.10714872],
         [ 2.23606798, -1.10714872],
         ..., 
         [ 1.41421356, -0.78539816],
         [-1.41421356,  0.78539816],
         [-2.        , -0.        ]],

        ..., 
        [[-3.60555128, -0.5880026 ],
         [-2.23606798, -0.46364761],
         [-2.23606798, -0.46364761],
         ..., 
         [ 3.        ,  0.        ],
         [ 2.23606798,  0.46364761],
         [ 0.        ,  0.        ]],

        [[ 2.82842712, -0.78539816],
         [ 2.23606798, -1.10714872],
         [ 2.23606798, -1.10714872],
         ..., 
         [ 1.41421356,  0.78539816],
         [ 1.        ,  1.        ],
         [ 1.        ,  1.        ]],

        [[-1.41421356,  0.78539816],
         [ 0.        ,  0.        ],
         [ 0.        ,  0.        ],
         ..., 
         [-2.23606798,  1.10714872],
         [-2.23606798,  1.10714872],
         [ 2.        ,  1.        ]]],


       [[[ 0.        ,  0.        ],
         [ 0.        ,  0.        ],
         [ 0.        ,  0.        ],
         ..., 
         [ 0.        ,  0.        ],
         [ 0.        ,  0.        ],
         [ 0.        ,  0.        ]],

        [[-1.        , -0.        ],
         [-1.        , -0.        ],
         [-1.        , -0.        ],
         ..., 
         [ 0.        ,  0.        ],
         [ 1.41421356,  0.78539816],
         [ 1.        ,  0.        ]],

        [[ 3.16227766, -1.24904577],
         [ 2.23606798, -1.10714872],
         [ 2.23606798, -1.10714872],
         ..., 
         [ 1.41421356, -0.78539816],
         [-1.41421356,  0.78539816],
         [-2.        , -0.        ]],

        ..., 
        [[-3.60555128, -0.5880026 ],
         [-2.23606798, -0.46364761],
         [-2.23606798, -0.46364761],
         ..., 
         [ 3.        ,  0.        ],
         [ 2.23606798,  0.46364761],
         [ 0.        ,  0.        ]],

        [[ 2.82842712, -0.78539816],
         [ 2.23606798, -1.10714872],
         [ 2.23606798, -1.10714872],
         ..., 
         [ 1.41421356,  0.78539816],
         [ 1.        ,  1.        ],
         [ 1.        ,  1.        ]],

        [[-1.41421356,  0.78539816],
         [ 0.        ,  0.        ],
         [ 0.        ,  0.        ],
         ..., 
         [-2.23606798,  1.10714872],
         [-2.23606798,  1.10714872],
         [ 2.        ,  1.        ]]]])

Error I got from sklearn SVM:

ValueError: Buffer has wrong number of dimensions (expected 2, got 4)

I then tried to play around with a more simple 3D matrix to see if I can find a way around it, but it gave me the same error:

>>> from sklearn.svm import SVC
>>> model = SVC()
>>> features = np.array([[[1,2],[3,4]],[[8,9],[10,11]]])
>>> features.shape
(2, 2, 2)
>>> labels = [1,2]
>>> model.fit(features,labels)
ValueError: Buffer has wrong number of dimensions (expected 2, got 3)

And other sklearn algorithms (like Naive Bayes) give me similar errors saying that they only accept 2D matricies.

I figured that the problem was that not really with my implementation, but that SVM, Naive Bayes, and other ml algorithms in general cannot handle more than 2 dimensions of data.

I tried turning to neural networks in pybrain but I could not figure out a way to feed the matrix into a pybrain neural network.

I though of the possibility of using PCA or reconstructing the design of the matrix, but I would prefer to keep these features and the same dimensions and not mess around with them.

Is there any way, machine learning algorithm, or neural network that I can use in order to train a 4D matrix?

Answer 1

It is unclear from your example which is the sample axis. But training in sklearn can be done like this (if the first axis is your sample axis):

model.fit(features.reshape(len(features), -1), labels)

There is no benefit in knowing the 4D structure for any sklearn classifiers. The adopted standard throughout the package, to ensure simplicity and clarity, is to have the first matrix dimension enumerate the examples and the second one the features. So in your case, you need to ravel the 3 remaining feature dimensions into one long vector.