Reduce an embedding's dimensions using PCA

Question

I have 10 embedding vectors each of length 1536 that I wish to reduce the dimensions of using PCA. Should I pass these to the PrincipleComponentAnalysis.Learn and PrincipleComponentAnalysis.Transform methods as double[10][1536] or double[1536][10]?

Answer 1

PCA provides a wide range of applications, most importantly it rotates your high-dimensional data into a position where the first axis points into the most "important" direction.

In terms of data reduction, there are 3 key topics:

Dropping Needless Dimensions:

in case there are some unused dimensions, PCA allows you to identify them (eigenvalues are zero) and therefore use a low-dim representation of your data without any loss of quality but using less space. imagine 3 points in 5-dimensional space. obviously 3 points always lie in a 2D-plane. Therefore the first two axes (aka eigenvectors aka principal components returned by PCA span your 2D-plane, whereas all the other axes will have eigenvalue 0 - they are not required.

Compression:

Now imagine your 3 points lie almost in a line. This means, the second axis returned by PCA is less "important" to keep the characteristics of your data. You can only store the first axis and more or less, whatever you do with the simplified data will have a similiar output since the information you dropped is almost pointless. The eigenvalue of the second axis will be relatively small compared with the eigenvalue of the first axis. This property is especially usefull when it comes to displaying high-dimensional data, since a plot of the first two axes gives normally the best intuition for your data. There is a nice online tool to play around: https://biit.cs.ut.ee/clustvis/ Just enter some simple data to get used to the fundamental PCA-concepts.

Recreation of data:

Sometimes you want to store your data in the reduced form, but for processing, you like to get back the original (unrotated) data. In this case, you have to store the transformed data AND all the eigenvectors. In case you dropped some of the axes with small, but non-zero eigenvalues, the recreated data will not be 100% accurate. My favorite article to explain the basics of PCA application is https://de.wikipedia.org/wiki/Hauptkomponentenanalyse#Beispiele (unfortunately, it's not available in the english version)

Now back to your question: You want to run a single batch. Otherwise, the axes of your rotated data points into different directions for each batch - which does not help you in general. For your case, think about what you need at the end of the day in terms of the 3 topics above to decide which road to take. Good luck and have fun!