D3D9 HLSL Matrixed My Brain

Question

I'm seeking help from all of you expert with Direct3D. Frankly I'm getting nuts by a really weird behavior from HLSL. It's the way I specify matrices.

Now my question D3DX 9 functions:

D3DXMatrixOrthoOffCenterRH and others...

Do they return a matrix that's used in this form: v' = v * M // Column major

or v' = M * v // Row major ???

I noticed the D3DXMatrixTrnaslate is in Column major and I assume the same for all functions.

Now does D3DX Project and Unproject functions takes in the same form returned by the above functions?

Have anyone experienced a really craziness from HLSL matrix multiplication?

Thanks.

Answer 1

I'll keep this brief. The RH functions return column-major (http://en.wikipedia.org/wiki/Column_vector, for ref.) matrices and are not the layout of choice for HLSL (or properly optimized assembly) shaders, since they can not perform a 3D transform of a vector using dot products. So use the LH or row-major (V'= M * V, indeed) layout or at least transpose before uploading to shader constant memory.

For a hands-on example take the DirectX documentation, look up D3DXMatrixLookAtLH() and D3DXMatrixLookAtRH(). On that page a layout of the matrix is given, you can compare the layouts, and see why dot4'ing each row with a X,Y,Z,1 vector performs the affine transform for X, Y, Z and W respectively (given that you use the LH/row-major matrix).