HLSL mul and D3DXMATRIX order mismatch

Question

I'm trying to multiply the transformation matrix in shader with vectors directly without doing unnecessary transportation. According to HLSL's mul documentation:

mul(x, y) Multiplies x and y using matrix math. The inner dimension x-columns and y-rows must be equal.

x [in] The x input value. If x is a vector, it treated as a row vector.

y [in] The y input value. If y is a vector, it treated as a column vector.

I have in the C++ code:

const D3DXMATRIX viewProjection = view * projection;
...
const D3DXMATRIX modelViewProjection = model * viewProjection;

where modelViewProjection is row-major order matrix that is copied to a constant buffer, not transposed. However, for this to work in the HLSL i need to multiply the transformation matrix with the position vector as:

output.position = mul(transformation, position);

which is the opposite of what the mul documentation is saying.

Can someone explain where is the mismatch here?

Answer 1

The deprecated D3DXMath library and the more modern DirectXMath use row-major matrix order. The HLSL language defaults to using column-major matrix order as it's slightly more efficient for multiplies. Therefore, most use of setting constant buffer constants will transpose matrix data. In almost all cases, any 'cost' of transposing the matrix here is completely hidden by all the other latencies in the system.

You can of course tell HLSL to use row-major matrix order instead, which means the HLSL mul needs to do an extra instruction on every vertex which is why it's usually worth doing the transpose on the CPU once per update instead.

See MSDN