Getting incorrect vectors when trying to do mouse picking in OpenGL 3

Question

I am trying to get a direction vector to where my cursor is on the screen, however it gives me large values instead of the actual values. When doing mouse picking I am getting extremely small numbers for my world coordinates such as

Mouse is pointing at World X: 4.03225e-05 Y: -0.00048387 Z: -1

Am I doing something wrong here, I have adapted this code from here. I am using glm::intersectLineTriangle() to test if the line is pointing at the triangles world coordinates in question.

Here is the relevant code:

double mouse_x, mouse_y;
glfwGetCursorPos(GL::g_window, &mouse_x, &mouse_y);

int width, height;
glfwGetFramebufferSize(GL::g_window, &width, &height);

// these positions must be in range [-1, 1] (!!!), not [0, width] and [0, height]
float mouseX = (float)mouse_x / ((float)width  * 0.5f) - 1.0f;
float mouseY = (float)mouse_y / ((float)height * 0.5f) - 1.0f;

glm::mat4 invVP = glm::inverse(projection * view);
glm::vec4 screenPos = glm::vec4(mouseX, -mouseY, 1.0f, 1.0f);
glm::vec4 worldPos = invVP * screenPos;

glm::vec3 mouseClickVec = glm::normalize(glm::vec3(worldPos));

std::cout << "Mouse is pointing at World X: " << mouseClickVec.x <<  " Y: " << mouseClickVec.y << " Z: " << mouseClickVec.z << '\n';

glm::vec3 intersect;

if(glm::intersectLineTriangle(glm::vec3(0.0f, 0.0f, 0.0f), mouseClickVec, m_vertices_world[0], m_vertices_world[0], m_vertices_world[0], intersect))
{
 std::cout << "Intersect at X: " << intersect.x <<  " Y: " << intersect.y << " Z: " << intersect.z << '\n';
 setColor(glm::vec4(0.0f, 0.0f, 1.0f, 1.0f));
}
else
{
    setColor(glm::vec4(1.0f, 0.0f, 0.0f, 1.0f));
}

Answer 1

When using orthographic (parallel) projection, the point of view is not (0, 0, 0), (not in view space and of course not in world space). You have to create a ray from the near plane (-1) to the far plane (1):

glm::vec4 worldPosFar = invVP * glm::vec4(mouseX, -mouseY, 1.0f, 1.0f);
glm::vec3 mouseClickVecFar = glm::normalize(glm::vec3(worldPosFar));

glm::vec4 worldPosNear = invVP * glm::vec4(mouseX, -mouseY, -1.0f, 1.0f);
glm::vec3 mouseClickVecNear = glm::normalize(glm::vec3(worldPosNear));

if (glm::intersectLineTriangle(mouseClickVecNear, mouseClickVecFar,
    m_vertices_world[0], m_vertices_world[0], m_vertices_world[0], intersect))
{
    // [...]
}