OpenGL selective offscreen render to one of 2 textures in one fragment shader

Question

I'm writing a glsl shader for rotation manipulator. I'm aiming at Autodesk Maya style, 3 axes for each of x y and z rotation and one axis over them for rotation along camera forward axis equals 4;

The user is able to limit the result of his manipulations to a particular axis by selecting it directly in the viewport.

In order to enable object selection, I'm using custrom framebuffer with 2 attached textures as render targets - one for color, one for selection IDs.

The problem is that having theese 4 axes drawn by GL_LINES mode arises a problem for a user. If the lines are drawn thin and nice, the user should click on them very precisely to activate the particular axis, because he has to select exacly the pixel with the proper object_id of the axis. I want to ease his life making 2 separate drawings. Thin lines for color output and 3 times wider lines for objectid output texture.

I'm a newbie to openGL. Now my naive logic is passing a boolean uniform that tells the fragment shader wheather to draw to color or to objectid buffer.

Here are the drawing commands with passed render_target selector

    GLfloat lw;
    glGetFloatv(GL_LINE_WIDTH,&lw);
    glUniform1ui(7,0);//render to RGB
    glDrawArrays(GL_POINTS,0,1);
    glUniform1ui(7,1);//render to OBJECTID
    glLineWidth(10);
    glDrawArrays(GL_POINTS,0,1);
    glLineWidth(lw);

And here is the fragment shader:

#version 450
//inputs
layout(location=6) uniform uint manipulator_selected_axis;
layout(location=7) uniform uint render_to_selection_buffer;

//outputs
layout(location=0) out vec4  out_color;
layout(location=1) out float out_objectid;

void main(void)     
{
    if (render_to_selection_buffer==1)
    {
        switch (gl_PrimitiveID)
        {
            case 0: //CAMERA FORWARD
                out_objectid=float(253)/256;
                break;
            case 1: //X 
                out_objectid=float(250)/256;
                break;
            case 2: //Y
                out_objectid=float(251)/256;
                break;
            case 3: //Z
                out_objectid=float(252)/256;
                break;
        }
        out_color=vec4(0,0,0,0);
    }
    else
    {
        vec4 active_axis_color=vec4(1.0,1.0,0.0,1.0);
        switch(gl_PrimitiveID)
        {
            case 0: //CAMERA FORWARD
                if(manipulator_selected_axis==0)
                    out_color=active_axis_color;
                else
                    out_color =vec4(1.0,0.5,1.0,1.0);
                break;
            case 1: //X
                if(manipulator_selected_axis==1)
                    out_color=active_axis_color;    
                else
                    out_color =vec4(1.0,0.0,0.0,1.0);
                break;
            case 2: //Y
                if(manipulator_selected_axis==2)
                    out_color=active_axis_color;    
                else
                    out_color =vec4(0.0,1.0,0.0,1.0);
                break;
            case 3: //Z
                if(manipulator_selected_axis==3)
                    out_color=active_axis_color;    
                else
                    out_color =vec4(0.0,0.0,1.0,1.0);
                break;
            case 4:
                out_color =vec4(0.6,0.6,0.6,1.0);
                break;
        }
        out_objectid=0;

    }
}

I really dont' like the logic that I'm using for that, because it looks ugly and unoptimized. First, because the shader has to run twice, and it has got a rather complex geometry shader, where I do circles generation. Second, because my fragment shader has lot's of if else stuff in there and I was told that gpu doesnt like branching.

So QUESTION 1 what is the alternative proper way to write such a shader that does the same? QUESTION 2 is more technical.

Here i provide a screenshot of what I get by theese 2 glDrawArrays call. As you can see I have black contours in my colorbuffer. The problem is that when the framgent shader executes the render code for out_objectid target, it still writes some dirty random info to the out_color like in the picture below, so I had to overrite it with out_color=vec4(0,0,0,0). At least it gives clean black.

So QUESTION 2. Where is my mistake? How do i prevent objectid block to write to out_color? Thanks UPDATE: Following what I was suggested I came to this kind of effect. I can't descibe what is happening in the insides of openGL. It looks like it still writes to GL_DEPTH_COMPONENT and makes some undefined depth tests that lead to such result. I dont want any depth writes during this draw call, because the manipulator is always on-top. Maybe you could give me additional advice.

Drawing code

glBindBuffer(GL_ARRAY_BUFFER,0);
glUseProgram(program_id_ctl_manip_color);

glEnable(GL_VERTEX_PROGRAM_POINT_SIZE);
glUniform4f(0,0,0,0,1);
glUniform1f(5,0.67);
glUniform1ui(6,manipulator_axis_highlight);
glUniformMatrix4fv(10,1,GL_TRUE,cam.matrix_view.m);
glUniformMatrix4fv(14,1,GL_TRUE,cam.matrix_projection_ortho.m);
glUniformMatrix4fv(18,1,GL_TRUE,cam.matrix_camera.m);
glDrawBuffer(GL_COLOR_ATTACHMENT0);
glDrawArrays(GL_POINTS,0,1);


GLfloat lw;
glGetFloatv(GL_LINE_WIDTH,&lw);
glLineWidth(6);
glUseProgram(program_id_ctl_manip_objectid);
glUniform4f(0,0,0,0,1);
glUniform1f(5,0.67);
glUniform1ui(6,manipulator_axis_highlight);
glUniformMatrix4fv(10,1,GL_TRUE,cam.matrix_view.m);
glUniformMatrix4fv(14,1,GL_TRUE,cam.matrix_projection_ortho.m);
glUniformMatrix4fv(18,1,GL_TRUE,cam.matrix_camera.m);
glDrawBuffer(GL_COLOR_ATTACHMENT1);
glDrawArrays(GL_POINTS,0,1);
GLenum buffers[2]={GL_COLOR_ATTACHMENT0,GL_COLOR_ATTACHMENT1};
glDrawBuffers(2,buffers);

glLineWidth(lw);

Answer 1

Let's get to question 2 first:

Where is my mistake? How do i prevent objectid block to write to out_color

GL simply does not work that way. The rasterizer produces fragments, and your fragment shader has to determine output values for all of the render targets you have attached - or it can discard the whole fragment, so that no oupt of it is ever written to the framebuffer. If you do not explicitely write to an output variable, the contents of it will be undefined - most likely you get some content which is in the particular registers from the last calculations, explaining the random noise in your last image. Writing it as black will of course lead to the first image.

You could do some things to prevent that black border, for example:

• enable alpha blending and write fully transpaernt pixels to your out_color.
• just set the draw buffer for your out_color output to GL_NONE

There are probably a lot of other strategies which might schieve the same result, but they all aren't really efficient, which brings us to question 1:what is the alternative proper way to write such a shader that does the same?

what is the alternative proper way to write such a shader that does the same?

There are different solutions of course. The straight-forward solution is to simply use two different shaders. You have to switch a uniform value between the draw calls anyway, so you can't use both modes in a single draw call. You can simply switch the shader, also.

The other issue is that innner switch. Don't do that. You can input the different colors and object IDs for the lines as vertex attributes. Having such a switch in the fragment shader is quite unusual and not really efficient. On the upside, you still have fully uniform control flow, so you will not see the worst case performance-wise, but I would still consider your construct as bad style at least.

You don't need multiple render targets for that. MRT only makes sense if you want the same fragments for all render targets, you just want to output more data than a vec4 per fragment, and do so in a single pass. But in your case, you want actually different geometry, and you are doing it multi-pass already, so you gain nothing. MRT just makes things even more complicated in your case.

Just use two shaders, and switch the shaders and the draw buffer (and only one draw buffer is needed) between them would be an easy solution.

There is another issue with your code: glLineWidth() with values besides 1.0 are deprecated. Your code will not work on modern GL core profiles. If you want wide lines, you should actually draw traingle-based primitives.