In modern OpenGL, how do I avoid overwriting data in FBO colour attachments with invalid fragments?

Question

I have a batched rendering pipeline that uploads a bunch of quads to OpenGL. These are rendered to an FBO. Naturally, one of the attachments is visual output and another is depth, allowing for post-processing etc. However, a third attachment is intended for storing object IDs for picking, in a similar style to that explained here.

There's a small catch. I would like some rendered items to be 'transparent' to picking. Essentially, some things (particles or visual decoration or whatever) are not pickable themselves, but should not prevent the picking of objects behind them.

How might this be accomplished? Web searches suggest that glAlphaFunc might once have fulfilled this role, as I could have packed the object id into a texture with a transparency channel, and then deleted any fragments which I set the transparency to 0. A bit fiddly, but workable. However, that's been depreciated.

The other common recommendation for similar problems is to use the discard keyword. This isn't workable because it deletes all output elements of the fragment, not just for a specified attachment.

What's the solution here? Is there something clever I can do with blend functions that I just can't see? I'm pretty flummoxed.

---

Fragment shader looks something like this:

#region Fragment

#version 440 core

layout(location = 0) out vec4 colour; // Visual
layout(location = 1) out uint interactLayer; // Picking

in vec2 texCoord;
in flat int texUnit;
in flat uint objectID;

uniform sampler2D[32] u_texData;

void main()
{
    vec4 c1 = fragmentColour * texture(u_texData[texUnit], texCoord);
    
    if (c1.w == 0.0f) { discard; }
    colour = c1;
    
    if (objectID == 0xffffffffu) {
        // Please don't write to interactLayer values,
        // but also please don't discard the 'colour' value.
    }
    else {
        interactLayer = objectID;
    }
}

Answer 1

Discarding a fragment is binary; it is either discarded (along with all its results) or it is accepted.

You also can't do blending to a non-normalized integer image format. Blending only works between types that are floating point (including normalized integers).

Your best bet is to just use a float image format for your index. Cast your integer to the shader, then employ blending to turn it on/off. This will require writing a vec4 in the shader so that you can write an appropriate alpha value. Note that just because you're writing a vec4 doesn't mean the image format must have 4 components; it can still have 1 component and blending ought to work (as long as you're only using the source alpha).

Answer 2

In case anyone comes across this later on, I solved this using glColorMaski, which simply makes it so certain color values for the given buffer cannot be written to.