How to set outline for stencil DirectX 11

Question

I'm pretty new to DirectX and graphics coding. I'm using SlimDX for implementation. I'm drawing a map that shows a visual representation of data for a particular area. All I want to do is take a state shape, and if a pixel is within that area, draw it, otherwise, don't. That's basically what a stencil is, right? But I can't figure out how I'm supposed to set that polygon for the stencil buffer. Or is that not what the stencil buffer is used for?

Answer 1

You're close... that's one of the situations the stencil buffer was created for. The only issue is that you'll need to set up your depth-stencil buffer to be a readable texture (to the GPU) if you want to do true stencil buffering here, since you'll need to read the stencil buffer to figure out what pixels need to be discarded. There's an "easier" way to do this, and that involves tricking the depth buffer into doing the read/compare/discard steps for us.

In normal 3D rendering, we draw all objects to the back buffer. When we do this, the GPU uses the depth stencil buffer to figure out which pixels are "behind" other, previously drawn, pixels. The farthest a pixel can be is at Z value 1.0f, and the closest it can be is at Z value 0.0f. What we can do is clear the depth stencil buffer to zero, set the depth comparison to always pass, and when we draw our shape, our vertex shader sets the Z value of the vertex to draw to W. When we end up doing the perspective divide (part of rasterization), we get a Z coordinate of 1.0f, the farthest it can possibly be. In our second rendering pass, rendering can proceed as normal, and the GPU will cut out the pixels of the drawn data that aren't in the area of the shape.

Put another, simpler way: In this method, we're effectively holding a piece of paper in front of the scene with a hole in it the exact shape of our stencil. When the actual scene gets drawn, it's drawn 'behind' the stencil in all places except where our shape is (the 'hole'), making an exact cutout. In normal 3D rendering, things typically fall between 0 and 1. All of those object will be drawn normally, but anything outside the area of the stencil will fail the depth test (because we set everything to zero except where the shape was) and be clipped.

This method requires two rendering passes, one where the stencil shape is rendered to the depth buffer, and the second to render the data color to the back buffer.

Note that this method will likely need tweaking if you plan on doing more than is in the description of the question. It works for simple scenarios, but for more complex scenes, you'll want to read up on the stencil buffer. This, while in C/C++, is an excellent overview on the topic, and is directly applicable to SlimDX, as SlimDX is merely a thin wrapper over the native C/C++ API. You can use this to cross reference the C/C++ objects and structures with the C# ones in SlimDX (look at the Unmanaged counterpart field in the class/structure documentation pages).

Additionally, this method will fail with scenes that are all rendered at the same depth level (though this shouldn't be done anyway because Z-fighting will be kicking your chair the whole way). A better method of doing 2D graphics in 3D is to place the components on different Z axis 'layers' and draw them in 3D with an orthographic projection (which prevents the shrinking of farther images due to perspective).

Finally, as I haven't written pixel shader code in some time, I don't quite remember if writing one to Z in a pixel shader will cause a one to be written to the depth buffer. I'm pretty sure it does, but I'd need to confirm that before I give it as solid advice. If so, then setting the Z value of the point in the pixel shader can replace the assignment of W to Z in the vertex shader.