Overlapping Custom Depth Stencils
In this post I’ll assume that you are already familiar with UE4 editor basics, material basics, scene depth and post process materials. If you’re not, please do some research and experiments first.
Basics
If you’re already familiar with using Stencils in materials, simply skip this section and jump to the second part.
In Post Process materials it’s really useful to be able to distinguish between different objects for multiple reasons: highlighting, outlines, etc. The solution for this in UE4 is provided by a Stencil buffer in the Custom Depth pass. Let’s see how to set up such workflow, that should clear most things up.
First, you have to enable the feature in Project Settings → Engine → Rendering → Postprocessing → Custom Depth-Stencil Pass:
You probably need to restart the editor to take effect. Then, go to the details of your component, and in the extended section of your Rendering group, enable the Custom Depth Pass and set a Stencil value with which you’d like to identify your component. For this example, I set it to 200 (randomly, you can use any number between 0 and 255).
Now, if you place an actor with this component in the editor, and you enable the Custom Stencil visualization mode, you should see your component, and nothing else.
You should see it even if there is something else in the background, and even if there is a wall in front of it: the point of custom depth is, that the actual “depth” (distance from the camera) of the object does not affect the result. If you copy this actor, and add another one with another stencil, you should see them appear as well:
Great, so now we know that during gameplay, the stencil buffer is going to contain the objects you have set a stencil to! Accessing it in a Post Process material is really easy: create a SceneTexture node, and select CustomStencil.
If you simply use it directly as the Emissive Color and set this material as your Post Process material, you should see the buffer working.
Now, they are all white, because every pixel will have the value of the used stencil, in my case 200 and 201, both resulting in white when converted to a color (0 is black, 1 is white, everything larges than 1 is white). Of course we don’t want to use these values as colors, we merely did it to confirm that our buffer is what we expect it to be.
As an example, let’s try adding some white overlay on our boxes! This is how my material looks like:
I simply created a mask based on the stencil (200), and applied a white overlay to every object, which is marked with such stencil. As a result, I got this:
The left box has no stencil (but the result is the same even if it has a stencil of 201 or anything besides 200), the middle and the right one has a stencil of 200.
Certainly, you can use any math you want for creating a mask from the stencil buffer, you can filter stencils which are smaller than 55 or only select the odd ones, or… whatever you need.
Bit-writing stencil buffers
Now let’s move on to the actual topic of this post. If you were to place objects with different stencil behind each other, you’ll see that the one closer to the camera is going to supersede the ones behind it.
In many cases it’s alright (and even desired), however, what if you want to be able to control which stencil should overrule the other one? Or, what if you want to apply a red overlay to stencil 200, a green one to stencil 201, and you expect their combination (yellow) to be applied if both are present?It’s going to take slightly more work, but it’s definitely doable!
Let’s take a look at another Stencil property for Primitive Components: Custom Depth Stencil Write Mask.
Default is “All bits” for new projects, which results in the “common” stencil workflow: every pixel is going to have a value of 0–255, with the stencil applied closest to the camera. However, you can also kindly ask UE4 to, instead, use a specific bit of each pixel’s stencil value as the mask for each primitive component.
I’ll elaborate a bit on the technical side; it’s alright if you don’t understand it, you can still just use the end result. When using “All bits”, each pixel in the stencil texture is going to represent a scalar value between 0 and 255. For example, we can have the stencil value 200 assigned to a pixel, which is going to be represented with [11001000] in binary format. Now, instead, you can say “for this object, use the 1st bit of the stencil value, for that one, use the 7th”, so for pixels with the first object visible its binary value will be [00000001], for pixels with the 7th component visible it’ll be [01000000], and, intuitively, for pixels in which both are visible, it’s gonna be [01000001]. Of course, these numbers also have decimal representations (1, 64 and 65, respectively), but in this case, we are not going to use them literally, we only care about which bit is set, and which is not. Using this method, we will be able to store multiple stencils per pixel.
Even if it’s totally unclear for you what the hell I was talking about, the important thing to know is:
- When using the default mask, you have 255 stencil values, from which only the closest one is known, per-pixel
- When using the bit-based write mask, you have only 8 stencil values you can use, but any amount of them can be active on each pixel
Awesome, let’s see how can we use these bit-based stencils in practice!
First, update all your components to use a bit write mask (you can’t use the two methods in parallel, for obvious reasons).
It’s important to keep in mind, even though you’re using bits, you can’t leave the stencil value at 0: set it to the same number your bit represents (see the drop-down in the image)! Now let’s check the buffer visualization:
Awesome, they are added together as expected! We can finally work with stencil intersections. However, the material part is going to take slightly more work, as there isn’t a nice built-in way to handle these types of stencils. There are two ways: building a kinda complicated and wasting material function, or by using HLSL (High Level Shading Language). Let’s see the graph first:
So what happens here is that we expect two inputs (the stencil buffer and the bit between 1 and 8), and we check if a pixel value, divided by the number, represented by the previous bit is odd or even. Not extremely trivial to understand, but works nevertheless.
The other way should be more performant, but you’re going to need a Custom node for it:
return ((1 << ((int)Bit - 1)) & (int)Mask) != 0;And that’s all. If you put any of these solutions in a BitTest material function, you can use them like so:
Now create two white boxes, one using the 1st bit, the other one using the 7th bit (or whichever you are using in your material graph). You should see that it is going to result in what we wanted to achieve:
If you’re a technical person, you can implement all the bitwise operators (NOT, OR, XOR, etc) in a similar way. For example, if you also wanted the ability to check if a pixel has ONLY a specific stencil, you could extend the graph like the following:
return (float)((1 << ((int)Bit - 1)) == (int)Mask);I know I didn’t really go deep into separate topics, I really wanted to avoid a 300 page book for solving this specific problem. I do hope though that the overall solution is clear and you’ll be able to create some beautiful materials with this (hopefully) new knowledge. If there is any explanation that you really miss or you found a mistake, let me know in the comments!
