Pixel history's tex after column is incorrect on Vulkan for multiple primitives touching a single fragment in a single draw call with framebuffers without a stencil

[w-pearson]

Description

I've been looking into implementing dual source blending in pixel history in Vulkan (#2461), and made some progress on it (I've successfully implemented swapping index 0 and 1). However, I noticed that the "tex after" column had incorrect results in it. After some investigation, I've determined that this is an issue with Vulkan pixel history in general, specifically related to framebuffers that don't have a stencil attachment.

I've come to this conclusion because it does work correctly in the VK_Pixel_History test (for the multisampled test, EID 59, at around 140, 140), and that test has a stencil buffer. Additionally, the Vulkan pixel history implementation uses the original framebuffer for post-modification, but still assumes it has a stencil. The Vulkan specification says (emphasis mine):

If the stencil test is not enabled, as specified by vkCmdSetStencilTestEnable or VkPipelineDepthStencilStateCreateInfo::stencilTestEnable, or if there is no stencil attachment, the coverage mask is unmodified by this operation.

i.e. at least as far as I can tell, all of the fragments pass the test. So for 3 triangles, instead of allowing only a fragment from the first triangle, then a fragment from the second triangle, then a fragment from the third triangle, it draws all three triangles three times. (This can also be seen if blending is enabled, though I don't have a great example of this set up.)

OpenGL is also broken in this case, but I haven't investigated what's going on there and it seems to be broken in a different way.

Steps to reproduce

The easiest way to reproduce is to use the following set of modifications to the renderdoc demos program, which change all of the "simple triangle" demos to instead draw 3 overlapping triangles in a single draw call:

Patch to demos to help with reproducing

diff --git a/util/test/demos/d3d11/d3d11_simple_triangle.cpp b/util/test/demos/d3d11/d3d11_simple_triangle.cpp
index 7a65c35c6..45c2daf8e 100644
--- a/util/test/demos/d3d11/d3d11_simple_triangle.cpp
+++ b/util/test/demos/d3d11/d3d11_simple_triangle.cpp
@@ -44,11 +44,11 @@ RD_TEST(D3D11_Simple_Triangle, D3D11GraphicsTest)
     ID3D11VertexShaderPtr vs = CreateVS(vsblob);
     ID3D11PixelShaderPtr ps = CreatePS(psblob);
 
-    ID3D11BufferPtr vb = MakeBuffer().Vertex().Mappable().Size(sizeof(DefaultTri));
+    ID3D11BufferPtr vb = MakeBuffer().Vertex().Mappable().Size(sizeof(ThreeTriangles));
 
     D3D11_MAPPED_SUBRESOURCE mapped = Map(vb, 0, D3D11_MAP_WRITE_DISCARD);
 
-    memcpy(mapped.pData, DefaultTri, sizeof(DefaultTri));
+    memcpy(mapped.pData, ThreeTriangles, sizeof(ThreeTriangles));
 
     ctx->Unmap(vb, 0);
 
@@ -77,7 +77,7 @@ RD_TEST(D3D11_Simple_Triangle, D3D11GraphicsTest)
 
       ctx->OMSetRenderTargets(1, &bbRTV.GetInterfacePtr(), NULL);
 
-      ctx->Draw(3, 0);
+      ctx->Draw(9, 0);
 
       Present();
     }
diff --git a/util/test/demos/d3d12/d3d12_simple_triangle.cpp b/util/test/demos/d3d12/d3d12_simple_triangle.cpp
index 1d221130b..bf817883a 100644
--- a/util/test/demos/d3d12/d3d12_simple_triangle.cpp
+++ b/util/test/demos/d3d12/d3d12_simple_triangle.cpp
@@ -39,7 +39,7 @@ RD_TEST(D3D12_Simple_Triangle, D3D12GraphicsTest)
     ID3DBlobPtr vsblob = Compile(D3DDefaultVertex, "main", "vs_4_0");
     ID3DBlobPtr psblob = Compile(D3DDefaultPixel, "main", "ps_4_0");
 
-    ID3D12ResourcePtr vb = MakeBuffer().Data(DefaultTri);
+    ID3D12ResourcePtr vb = MakeBuffer().Data(ThreeTriangles);
 
     ID3D12RootSignaturePtr sig = MakeSig({});
 
@@ -111,7 +111,7 @@ RD_TEST(D3D12_Simple_Triangle, D3D12GraphicsTest)
       OMSetRenderTargets(cmd, {rtv}, MakeDSV(dsvMStex).CreateCPU(0));
       OMSetRenderTargets(cmd, {rtv}, {});
 
-      cmd->DrawInstanced(3, 1, 0, 0);
+      cmd->DrawInstanced(9, 1, 0, 0);
 
       FinishUsingBackbuffer(cmd, D3D12_RESOURCE_STATE_RENDER_TARGET);
 
diff --git a/util/test/demos/gl/gl_simple_triangle.cpp b/util/test/demos/gl/gl_simple_triangle.cpp
index f6604aa41..83c9eb0c5 100644
--- a/util/test/demos/gl/gl_simple_triangle.cpp
+++ b/util/test/demos/gl/gl_simple_triangle.cpp
@@ -41,7 +41,7 @@ RD_TEST(GL_Simple_Triangle, OpenGLGraphicsTest)
 
     GLuint vb = MakeBuffer();
     glBindBuffer(GL_ARRAY_BUFFER, vb);
-    glBufferData(GL_ARRAY_BUFFER, sizeof(DefaultTri), DefaultTri, GL_STATIC_DRAW);
+    glBufferData(GL_ARRAY_BUFFER, sizeof(ThreeTriangles), ThreeTriangles, GL_STATIC_DRAW);
 
     ConfigureDefaultVAO();
 
@@ -79,7 +79,7 @@ RD_TEST(GL_Simple_Triangle, OpenGLGraphicsTest)
 
       glViewport(0, 0, GLsizei(screenWidth), GLsizei(screenHeight));
 
-      glDrawArrays(GL_TRIANGLES, 0, 3);
+      glDrawArrays(GL_TRIANGLES, 0, 9);
 
       Present();
     }
diff --git a/util/test/demos/test_common.cpp b/util/test/demos/test_common.cpp
index 0441aba17..0288ae324 100644
--- a/util/test/demos/test_common.cpp
+++ b/util/test/demos/test_common.cpp
@@ -33,6 +33,18 @@ const DefaultA2V DefaultTri[3] = {
     {Vec3f(0.5f, -0.5f, 0.0f), Vec4f(0.0f, 1.0f, 0.0f, 1.0f), Vec2f(1.0f, 0.0f)},
 };
 
+const DefaultA2V ThreeTriangles[9] = {
+    {Vec3f(-0.5f, -0.5f, 0.0f), Vec4f(1.0f, 0.0f, 0.0f, 1.0f), Vec2f(0.0f, 0.0f)},
+    {Vec3f(-0.5f, 0.5f, 0.0f), Vec4f(1.0f, 0.0f, 0.0f, 1.0f), Vec2f(0.0f, 1.0f)},
+    {Vec3f(0.5f, -0.5f, 0.0f), Vec4f(1.0f, 0.0f, 0.0f, 1.0f), Vec2f(1.0f, 0.0f)},
+    {Vec3f(-0.5f, -0.5f, 0.0f), Vec4f(0.0f, 1.0f, 0.0f, 1.0f), Vec2f(0.0f, 0.0f)},
+    {Vec3f(0.0f, 0.5f, 0.0f), Vec4f(0.0f, 1.0f, 0.0f, 1.0f), Vec2f(0.0f, 1.0f)},
+    {Vec3f(0.5f, -0.5f, 0.0f), Vec4f(0.0f, 1.0f, 0.0f, 1.0f), Vec2f(1.0f, 0.0f)},
+    {Vec3f(-0.5f, -0.5f, 0.0f), Vec4f(0.0f, 0.0f, 1.0f, 1.0f), Vec2f(0.0f, 0.0f)},
+    {Vec3f(0.5f, 0.5f, 0.0f), Vec4f(0.0f, 0.0f, 1.0f, 1.0f), Vec2f(0.0f, 1.0f)},
+    {Vec3f(0.5f, -0.5f, 0.0f), Vec4f(0.0f, 0.0f, 1.0f, 1.0f), Vec2f(1.0f, 0.0f)},
+};
+
 /* XPM */
 const char *SmileyTexture[63] = {
     /* columns rows colors chars-per-pixel */
diff --git a/util/test/demos/test_common.h b/util/test/demos/test_common.h
index 0de06eb4e..f89101b46 100644
--- a/util/test/demos/test_common.h
+++ b/util/test/demos/test_common.h
@@ -166,6 +166,7 @@ struct DefaultA2V
 };
 
 extern const DefaultA2V DefaultTri[3];
+extern const DefaultA2V ThreeTriangles[9];
 extern const char *SmileyTexture[63];
 
 struct Texture
diff --git a/util/test/demos/vk/vk_simple_triangle.cpp b/util/test/demos/vk/vk_simple_triangle.cpp
index f7f5ed4d1..93e7845d1 100644
--- a/util/test/demos/vk/vk_simple_triangle.cpp
+++ b/util/test/demos/vk/vk_simple_triangle.cpp
@@ -57,11 +57,11 @@ RD_TEST(VK_Simple_Triangle, VulkanGraphicsTest)
     VkPipeline pipe = createGraphicsPipeline(pipeCreateInfo);
 
     AllocatedBuffer vb(
-        this, vkh::BufferCreateInfo(sizeof(DefaultTri), VK_BUFFER_USAGE_VERTEX_BUFFER_BIT |
-                                                            VK_BUFFER_USAGE_TRANSFER_DST_BIT),
+        this, vkh::BufferCreateInfo(sizeof(ThreeTriangles), VK_BUFFER_USAGE_VERTEX_BUFFER_BIT |
+                                                                VK_BUFFER_USAGE_TRANSFER_DST_BIT),
         VmaAllocationCreateInfo({0, VMA_MEMORY_USAGE_CPU_TO_GPU}));
 
-    vb.upload(DefaultTri);
+    vb.upload(ThreeTriangles);
 
     AllocatedImage offimg(this, vkh::ImageCreateInfo(4, 4, 0, VK_FORMAT_R32G32B32A32_SFLOAT,
                                                      VK_IMAGE_USAGE_TRANSFER_DST_BIT),
@@ -113,7 +113,7 @@ RD_TEST(VK_Simple_Triangle, VulkanGraphicsTest)
       vkCmdSetViewport(cmd, 0, 1, &mainWindow->viewport);
       vkCmdSetScissor(cmd, 0, 1, &mainWindow->scissor);
       vkh::cmdBindVertexBuffers(cmd, 0, {vb.buffer}, {0});
-      vkCmdDraw(cmd, 3, 1, 0, 0);
+      vkCmdDraw(cmd, 9, 1, 0, 0);
 
       vkCmdEndRenderPass(cmd);
 

1. Launch the modified demos program in renderdoc.
2. Start the VK_Simple_Triangle test.
3. Make a capture.
4. Close the test.
5. In renderdoc, right-click at around (200, 200) (a location hit by all 3 triangles).
6. Select pixel history
7. Observe that while the "pixel out" column is correct for all three primitives, the "tex after" column is incorrect.

The same process can be repeated with GL_Simple_Triangle and D3D11_Simple_Triangle. Here are screenshots of the results:

Vulkan:

OpenGL:

D3D11 (correct):

Environment

• RenderDoc version: v1.25, v1.26
• Operating System: Windows 10
• Graphics API: Vulkan, OpenGL. Direct3D 11 gives correct results. Direct3D 12 does not support pixel history.
• GPU: Intel UHD Graphics

[baldurk]

I believe those commits should fix this on GL and Vulkan, at least for your test case provided. Adding more permutations of pixel history tests to make sure it works under different rendering setups is on my (very very long) list of tests to add.

[w-pearson]

Thanks, 64b511a fixes it for Vulkan for me.

OpenGL seems to still be broken. But it seems to be broken in general, not just in this case, so perhaps there's a separate Intel issue? I'll create a new report for that.

I notice you also added marker regions to GL pixel history (04a0fdb), and marker regions also exist for the Vulkan implementation. I assume this is some kind of debugging feature - is there any documentation on how to use it? (I tried launching renderdoc in renderdoc, which seemed like the obvious if a bit silly approach, but that didn't do anything.)

[baldurk]

Yes please open a new issue if there are other issues with GL's pixel history, it's working fine for me. Intel GL is a bit dodgy on windows so there may be something there.

Self-hosted capture is possible by renaming the 'renderdoc' and 'renderdoccmd' projects to 'rdocself' and 'rdocselfcmd' and capturing that way, since then it produces a differently named dll that can be capturing while main renderdoc dll is replaying. You'll get new menu items under tools for starting and ending the capture.