How do I cleanly split an object and its ground shadow in only one render pass?

[Laserschwert]

I am still trying to figure out a workflow to properly render an object and its shadow on the ground in a way that I a) only need to render once, b) can cleanly turn off the shadow, and c) have both elements on a transparent background.

Right now I always have to render twice, once with the object, while the (matte) ground plane is set to not affect the Alpha, and once with the ground affecting alpha, while the object is set to be invisible to the camera:



Simply using an ID matte to mask out the object doesn't work here, because the edge were the object and its shadow meet clearly has the problem of two anti-aliased edges being on top of another, resulting in a visible seam. So I was hoping that the VRayObjectSelect element would help me out here, as it does somehow create a non-premultiplied alpha for its object, which in theory should result in a clean edge. Unfortunately it's not so easy:



As you can see, while the teapot element does at least have a somewhat clean edge (it does look non-premultiplied, but not quite as clean), the ground shadow element doesn't follow suit. I would have assumed that both elements would "extend" their pixels outwards into "zero alpha space", so that the edge where both elements meet wouldn't suffer from a seam, but to no avail.

So, anybody have an idea how to properly set this up? Is this possible at all? Maybe setting the ground to being a matte object isn't the way to go? How exactly do I use the "filter"-output of the ObjectSelect element?

[nikolay.kusht]

Not sure if this would give you the desired results, but this is what we got with our colleagues. We generated an extra alpha channel for the objects in our test scene and later use it to mask them so you could have the ability to color correct them. The shadows have the original alpha of the image.

[Laserschwert]

Unfortunately the original alpha also contains the objects, and I can't cleanly remove them from the alpha.

[svetlozar.draganov]

There are other ways to extract the alpha of the object and then to subtract it from the original alpha.
One of those method is through ObjectSelect element, just switch to RAW mode and V-Ray will generate a separate alpha channel for object only.

Here is the way we did it in Nuke.
1.RGB: http://ftp.chaosgroup.com/support/sc...6_16-26-54.jpg
2.RGB alpha: http://ftp.chaosgroup.com/support/sc...6_16-27-19.jpg
3.ObjectSelect alpha: http://ftp.chaosgroup.com/support/sc...6_16-27-58.jpg
4.Merge RGB with Background: http://ftp.chaosgroup.com/support/sc...6_16-29-42.jpg
Here we replaced the original alpha in the RGB with ObjectSelect alpha, that why in the end result the shadows are missing.
5.Merge previous result with RGB again: http://ftp.chaosgroup.com/support/sc...6_16-34-37.jpg
Here the RGB element with original alpha is merged bellow, so only the shadow becomes visible.

This approach allows to control the shadows visibility and object appearance:
http://ftp.chaosgroup.com/support/sc...6_17-04-23.gif

[Morbid Angel]

svetlozar, while this method works it still produces a black edge around the object. Its not possible to extract a clean edge when object and alpha are in the same rgb buffer. What would be great is if there was a render element which would put the shadow into it without the object visible (but I would expect this would require some visibility override for that render element)

[svetlozar.draganov]

That's correct but the discrepancy is unnoticeable, I can't notice it without comparing with reference image, while the dark outline in the first post is clearly visible even without reference image.

[Morbid Angel]

The issue will be much worse when you apply this to the actual shot, for example object moving with motion blur, you will get notable edges there, some times, we get really pixel *ucked for things like this. It would be great if it was possible to compute the shadow but I imagine it would require rendering it twice...Anyway I lived with 2 stage approach for years not going to suffer if it does not happen

[svetlozar.draganov]

I'm not sure if it is technically possible to render separated shadows and object rendering element in a single pass and have the proper masks for both.
Is there another render engine that has that capability?

[Morbid Angel]

No, not from derriving element from primary rgba buffer That's why I said it could be a second render for a custom buffer, like for example extra tex, except with custom visibility.

[svetlozar.draganov]

Understood, that's for the additional details.

[Laserschwert]

Thinking back to the days of RPF/RLA and compositing in combustion, didn't the RPF format have something like a coverage element, which basically defined pixels values BEHIND objects? I think it was used for cleaner post-DOF effects. Of course, for stuff like transparent objects or motion blurred elements this wouldn't necessarily help.

But yeah, having a "invisible to camera" render element would be interesting (which basically does a separate rendering of only an include/exclude list, with everything else being "not visible to camera"), but then you would want render elements for those as well, and it opens a whole new can of worms.