Render Mask bug ?

[AltoR]

Hi,

I recently tried the new feature called "Render Mask". I tried with a simple b&w jpg texture. So far it works, but the mask boundaries are awful, no matter how high are the render settings.
Thank you for helping me out,

[yavor.rubenov]

Hello,

Is it possible that there are not entirely black pixels in the mask image at the edges ? The render mask reads the texture and if it is not pure black - renders that pixel so you might get results like this.
Another thing - what image sampler are you using - there are some problems with the adaptive subdivision sampler that also produce some artifacts on mask edges.

Best regards,
Yavor

[AltoR]

The texture is highly contrasted with 2 pure black & white surfaces (see below a sample of the texture seen at real pixel size). I use the Adaptive image sampler, but the problem still occurs whatever image sampler I use.

[yavor.rubenov]

It would be best if you convert the image to pure black and white - in the sample for example there are pixels that have value like 2,2,2 RGB - which is still enough for the render mask to decide to render those pixels.

Also the mask image should be the same resolution as the output image.

[AltoR]

Thank you.
The texture mask already has the exact same resolution as the render.
I think that RGB values like 2,2,2 in the sample I provided are jpg compression artefacts. Whereas in the original full texture, I can only see pure 0,0,0 and 255,255,255 values.
Anyway, as you suggest, I converted the texture to b&w jpg but the problem remains the same (I tested with Fixed image sampler).

[yavor.rubenov]

Could you share the full mask texture then? (My email is yavor.rubenov@chaosgroup.com)

[AltoR]

Of course. I just send it. Thank you.

[yavor.rubenov]

Thanks for the image.

There are pixels with values above zero near the edges -

I've loaded the mask in the Vray VFB and used the Levels color correction to see the pixels.

I can think of two workarounds:
1. In the output section of the bitmap texture (or VrayHDRI) you can set some little negative RGB offset
2. Initially I saved the as monochrome bitmap with MSPaint - this cuts the pixels with very low values to 0. But it turns out that max bitmap texture can't load those bmps, so I had to save the monochrome bmp again to full 24bit color bmp and then it loads ok.

And for the future - we should probably think of some way to set a threshold value for the mask.

Best Regards,
Yavor

[Joelaff]

Couldn't you just threshold the image in Photoshop? Or convert to Black and White 1 bit and back?

Also, don't save the mask as a JPEG, or the JPEG will introduce its own artifacts. Use a TGA, or PNG or something.

[AltoR]

Thank you Yavor and Joelaff for your answers.
I fixed the problem by using a PNG mask instead of the JPG one. I have no more weird grey areas like before.
So in the present case, VRay Render Mask feature works fine for my purpose. But I think it would be best if the rendered mask were antialiased and this is not the case. Let's say you create a very accurate render mask in Photoshop, you then loose this accuracy in the rendering stage because of bad antialiasing. Maybe this could be fixed in the future with the threshold value ?

[Art48]

Uhm what do you think would that AA do?
Render half a pixel? Render a Pixel but with <1 opacity? In my mind this doesn't make any sense, correct me if I'm wrong.

[AltoR]

The goal of such a feature is quite simple to understand: if the render boundaries are not antialiased, when one export the render in a compositing software like Photoshop, then the render boundaries will not be neat. Just like any other mask created by VRay (Multimatte Elts, ObjectID etc ...) I think that render mask should come antialiased too.

[Art48]

Well if your goal is to composite that new rendering into the old one then youd expect the borders to be exactly the same as in the old rendering. Therefore it wouldn't matter if the pixels are antialiased there or not since they are exactly the same as the old ones either way.

[AltoR]

There is no old rendering: it is a photomontage. So the purpose is to directly paste the rendering onto the background. If the borders are not clean, VRay render mask feature is less interesting. It still does the job by not calculating the whole image (time saver), but I will still have to do a clean mask in Photoshop for the final image.

[Art48]

Ah okay, got it..
Well it should be relatively easy for the devs to simply use the render mask in a filtered state after rendering as alpha channel of the image.