Would it be possible to output the "gaps" as a separate mask or fill them with a solid color instead of the current tiling modes? #58
Replies: 1 comment · 4 replies
-
|
Excellent question! 🙂 I've tried this in the past and couldn't get it just right, but also didn't try enough haha I know a way to quantify the gaps, which brings me back to good old calculus 3 days. We need to compute the gradient of each surface point (a normal vector with extra info), the more this vector is tilted to the sides, the more we're on a cliff in the scene Calculating the tilt is easy, it's the angle between the gradient and the scene's plane, but defining what are gaps based on the cliffs isn't inherently mathematical and subjective. We'll get a value between 0 and 1 (after normalization) telling how steep a given pixel takes place on the scene's surface, which has zero knowledge of the surroundings and could give false positives I'll give this a second chance in time, as I probably messed up the gradient calculation before |
Beta Was this translation helpful? Give feedback.
-
|
This seems quite accurate (?), would you say it works for the use case of inpainting? video.mp4I don't think I can do much better than this math-wise with my current shader experience, the artifacts are mostly due the relatively low resolution depth maps, and edges are.. complicated 😅 |
Beta Was this translation helpful? Give feedback.
-
|
Hello! Thank you for your great contribution to this project. I'm also very interested in extracting masks and performing inpainting-based restoration. I would like to ask how the mask detection is implemented in your work. Is there any related code available for reference? @Tremeschin |
Beta Was this translation helpful? Give feedback.
-
|
Surely have @KazusaKitakawa, it's a heuristic utilizing some calculus/geometry knowledge within the raymarching (1) (2) model of the project. I basically keep track of the previous and current depthmap heights to estimate a derivative value relative to the step size Combining the two, the further away the normal's angle is from the screen plane (I use Z=Forward), combined with how fast it changes, roughly gives me a "global" perceptual steepness value that depends on the surface max height Note: The projections are extremely non-linear, a step can't have knowledge of the neighbors distortions to use a simpler approach Then just about checking for some threshold and overriding the color as needed. No clue what the units are here! |
Beta Was this translation helpful? Give feedback.
-
|
Thanks a lot for your detailed reply! I’ll go through that part of the code. Great work! |
Beta Was this translation helpful? Give feedback.
Uh oh!
There was an error while loading. Please reload this page.
-
I have little knowledge on how shaders work, but I would like to use LaMa to inpaint the gaps instead of the current solutions with repeat/mirror.
So I was wondering if its feasible to output the frame with whatever tiling mode selected and a black/white mask depicting the gap.
Or directly fill the gaps in the frame with bright green or something instead of a tiling mode.
Beta Was this translation helpful? Give feedback.
All reactions