Geometry-Driven Shadow Harmonisation for Composited Faces: A Multiplicative, Albedo-Preserving Relighting Pipeline
Authors & Institutions
Vijesh KP
Independent researcher (no institutional affiliation stated in the paper)
What Problem It Solves
The paper targets the narrow case where donor color and identity are acceptable but form shadows are missing. It asks how to inject nose, socket, lip, and cavity shadows from approximate face geometry without brightening pixels, changing chromaticity, or synthesizing a new face.
Key Result
On the paper's analytic face heightfield with a known light, the default setting modifies 56.5% of face pixels, gives mean gain 0.938 overall and 0.890 on modified pixels, and sends 3.4% of pixels to the 0.82 floor. It guarantees zero brightened pixels and no mathematical hue-angle shift beyond floating-point noise. These figures characterize the bounded operator rather than perceptual quality: the paper provides no paired real-composite benchmark or baseline comparison. It cannot add highlights, correct an already dark donor, remove opposing source light, or avoid double-shadowing residual donor shading.
Abstract
Face swapping and face compositing pipelines routinely produce a face that is geometrically well aligned but photometrically implausible: the donor face carries flat, near-frontal studio illumination while the host body and background carry directional scene light. Most existing remedies re-synthesise the face through colour transfer, neural relighting, or inverse rendering, and therefore risk altering identity, skin tone, and texture. We present a conservative alternative: geometry-driven form-shadow injection. The pipeline never repaints the face. It estimates a per-pixel gain field $g\in[g_{\min},1]$ from a rasterised 3D face proxy and multiplies it channel-uniformly onto linear RGB, so the operator can only darken and cannot shift chromaticity. A dense landmark mesh is rasterised into a depth buffer, from which we derive surface normals, a cavity term, and screen-space cast shadows. Key-light direction is estimated from host-side cues (body, background, hair halo); on-face cues are downweighted because they recover the donor's lighting. Shadow magnitude is not matched to the host: it is set by a three-parameter transfer $(τ,σ,g_{\min})$. The shading field is divided by its 75th percentile over skin, then gated, scaled, clamped, smoothed, and re-clipped inside a feathered, skin-gated face mask. On an analytic face heightfield, the default $(τ,σ,g_{\min})=(0.90,0.45,0.82)$ modifies 56.5% of face pixels with mean gain 0.938 (0.890 on modified pixels) and drives 3.4% of pixels to the floor. Hue invariance is a corollary of the operator. We analyse the transfer in closed form, ablate its parameters, and discuss failure modes of a monotone, darkening-only formulation, including double-shadowing of non-flat donors.
Research Starting Point
A swapped face can be geometrically aligned and identity-correct yet still look pasted on because the donor carries flat studio light while the host scene has a directional key. Generative relighting can repair more effects, but it may also change skin tone, pores, texture, or identity after the face has already been approved. Production teams sometimes need a deliberately limited operator whose worst-case photometric change is known.
Method
A face detector and 468-point mesh produce a rasterized depth buffer, surface normals, cavity proxy, and screen-space cast shadows in a 512-pixel crop. Host-side body, background, neck, and hair-halo cues vote for key-light direction while donor-face cues are downweighted. The normalized shading map passes through a three-parameter threshold, strength, and minimum-gain transfer, then guided smoothing and a feathered skin mask. The same scalar gain is multiplied into all linear-RGB channels and clipped to a range from 0.82 to 1.0.
Paper Summary
This is a conservative post-compositing control with an understandable damage bound, not a complete relighting system. It is attractive when identity preservation outranks dramatic realism, provided real shots are reviewed for wrong light direction and double shadows.