CSIC and UPF have developed a prototype of the capture wizard that suggests to the user how to adjust the illumination on his/her side so as to optimize the visual appearance of the final, composited image that superimposes the user on the studio image.
CSIC has developed a style-transfer method that matches the colors and contrast of the user image to the studio image: this method is based on image statistics and geometry, and preliminary tests show that its results outperform the state of the art both from academia and the industry, including deep-learning approaches. The wizard makes an estimation of the user image quality after being color-matched to the studio by computing a quality score using a combination of the machine-learning-based metric VMAF (comparing the original user image with the studio-matched user image) and a no-reference structure metric SI (evaluating only the studio-matched user image). This score is computed for several versions of the user image, where the exposure is (artificially) modified.
The optimal exposure is found (i.e. the one providing the best quality score), and a suggestion is communicated to the user to increase or decrease the overall illumination.
Head Image: An image from CSIC and UPF tests, where we transform an underexposed user image with a color cast to match the studio appearance.
CSIC & UPF