Learning-based super resolution methods are becoming part of popular tools used in many imaging applications. The main driver behind their use is that they allow for an increase in the definition of details in content captured by a camera, beyond what its native capabilities can offer. Most such learning-based have been developed and trained by making use of uncompressed or mildly compressed contents. In all AdMiRe use cases, the video of remote participants is captured by a consumer grade camera (smart phone, webcam, etc.) and compressed before being streamed to the studio, to be edited in real-time and merged to other content produced in a professional environment. Therefore, not only super resolution can help in improving the definition of the video to match it with more professional capture devices, but also it must consider that compression artifacts might exist in the content because of limited bandwidth available to most consumers that prohibits streaming uncompressed or mildly compressed video.
In the framework of AdMiRe, EPFL has explored the impact of compression in the quality of the content after compression and decompression as a function of the coding bit rate and has designed and proposed an innovative solution that allows for super resolution to be applied in the compressed domain of a learning-based compression scheme that expects such a post-processing after decompression. Results show that better super resolution could be obtained using this new approach in the case of very low bit rate communication channels where a conventional super resolution is the operation used.
The details of the above-mentioned solution were submitted as a paper to the Applications of Digital Image Processing conference of the SPIE Optics & Photonics Symposium and accepted for publication and presentation at the conference in August 2021. It can be consulted in open access from: https://infoscience.epfl.ch/record/288672?&ln=en
Qualitative results of compressed-domain super resolution. From left to right: bicubic 4x-upscaling performed on a fully decompressed low resolution image, Lanczos 4x-upscaling performed on a fully decompressed low resolution image, compressed-domain super resolution (CDSR) applied to the latent space of a leaning-based codec. In all three cases, the image was compressed with bmshj2018-hyperprior at a quality level 6 using pre-trained model provided by the authors of the codec
Post written by EPFL