Efforts continue in integration and optimization of foreground extraction and super resolution modules for an end-to-end workflow of AdMiRe

After their delivery and based on initial feedback received from end user partners in the project, EPFL and Brainstorm continued work to integrate and optimize the foreground and super resolution modules in an end-to-end system which serves as the backbone of upcoming validation tests planned during the last months of the project.

The following summarizes the main achievements and ongoing efforts in this direction.

In order to maintain the streaming and processing of the end-to-end system under certain conditions such as limitation on available bandwidth and its fluctuation, the streaming modules adapts the frame resolution automatically, posing a challenge to foreground extraction and super resolution modules. To cope with this, the input interface and themodules were adapted to better perform in situations where video resolutions change. Hence, the updated modules are capable of automatically adapt to the changing video resolution.

Some end users reported low quality results as the output of foreground extraction module in certain situation due to excessive compression of the output video. This issue which was also resolved with an updated release of foreground extraction and super resolution modules. After a series of experimentations, and considering processing speed of CPU, the video frame rate were restricted to 30 frames per second in order to guarantee the entire workflow remains robust in a larger variety of configurations and hardware platforms, while guaranteeing real-time performance.

Furthermore, most appropriate video coding formats and encoding parameter for the output of the video stream was identified in order to offer good quality results under constraints. Further optimization work is in progress on both foreground extraction and super resolution modules and their sequential pipeline connection in the workflow and in real-time.

Finally, in order to assist operators, a graphical user interface is under development to offer easier control to operators in selection of most optimal parameters in specific situations and configurations. Such an interface will allow operators to select which video effects is needed, and the corresponding parameter such as upscaling parameter in super resolution through a user-friendly communication between server and clients including selection of IP address and port number.