A Wizard has been developed as part of the mobile app to enable user teleportation to the studio, which is one of the goals of AdMiRe.The Wizard is part of a user interface based on a Lobby metaphor.

As indicated in a previous post, the Wizard should assist the user in teleporting with the highest possible quality while taking into account the limitations of their devices. The Wizard has functionalities to notify users on how to improve certain aspects of the scene  (such as light, background, etc.), their pose in the video. It can also propose some image processing techniques such as basic filters or reposition of lights in order to match the studio situation.

AdMiRe’s mobile app already contains a working version of the Wizard. In this post, we will discuss some aspects of its interface as well as the functionalities that are already available.

Device selection

The Wizard interface includes a section on device selection, resolution and testing the different available devices.

Battery check

Since streaming consumes a significant amount of battery power, a battery level and consumption rate check is included. The system decides at this step if the user is allowed to enter the stream.

User positioning in the frame advice based on machine learning

A smart positioning step is part of the Wizard. It is based on the machine learning based algorithm MediaPipe Face Detection, using TensorFlowJS on the web, including BlazeFace and making use of the mobile GPU for inference. Up to 6 faces with their corresponding landmarks can be detected. To help with positioning, this step of the Wizard warns the user if the body is too close to the top or sides of the frame.

Illumination assessment and colour matching

The following picture portrays the mock-up interface for illumination assessment which is currently being integrated in the app, as well as the super-resolution and foreground extraction functionalities. These functionalities have been discussed in previous posts.

 

From left to right: the devices step, battery step, positioning step and the illumination step.