First-ever Volumetric Video Call Conducted over a Public 5G Infrastructure
At VRDays Europe 2020, Dutch research organization CWI and Sound, a partner in the project, presented VRTogether, showcasing the world's first live 3D video conference utilizing point clouds over a commercial 5G network. The demonstration highlighted the platform's potential in emergency healthcare services.
The live demo featured a doctor in a medical examination room communicating with an (acting) injured patient outside in the street through volumetric video conferencing based on point clouds. The doctor was captured using Azure Kinect cameras, and the experience was projected on a big screen in real-time. The patient, located near Science Park in Amsterdam, joined the session via a Samsung S20 Ultra 5G smartphone, streaming in real-time over KPN's standard 5G network.
The capture application on the phone utilized both depth and color cameras to capture depth and color frames, which were then transformed into a point cloud. For the demo, a depth resolution of 240×180 pixels produced an average of 20,000 points per frame, streaming at 20fps resulting in 51.2 Mbits/second.
VRTogether was used to orchestrate the experience, offering an end-to-end pipeline for the delivery of volumetric video as point clouds. The platform provides a real-time experience, allowing for exploration of new communication and collaboration methods using volumetric video. It also features optimization mechanisms based on interaction and human behavior context, as well as extensibility for customizing capturing, compression, delivery, and rendering components according to interaction needs.
Modern volumetric video conferencing, such as the one demonstrated by VRTogether, utilizes AI-powered models to create lifelike 3D visualizations from standard 2D video. This enables participants to interact from various angles while employing light field displays for natural eye contact, spatial audio for enhanced immersion, and cloud-based infrastructures for seamless integration with existing workflows. Real-time compression, like the one in VRTogether, helps ensure smooth, high-fidelity streaming even over standard enterprise networks. In addition, cross-lingual collaboration features allow for real-time AI-powered speech translation.
While there is no widely recognized VRTogether platform specifically for volumetric video conferencing, the information provided suggests that it shares similar features with advanced compression for point cloud data, aligning with technology described for Google Beam and related systems. The award-winning compression algorithm for point clouds used by VRTogether is believed to be MPEG's V-PCC, known for its efficiency and industry recognition.
Data-and-cloud-computing technology played a significant role in the live demo of VRTogether, with the patient's phone utilizing cloud-based infrastructure for real-time compression, ensuring smooth streaming of the volumetric video conference. The VRTogether platform, featuring optimization mechanisms and extensibility for customizing capturing, compression, delivery, and rendering components, aligns with advanced data-and-cloud-computing technologies like Google Beam.
