METHOD, SYSTEM AND APPARATUS FOR CAPTURE-BASED IMMERSIVE TELEPRESENCE IN VIRTUAL ENVIRONMENT

• US 20160234475A1
• Filed: 09/17/2014
• Published: 08/11/2016
• Est. Priority Date: 09/17/2013
• Status: Active Grant

• Current Assignee: Ideaction Inc.
• Original Assignee: Société des Arts Technologiques
• Inventors: Courchesne, Luc, Roy, Bruno, Durand, Emmanuel, Wozniewski, Mike, Bouillot, Nicolas
• Granted Patent: US 10,602,121 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06V 20/653 by matching three-dimensional models, e.g. conformal mapping of Riemann surfaces
• CPC Class Codes: G06V 40/174 Facial expression recognition
• CPC Class Codes: G06V 40/176 Dynamic expression
• CPC Class Codes: G06V 40/28 Recognition of hand or arm movements, e.g. recognition of deaf sign language static hand signs G06V40/113
• CPC Class Codes: H04N 13/204 using stereoscopic image cameras stereoscopic photography G03B35/00
• CPC Class Codes: H04N 7/157 defining a virtual conference space and using avatars or agents computer conference optimisation or adaptation H04L12/1827
• CPC Class Codes: H04S 2400/15 Aspects of sound capture and related signal processing for recording or reproduction
• CPC Class Codes: H04S 7/303 Tracking of listener position or orientation

Abstract

An apparatus for capture-based telepresence in a virtual environment session, comprises 3D capture devices to continuously capture image data of a first user. A processing unit comprises: an image processor for continuously producing a first 3D representation of the first user based on the image data, the first 3D representation having at least punctually a user specific position in a virtual environment. A data transmitter transmits the first 3D representation for remote use in the virtual environment session. A data receiver continuously receives a second 3D representation of a second user, image data for the virtual environment, and at least punctually receiving user specific position of the second 3D representation relative to the virtual environment. A rendering engine outputs for display the 3D representation of the second user positioned relative to the first user as inserted in the virtual environment based on said user specific positions. A method for participating in a virtual environment session is also provided.

Claims

1. An apparatus for capture-based telepresence in a virtual environment session, comprising:

at least one three-dimensional (3D) capture device to continuously capture image data of a first user;

a processing unit comprising at least;

an image processor for continuously producing a first 3D representation of the first user based on the image data, the first 3D representation having at least punctually a user specific position in a virtual environment;

a data transmitter for transmitting the first 3D representation for at least a remote use in the virtual environment session;

a data receiver for continuously receiving a second 3D representation of a second user, image data for the virtual environment, and at least punctually receiving user specific position of the second 3D representation relative to the virtual environment; and

a rendering engine for outputting at least the 3D representation of the second user positioned relative to the first user as inserted in the virtual environment based on said user specific positions;

at least one display device for displaying the output.

2. The apparatus according to claim 1, wherein the at least one display device comprising an immersive image screen deployed in up to 360°

around the participant.

3. The apparatus according to claim 2, wherein the at least one display device is one of a hemispherical screen, a frusto-spherical screen, a cylindrical screen, a set of flatscreens, tablet and head mounted display with orientation tracking, with the first user physically located in a central portion of said at least one display device.

4. The apparatus according to claim 1, comprising three of the 3D capture device, with the first user physically located in a central portion relative to the 3D capture devices surrounding the first user.

5. The apparatus according to claim 1, further comprising:

an audio capture device to continuously capture audio data emitted at least by the first user and for transmission via the data transmitter for at least a remote use in the virtual environment session;

an audio driver for producing audio content of the second user received by the data receiver; and

speakers for outputting the audio content.

6. The apparatus according to claim 5, wherein the audio capture device, and the speakers are part of a multi-channel audio system deployed in up to 360°

around the participant.

7. The apparatus according to claim 1, further comprising

a command processor to receive commands from the first user and for transmission via the data transmitter for interfacing with the virtual environment session.

8. The apparatus according to claim 7, further comprising an interface for communicating with the command processor.

9. The apparatus according to claim 1, wherein the first 3D representation has at least punctually a user specific position and orientation in the virtual environment, and further wherein the data receiver receives at least punctually a user specific position and orientation of the second user, the rendering engine outputting the 3D representation of the second user oriented relative to the virtual environment.

10. The apparatus according to claim 9, wherein the rendering engine outputs a reference orientation landmark for calibration of the first user'"'"'s orientation in the virtual environment, and wherein the user specific orientation of the first user is based on the reference orientation landmark.

11. The apparatus according to claim 10, wherein the rendering engine outputs a mirror image of the processed 3D representation of the first user as the reference orientation landmark, the mirror image being orientable to set the user specific orientation of the first user.

12. The apparatus according to claim 1, wherein the at least one 3D capture device continuously captures a 3D point cloud of the first user.

13. The apparatus according to claim 12, wherein the at least one 3D capture device continuously captures the 3D point cloud of the first user with chromatic and luminance data.

14. A system for operating a virtual environment session, comprising:

at least two of the apparatus of claim 1; and

a virtual environment server comprising;

a virtual environment manager for providing the image data for the virtual environment and for managing a coordinate system of a virtual environment based on at least on the user specific position and orientation of the users in the virtual environment.

15. The system according to claim 14, wherein the virtual environment server further comprises an asset manager for recording and storing modifications to the virtual environment.

16. The system according to claim 14, wherein the virtual environment server further comprises a control unit for administrating virtual environment sessions, the administrating comprising at least one of login, troubleshooting, technical support, calibrating, events synchronisation, stream monitoring, access privileges/priority management.

17. A method for participating in a virtual environment session comprising

continuously receiving image data of a first user;

continuously producing a first 3D representation of the first user based on the image data, the first 3D representation having at least punctually a user specific position and orientation in a virtual environment;

transmitting the first 3D representation for at least a remote use in the virtual environment session;

continuously receiving a second 3D representation of a second user, image data for the virtual environment, and at least punctually receiving user specific position of the second 3D representation relative to the virtual environment; and

outputting for display at least the 3D representation of the second user positioned relative to the first user as inserted in the virtual environment based on said user specific positions.

21. The method according to claim 17, further comprising receiving at least punctually a user specific orientation of the second user, and outputting for display the 3D representation of the second user oriented relative to the virtual environment based on the user specific orientation of the second user.

22. The method according to claim 21, further comprising outputting a reference orientation landmark for calibration of the first user'"'"'s orientation in the virtual environment, the user specific orientation of the first user based on the reference orientation landmark.

23. The method according to claim 22, wherein outputting the reference orientation landmark comprises outputting a mirror image of the 3D representation of the first user as the reference orientation landmark, the mirror image being orientable to set the user specific orientation of the first user.

18-20. -20. (canceled)

24-29. -29. (canceled)