System and method for 3D telepresence

• US 10,701,320 B2
• Filed: 12/27/2018
• Issued: 06/30/2020
• Est. Priority Date: 11/14/2014
• Status: Active Grant

• Current Assignee: Interdigital Vc Holdings, Inc. (InterDigital, Inc.)
• Original Assignee: PCMS Holdings Incorporated (InterDigital, Inc.)
• Inventors: Valli, Seppo T.
• Primary Examiner(s): Nguyen, Phung-Hoang J
• Application Number:
  US16/233,828
• Publication Number:
  US 20190132554A1
• Time in Patent Office: 551 Days
• CPC Class Codes: G03B 21/00 Projectors or projection-type viewers; Accessories therefor devices for changing pictures G03B23/00
• CPC Class Codes: G06T 15/005 General purpose rendering architectures
• CPC Class Codes: G06T 2215/16 Using real world measurements to influence rendering
• CPC Class Codes: H04N 7/157 defining a virtual conference space and using avatars or agents computer conference optimisation or adaptation H04L12/1827

Abstract

Systems and methods are described that enable a 3D telepresence. In an exemplary method, a 3D image stream is generated of a first participant in a virtual meeting. A virtual meeting room is generated. The virtual meeting room includes a virtual window, and the 3D image stream is reconstructed in the virtual window. The first participant thus appears as a 3D presence within the virtual window. The virtual meeting room may also include virtual windows providing 3D views of other participants in the virtual meeting and may further include avatars of other meeting participants and/or of the first meeting participant.

Claims

1. A method comprising:

receiving a 3D-data stream depicting a first participant and a live environment of the first participant;

rendering a common virtual meeting space having a virtual window for display of the first participant;

rendering a first perspective projection stream of the first participant and the live environment of the first participant from the received 3D-data stream from a first virtual viewpoint within the common virtual meeting space using depth cues derived from the received 3D-data stream to enable perspective changes as a function of the first virtual viewpoint when altered by rotating the first virtual viewpoint about a spatial axis perpendicular to a floor;

displaying the rendered first perspective projection stream of the first participant and the live environment of the first participant in the virtual window for the first virtual viewpoint;

rendering a second perspective projection stream of the first participant and the live environment from the received 3D-data stream from a second virtual viewpoint within the common virtual meeting space; and

displaying the rendered second perspective projection stream in the virtual window for the second virtual viewpoint.

2. The method of claim 1, wherein:

the first perspective projection stream is displayed at a first time for a second participant; and

the second perspective projection stream is displayed at a second time for the second participant.

3. The method of claim 2, wherein the second participant is depicted as an avatar in the virtual meeting space.

4. The method of claim 2, wherein the second participant is depicted in a second virtual window in the virtual meeting space.

5. The method of claim 1, further comprising selecting between the first and second virtual viewpoints based on an input from at least one of the first participant and a second participant.

6. The method of claim 1, wherein:

the rendered first perspective projection stream is displayed for a second participant; and

the rendered second projection stream is displayed for a third participant.

7. The method of claim 6, wherein:

the first perspective projection stream is displayed at a first time; and

the second perspective projection stream is displayed at a second time.

8. The method of claim 6, wherein:

the first perspective projection stream is displayed at a first time; and

the second perspective projection stream is displayed at the first time.

9. The method of claim 1, wherein the 3D-data stream depicting the first participant is a live video feed of the participant in a remote meeting site.

10. The method of claim 1, wherein the 3D-data stream is a rendered avatar of the first participant.

11. An apparatus comprising:

a processor configured to receive a 3D-data stream depicting a first participant, and to generate first and second perspective projection streams based respectively on first and second virtual viewpoints in a common virtual meeting space;

a rendering system configured to render first and second perspectives of the virtual meeting space, the first and second perspectives respectively displaying the first and second perspective projection streams in a virtual window in the virtual meeting space using depth cues derived from the received 3D-data stream to enable perspective changes as a function of the first and second virtual viewpoints wherein each of the first and second virtual viewpoints is altered by rotating about a respective spatial axis perpendicular to a floor in the virtual meeting space; and

a transmission unit configured to transmit the rendered first and second perspectives for display.

12. The apparatus of claim 11, wherein the transmission unit is further configured to:

transmit the first perspective to a second participant at a first time; and

transmit the second perspective to the second participant at a second time.

13. The apparatus of claim 12, wherein the second participant is depicted as an avatar in the virtual meeting space.

14. The apparatus of claim 12, wherein the second participant is depicted in a second virtual window in the virtual meeting space.

15. The apparatus of claim 11, wherein the transmission unit is further configured to select between the first and second virtual viewpoints based on an input from at least one of the first participant and a second participant.

16. The apparatus of claim 11, wherein the transmission unit is further configured to:

transmit the first perspective projection stream to a second participant; and

transmit the second perspective projection stream to a third participant.

17. The apparatus of claim 16, wherein:

the first perspective projection stream is transmitted at a first time; and

the second perspective projection stream is transmitted at a second time.

18. The apparatus of claim 16, wherein:

the first perspective projection stream is transmitted at a first time; and

the second perspective projection stream is transmitted at the first time.

19. The apparatus of claim 11, wherein the 3D-data stream depicting the first participant is a live 3D-video feed of the first participant in a remote meeting site.