TELEPRESENCE OF MULTIPLE USERS IN INTERACTIVE VIRTUAL SPACE

• US 20180174347A1
• Filed: 12/20/2016
• Published: 06/21/2018
• Est. Priority Date: 12/20/2016
• Status: Active Grant

• Current Assignee: Sony Interactive Entertainment LLC (Sony Group Corp.)
• Original Assignee: Sony Interactive Entertainment LLC (Sony Group Corp.)
• Inventors: Chaney, Roxy
• Granted Patent: US 10,699,461 B2
• Time in Patent Office: Days
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/016 Input arrangements with force or tactile feedback as computer generated output to the user
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06F 3/1454 involving copying of the display data of a local workstation or window to a remote workstation or window so that an actual copy of the data is displayed simultaneously on two or more displays, e.g. teledisplay
• CPC Class Codes: G06T 13/40 of characters, e.g. humans, animals or virtual beings
• CPC Class Codes: G09G 2350/00 Solving problems of bandwidth in display systems
• CPC Class Codes: G09G 2354/00 Aspects of interface with display user
• CPC Class Codes: G09G 2370/02 Networking aspects
• CPC Class Codes: G09G 2370/22 Detection of presence or absence of input display information or of connection or disconnection of a corresponding information source
• CPC Class Codes: G09G 5/12 Synchronisation between the display unit and other units, e.g. other display units, video-disc players
• CPC Class Codes: G09G 5/377 for mixing or overlaying two or more graphic patterns G09G5/02, G09G5/397 take precedence
• CPC Class Codes: H04N 7/157 defining a virtual conference space and using avatars or agents computer conference optimisation or adaptation H04L12/1827
• CPC Class Codes: H04N 9/3179 Video signal processing therefor
• CPC Class Codes: H04N 9/3194 including sensor feedback

Abstract

A telepresence communication uses information captured by a first capture device about a first user and information captured by a second capture device about a second user to generate a first avatar corresponding to the first user and a second avatar corresponding to the second user. A scene can be rendered locally or by a remote server in which the first avatar and the second avatar are both rendered in a virtual space. The first avatar is rendered to move based on movements made by the first user as captured by the first capture device, and the second avatar is rendered to move based on movements made by the second user as captured by the second capture device. The avatars may be realistic, based on avatar templates, or some combination thereof. The rendered scene may include virtual interactive objects that the avatars can interact with.

Claims

1. A method for telepresence communication, the method comprising:

receiving a first visual dataset captured by a first capture device;

generating a skeleton based on the first visual dataset corresponding to a first user as captured by the first capture device;

generating a first avatar a representative of the first user by applying a first three-dimensional model to the skeleton and a first surface texture to the first three-dimensional model;

receiving data corresponding to a second avatar representative of a second user, wherein the second user is remote from the first user, wherein the second avatar comprises a second three-dimensional model and a second surface texture;

identifying a first movement made by the first user based on the first visual dataset captured by the first capture device;

receiving a second visual dataset identifying a second movement made by the second user as captured by a second capture device;

generating a first movement representation of the first movement by the first avatar;

generating a second movement representation of the second movement by the second avatar;

rendering a virtual space to be displayed via a first display; and

rendering the first avatar performing the first movement representation and the second avatar performing the second movement representation within the virtual space be displayed via the first display.

2. The method of claim 1, wherein the first display is a projector.

4. The method of claim 1, wherein the first three-dimensional model and the first surface texture of the first avatar are based on the first visual dataset corresponding to the first user as captured by the first capture device.

5. The method of claim 1, wherein the first three-dimensional model and the first surface texture of the first avatar are based on a previously-selected avatar template.

7. The method of claim 1, further comprising modifying the first three-dimensional model of the first avatar using a physics engine based on the first movement representation.

8. The method of claim 1, further comprising:

generating a virtual interactive object;

rendering the virtual interactive object in the virtual space using the first display; and

rendering movement of the virtual interactive object relative to the virtual space using the first display.

9. The method of claim 1, further comprising:

identifying occurrence of a trigger condition, wherein the trigger condition is based on at least one a gesture detection, a collision detection, a time-based event, or a detected voice command; and

performing an action in response to identifying the occurrence of the trigger condition.

10. The method of claim 9, wherein the action includes adjusting a network-connected device, wherein the network-connected device is one of a lighting device, a heating device, an air conditioning device, a thermostat, a ventilation device, a fan, a kitchen appliance, a laundry appliance, an energy generator device, an energy storage device, a router, a vacuum cleaner, or a vehicle.

11. The method of claim 9, wherein the action includes causing a transfer of funds.

12. The method of claim 9, wherein the action includes drawing a shape within the virtual space.

13. The method of claim 9, wherein the action includes modifying at least one of the first avatar or the second avatar.

3. (canceled)

6. (canceled)

14. A system for telepresence communication, the system comprising:

a first capture device that captures a first visual dataset identifying a first movement made by a first user;

a communication transceiver that receives;

data corresponding to a second avatar representative of a second user, and

a second visual dataset identifying a second movement made by the second user as captured by a second capture device, wherein the second user is remote from the first user, wherein the second avatar comprises a second three-dimensional model and a second surface texture;

a memory that stores instructions; and

a processor coupled to the memory, wherein execution of the instructions by the processor causes the processor to;

generate a skeleton based on the first visual dataset corresponding to the first user as captured by the first capture device,

generate a first avatar representative of the first user by applying a first three-dimensional model to the skeleton and a first surface texture to the first three-dimensional model,

generate a first movement representation of the first movement by the first avatar,

generate a second movement representation of the second movement by the second avatar,

render a virtual space to be displayed via a first display, and

render the first avatar performing the first movement representation and the second avatar performing the second movement representation within the virtual space to be displayed via the first display.

15. The system of claim 14, further comprising the first display, wherein the first display is a projector.

17. The system of claim 14, wherein the first three-dimensional model and the first surface texture of the first avatar are based on the first visual dataset corresponding to the first user as captured by the first capture device.

18. The system of claim 14, wherein the first three-dimensional model and the first surface texture of the first avatar are based on a previously-selected avatar template.

16. (canceled)

19. A system for telepresence communication, the system comprising:

a communication transceiver that;

receives a first visual dataset identifying a first movement made by a first user as captured by a first capture device,

receives a second visual dataset identifying a second movement made by a second user as captured by a second capture device,

receives data corresponding to a first avatar to represent the first user,

receives data corresponding to a second avatar to represent the second user, and

transmits information corresponding to a scene, wherein the second user is remote from the first user, and the second avatar comprises a second three-dimensional model and a second surface texture;

a memory that stores instructions; and

a processor coupled to the memory, wherein execution of the instructions by the processor causes the processor to;

generate a skeleton based on the first visual dataset corresponding to the first user as captured by the first capture device,

generate the first avatar representative of the first user based on the data corresponding to the first avatar by applying a first three-dimensional model to the skeleton and a first surface texture to the first three-dimensional model,

generate the second avatar representative of the second user based on the data corresponding to the second avatar,

generate a first movement representation of the first movement by the first avatar,

generate a second movement representation of the second movement by the second avatar, and

render the scene, the scene including the first avatar performing the first movement representation and the second avatar performing the second movement representation within a virtual space.

20. The system of claim 19, wherein the system includes one or more servers remote from the first user and from the second user.