Telepresence of multiple users in interactive virtual space

• US 10,699,461 B2
• Filed: 12/20/2016
• Issued: 06/30/2020
• Est. Priority Date: 12/20/2016
• Status: Active Grant

• Current Assignee: Sony Interactive Entertainment LLC (Sony Group Corp.)
• Original Assignee: Sony Interactive Entertainment LLC (Sony Group Corp.)
• Inventors: Chaney, Roxy
• Primary Examiner(s): Leggett, Andrea C
• Application Number:
  US15/385,515
• Publication Number:
  US 20180174347A1
• Time in Patent Office: 1,288 Days
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/016 Input arrangements with force or tactile feedback as computer generated output to the user
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06F 3/1454 involving copying of the display data of a local workstation or window to a remote workstation or window so that an actual copy of the data is displayed simultaneously on two or more displays, e.g. teledisplay
• CPC Class Codes: G06T 13/40 of characters, e.g. humans, animals or virtual beings
• CPC Class Codes: G09G 2350/00 Solving problems of bandwidth in display systems
• CPC Class Codes: G09G 2354/00 Aspects of interface with display user
• CPC Class Codes: G09G 2370/02 Networking aspects
• CPC Class Codes: G09G 2370/22 Detection of presence or absence of input display information or of connection or disconnection of a corresponding information source
• CPC Class Codes: G09G 5/12 Synchronisation between the display unit and other units, e.g. other display units, video-disc players
• CPC Class Codes: G09G 5/377 for mixing or overlaying two or more graphic patterns G09G5/02, G09G5/397 take precedence
• CPC Class Codes: H04N 7/157 defining a virtual conference space and using avatars or agents computer conference optimisation or adaptation H04L12/1827
• CPC Class Codes: H04N 9/3179 Video signal processing therefor
• CPC Class Codes: H04N 9/3194 including sensor feedback

Abstract

A telepresence communication uses information captured by a first capture device about a first user and information captured by a second capture device about a second user to generate a first avatar corresponding to the first user and a second avatar corresponding to the second user. A scene can be rendered locally or by a remote server in which the first avatar and the second avatar are both rendered in a virtual space. The first avatar is rendered to move based on movements made by the first user as captured by the first capture device, and the second avatar is rendered to move based on movements made by the second user as captured by the second capture device. The avatars may be realistic, based on avatar templates, or some combination thereof. The rendered scene may include virtual interactive objects that the avatars can interact with.

Claims

1. A method for telepresence communication, the method comprising:

receiving a first visual dataset corresponding to a three-dimensional shape of a first user and captured by a first capture device, wherein the first visual dataset specifies a plurality of key points each having a respective depth on the body three-dimensional shape of the first user;

generating a skeleton based on the first visual dataset captured by the first capture device, wherein generating the skeleton includes extracting a set of the key points from the first visual dataset, the generated skeleton representing the extracted set of key points;

generating a first three-dimensional wireframe model that recreates the three-dimensional shape of the first user as a plurality of planar surfaces around the generated skeleton by connecting one or more sets of the key points each having the respective depth specified by the first visual dataset, wherein the first three-dimensional wireframe model includes points that match the extracted set of key points represented by the skeleton;

generating a first three-dimensional avatar by applying a first surface texture to at least one of the planar surfaces of the first three-dimensional wireframe model;

receiving a second three-dimensional avatar representative of a second user, wherein the second three-dimensional avatar comprises a second surface texture applied to at least one planar surface of a second three-dimensional wireframe model;

rendering a three-dimensional virtual scene that includes the first three-dimensional avatar and the second three-dimensional avatar;

identifying a first movement made by the first user based on the first visual dataset captured by the first capture device including movement data, wherein the movement data describes the first movement as performed by the skeleton;

generating a first three-dimensional movement representation to be performed by the first three-dimensional avatar in accordance with the movement data describing the first movement by the skeleton;

generating a second three-dimensional movement representation of a second movement indicated by a change to a depth of at least one key point of a second visual dataset corresponding to the second three-dimensional avatar; and

rendering the first three-dimensional avatar performing the first three-dimensional movement representation and the second three-dimensional avatar performing the second three-dimensional movement representation within the three-dimensional virtual scene.

2. The method of claim 1, further comprising projecting a display of the rendered three-dimensional virtual scene via a projector, the display projected within a space at a location of the first user.

3. The method of claim 2, wherein rendering the three-dimensional virtual scene is based on a non-planar topography of a projection surface within the space at the location of the first user.

4. The method of claim 3, further comprising identifying the non-planar topography based on at least one of analyzing a predefined pattern projected onto the non-planar topography and mapping the non-planar topography via one or more depth sensors.

5. The method of claim 2, further comprising adjusting a point of view of the display of the rendered three-dimensional virtual scene based on a position of the first user within the space relative to the first capture device.

7. The method of claim 1, further comprising generating the first surface texture based on the first visual dataset as captured by the first capture device.

8. The method of claim 1, further comprising generating the first surface texture based on a selected avatar template.

9. The method of claim 1, further comprising generating an alternative first three-dimensional wireframe model associated with a map of an avatar corresponding to the skeleton.

10. The method of claim 9, further comprising generating an alternative first three-dimensional avatar by applying an alternative first surface texture to at least one planar surface of the alternative first three-dimensional wireframe model.

11. The method of claim 9, further comprising mapping the skeleton representing the extracted set of key points to one or more portions of the avatar to generate the map, wherein the avatar is associated with an avatar template specifying one or more visual characteristics of the avatar.

12. The method of claim 1, wherein the three-dimensional virtual scene further includes a virtual interactive object associated with one or more rules of a physics engine governing movement of the virtual interactive object within the three-dimensional virtual scene.

13. The method of claim 1, further comprising:

storing information regarding a plurality of trigger conditions in a trigger database in memory, each trigger condition associated with a different corresponding action;

detecting one of the trigger conditions within the three-dimensional virtual scene;

identifying the action associated with the detected trigger condition by querying the trigger database; and

performing the identified action associated with the detected trigger condition.

14. The method of claim 13, wherein the action includes at least one of making a purchase, transferring funds, drawing a shape within the three-dimensional virtual scene, modifying at least one of the first three-dimensional avatar or the second three-dimensional avatar, adding an overlay to the three-dimensional virtual scene, and adjusting a network-connected device, wherein the network-connected device is one of a lighting device, a heating device, an air conditioning device, a thermostat, a ventilation device, a fan, a kitchen appliance, a laundry appliance, an energy generator device, an energy storage device, a router, a vacuum cleaner, or a vehicle.

15. The method of claim 1, further comprising receiving customization input from the first user specifying a visual alteration, and modifying the first three-dimensional avatar based on the visual alteration specified by the customization input, wherein the first three-dimensional avatar appears modified within the rendered three-dimensional virtual scene.

16. The method of claim 15, wherein modifying the first three-dimensional avatar includes altering at least one of the first three-dimensional wireframe model and the first surface texture.

17. The method of claim 1, further comprising sending the rendered three-dimensional virtual scene over a communication network to one or more computing devices that each generate a display of the rendered three-dimensional virtual scene.

18. The method of claim 17, further comprising receiving a plurality of visual datasets from the computing devices, wherein rendering the three-dimensional virtual scene is further based on the received plurality of visual datasets.

6. The method of 5, further comprising continuing to adjust the point of view of the display of the rendered three-dimensional virtual scene as the position of the first user moves within the space.

19. A system for telepresence communication, the system comprising:

a first capture device that captures a first visual dataset corresponding to a three-dimensional shape of a first user, wherein the first visual dataset specifies a plurality of key points each having a respective depth on the three-dimensional shape of the first user;

a communication transceiver that receives a second three-dimensional avatar representative of a second user, wherein the second three-dimensional avatar comprises a second surface texture applied to at least one planar surface of a second three-dimensional wireframe model;

a memory that stores instructions;

a processor coupled to the memory, wherein execution of the instructions by the processor causes the processor to;

generate a skeleton based on the first visual dataset captured by the first capture device, wherein generating the skeleton includes extracting a set of the key points from the first visual dataset, the generated skeleton representing the extracted set of key points,

generate a first three-dimensional wireframe model that recreates the three-dimensional shape of the first user as a plurality of planar surfaces around the generated skeleton by connecting one or more sets of the key points each having the respective depth specified by the first visual dataset, wherein the first three-dimensional wireframe model includes points that match the extracted set of key points represented by the skeleton,

generate a first three-dimensional avatar by applying a first surface texture to at least one of the planar surfaces of the first three-dimensional wireframe model,

render a three-dimensional virtual scene that includes the first three-dimensional avatar and the second three-dimensional avatar,

identify a first movement made by the first user based on the first visual dataset captured by the first capture device including movement data, wherein the movement data describes the first movement as performed by the skeleton;

generate a first three-dimensional movement representation to be performed by the three-dimensional avatar in accordance with the movement data describing the first movement by the skeleton,

generate a second three-dimensional movement representation of a second movement indicated by a change to a depth of at least one key point of a second visual dataset corresponding to the second three-dimensional avatar, and

render the first three-dimensional avatar performing the first three-dimensional movement representation and the second three-dimensional avatar performing the second three-dimensional movement representation within the three-dimensional virtual scene.

20. The system of claim 19, further comprising a projector that projects a display of the rendered three-dimensional virtual scene.