Avatar-Mediated Telepresence Systems with Enhanced Filtering

• US 20160134840A1
• Filed: 07/27/2015
• Published: 05/12/2016
• Est. Priority Date: 07/28/2014
• Status: Abandoned Application

• Current Assignee: Alexa Margaret Mcculloch
• Original Assignee: Alexa Margaret Mcculloch
• Inventors: McCulloch, Alexa Margaret
• Application Number:
  US14/810,400
• Publication Number:
  US 20160134840A1
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06T 13/40 of characters, e.g. humans, animals or virtual beings
• CPC Class Codes: G06V 20/20 in augmented reality scenes
• CPC Class Codes: G06V 20/647 by matching two-dimensional images to three-dimensional objects
• CPC Class Codes: G06V 40/165 using facial parts and geometric relationships
• CPC Class Codes: G06V 40/176 Dynamic expression
• CPC Class Codes: G06V 40/19 Sensors therefor
• CPC Class Codes: H04N 7/157 defining a virtual conference space and using avatars or agents computer conference optimisation or adaptation H04L12/1827

Abstract

Methods and systems using photorealistic avatars to provide live interaction. Several groups of innovations are described. In one such group, trajectory information included with the avatar model makes the model 4D rather than 3D. In another group, a fallback representation is provided with deliberately-low quality. In another group, avatar fidelity is treated as a security requirement. In another group, avatar representation is driven by both video and audio inputs, and audio output depends on both video and audio input. In another group, avatar representation is updated while in use, to refine representation by a training process. In another group, avatar representation uses the best-quality input to drive avatar animation when more than one input is available, and swapping to a secondary input while the primary input is insufficient. In another such group, the avatar representation can be paused or put into a standby mode.

Claims

1. A system, comprising:

input devices which capture audio and video streams from a first user'"'"'s actual appearance and movements;

a first computing system which receives video and audio data from the input devices, and accordingly generates, according to a known model, an animated photorealistic 3D avatar with trajectories and cues for animation, which substantially replicates appearance, gestures, and inflections of the first user in real time; and

a second computing system, remote from said first computing system, which uses said trajectories and cues to reconstruct a photorealistic real-time 3D avatar, in accordance with the known model, which varies, in accordance with said trajectories and cues, to match the appearance, gestures, inflections of the first user, and outputs said avatar to be shown on a display to a second user;

wherein the known model includes time-dependent trajectories for at least some elements of the user'"'"'s dynamically simulated appearance.

2. The system of claim 1, wherein said first computing system is a distributed computing system.

3. The system of claim 1, wherein said input devices include multiple cameras.

4. The system of claim 1, wherein said input devices include at least one microphone.

5. The system of claim 1, wherein said first computing system uses cloud computing.

6. A method, comprising:

capturing audio and video streams from a first user'"'"'s actual appearance and movements, and accordingly generating, according to a known model, a first animated photorealistic 3D avatar which, with associated trajectories and cues for animation, substantially replicates gestures, inflections, and general appearance of the first user in real time; and

transmitting the trajectories and cues for animation; and

receiving, from a second computing system, trajectories and cues to reconstruct a second photorealistic real-time 3D avatar in accordance with the known model, and reconstructing the second avatar, and displaying the reconstructed avatar to the first user;

wherein the known model includes time-dependent trajectories for at least some elements of a user'"'"'s dynamically simulated appearance.

7. The method of claim 6, wherein said first computing system is a distributed computing system.

8. The method of claim 6, wherein said input devices include multiple cameras.

9. The method of claim 6, wherein said input devices include at least one microphone.

10. The method of claim 6, wherein said first computing system uses cloud computing.

11. A system, comprising:

input devices which capture audio and video streams from a first user'"'"'s actual appearance and movements;

a first computing system which receives video and audio data from the input devices, and accordingly generates, according to a known model, a data stream which uses a known avatar model to define an animated photorealistic 3D avatar which replicates gestures, inflections, and general appearance of the first user in real time; and

a second computing system, remote from said first computing system, which uses said data stream and said known model to reconstruct a photorealistic real-time 3D avatar which replicates gestures, inflections, and general appearance of the first user, and

outputs said avatar to be shown on a display to a second user;

wherein, during normal operation, the second computing system outputs said avatar with photorealism which is greater than the maximum of the uncanny valley; and

wherein, if normal operation is impeded, the second computing system either outputs said avatar with photorealism which is less than the minimum of the uncanny valley, or else outputs trajectory and cues that have been predefined in sequence for such purpose.

12. The system of claim 11, wherein said first computing system is a distributed computing system.

13. The system of claim 11, wherein said input devices include multiple cameras.

14. The system of claim 11, wherein said input devices include at least one microphone.

15. The system of claim 11, wherein said first computing system uses cloud computing.

16. The system of claim 11, wherein the known model includes time-dependent trajectories for at least some elements of a user'"'"'s dynamically simulated appearance.

17-67. -67. (canceled)