Rendered audiovisual communication

• US 9,094,576 B1
• Filed: 03/12/2013
• Issued: 07/28/2015
• Est. Priority Date: 03/12/2013
• Status: Active Grant

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Karakotsios, Kenneth Mark
• Primary Examiner(s): Tieu, Binh
• Application Number:
  US13/797,394
• Time in Patent Office: 868 Days
• Field of Search: 348/14.01, 348/14.02, 348/14.07, 348/14.1, 348/14.12, 348/14.14, 348/14.16, 348/14.15
• US Class Current: 1/1
• CPC Class Codes: G06V 40/174 Facial expression recognition
• CPC Class Codes: G10L 15/02 Feature extraction for speech recognition; Selection of recognition unit
• CPC Class Codes: G10L 15/26 Speech to text systems G10L15/08 takes precedence
• CPC Class Codes: G10L 2015/025 Phonemes, fenemes or fenones being the recognition units
• CPC Class Codes: H04N 5/265 Mixing
• CPC Class Codes: H04N 7/157 defining a virtual conference space and using avatars or agents computer conference optimisation or adaptation H04L12/1827

Abstract

Systems and approaches are provided to allow for rendered audiovisual communication. An electronic device can be used to capture image information relating to physical features of a user. A model can be generated from the image information, and the model may be used to render audiovisual communication information from image and audio captured in real time. The rendered audiovisual communication data can simulate live video conferencing with substantial performance gains over conventional approaches to video conferencing. When the image capturing component of the electronic device is capable of depth imaging, stereo imaging, or other imaging techniques, the rendered audiovisual communication can be further enhanced with 3-D rendering of the user. Other aspects of audiovisual data, such as speech, background, and lighting conditions can also be rendered or synthesized to improve audiovisual communication.

Claims

1. A computer-implemented method for providing audiovisual communication between at least a first user and a second user, comprising:

capturing first image data using a camera of a first computing device, the first image data including a first view of a user;

analyzing at least a portion of the first image data to determine one or more features of the user;

obtaining a model of the user based at least in part upon the one or more features of the user;

capturing second image data including a second view of the user;

capturing audio data corresponding to the user corresponding to the second image data;

analyzing at least one of the second image data or the audio data to determine current values corresponding to the one or more features of the user; and

causing a third view of the first user to be displayed on a second computing device, the third view being rendered based at least in part upon the model and the current values corresponding to the one or more features of the first user.

2. The computer-implemented method of claim 1, wherein obtaining the model of the user based at least in part upon the one or more features of the user comprises one of:

generating a two-dimensional model or a three-dimensional model of the user based at least in part upon the one or more features;

recognizing the user based at least in part upon the one or more features and selecting a previously generated model corresponding to the user;

or

providing one or more previously generated models corresponding to the user based at least in part upon the one or more features, receiving a selection of one of the one or more previously generated models, and retrieving the one of the one or more previously generated models.

3. The computer-implemented method of claim 1, further comprising:

causing a fourth view to be displayed on the first computing device concurrently with display of the third view on the second computing device, the fourth view corresponding to the second user.

4. A computer-implemented method for providing audiovisual communication, comprising:

under control of one or more computer systems configured with executable instructions,

obtaining first image data including a first representation of at least one first portion of a user, the user being associated with a first computing device;

analyzing at least a portion of the first image data to determine one or more features of the user;

obtaining a model of the user based at least in part upon the one or more features;

obtaining second image data including a second representation of at least one second portion of the user;

obtaining audio data corresponding to the user, the audio data further corresponding to at least one of the first image data or the second image data;

causing rendered image data corresponding to the user to be generated based at least in part upon the model of the user and at least one of the first image data, the second image data, or the audio data; and

causing the rendered image data to be sent to a second computing device in communication with the first computing device.

5. The computer-implemented method of claim 4, wherein obtaining the model of the user comprises one of:

generating a two-dimensional model or a three-dimensional model of the user based at least in part upon one of the first image data or the second image data;

recognizing the user based at least in part upon one of the first image data or the second image data and selecting a previously generated model corresponding to the user;

or

providing one or more previously generated models corresponding to the user based at least in part upon one of the first image data or the second image data, receiving a selection of one of the one or more previously generated models, and retrieving the one of the one or more previously generated models.

6. The computer-implemented method of claim 4, further comprising:

receiving a request to modify the model of the user; and

causing a modified model to be generated based at least in part upon the request,

wherein the rendered image data is further based at least in part upon the modified model.

7. The computer-implemented method of claim 4, wherein at least one the first image or the second image data includes one or more background elements, the method further comprising:

receiving a request to modify the one or more background elements of the at least one of the first image data or the second image data,

wherein the rendered image data is further based at least in part upon modifications of the one or more background elements, the modifications corresponding to the request.

8. The computer-implemented method of claim 4, wherein the audio data includes speech of the user, the method further comprising:

receiving a request to synthesize the speech of the user according to one of a speech synthesizer or a speech model;

causing second audio data to be synthesized based at least in part upon the speech of the user and the one of the speech synthesizer or the speech model; and

causing the second audio data to be sent to the second computing device, the second audio data being at least substantially in sync with the rendered image data.

9. The computer-implemented method of claim 8, wherein:

the speech comprises a first language;

the request further specifies translation of the speech to second speech comprising a second language;

the second audio data is based at least in part upon the second speech; and

the rendered image data is based at least in part upon the second audio data.

10. The computer-implemented method of claim 4, wherein the audio data includes speech of the user, the method further comprising:

receiving a request to send text corresponding to the speech of the user;

causing the speech of the user to be converted to the text using a speech recognition process; and

causing the text to be to be sent to the second computing device, the text being at least substantially in sync with the rendered image data.

11. The computer-implemented method of claim 4, wherein the second computing device is associated with a second user, the method further comprising:

obtaining a second model of the second user based at least in part upon third image data including a first representation of at least one first portion of the second user;

obtaining fourth image data including a second representation of at least one second portion of the second user;

obtaining second audio data corresponding to the fourth image data;

generating second rendered image data corresponding to the second user based at least in part upon the second model of the second user and at least one of the third image data, the fourth image data, or the second audio data; and

sending the second rendered image data to the first computing device.

12. The computer-implemented method of claim 4, further comprising:

determining a display capability of the second computing device; and

sending the rendered image data to the computing device in a format that is supported by the display capability of the second computing device.

13. A computing device, comprising:

at least one processor;

a camera; and

a microphone; and

memory including instructions that, upon being executed by the at least one processor, cause the computing device to;

capture first image data including a first representation of at least one first portion of a user using the camera;

analyze at least a portion the first image data to determine one or more features of the user;

obtain a model of the user based at least in part upon the one or more features;

obtain second image data including a second representation of at least one second portion of the user;

capture, using the microphone, audio data corresponding to the user, the audio data further corresponding to at least one of the first image data or the second image data;

cause rendered image data corresponding to the user to be generated based at least in part upon the model of the user and at least one of the first image data, the second image data, or the audio data; and

cause the rendered image data to be sent to a second computing device in communication with the computing device.

14. The computing device of claim 13, wherein the instructions upon being executed by the at least one processor further cause the computing device to:

receive a request to modify one or more visual elements of the rendered image data; and

cause the rendered image data to be generated further based at least in part upon the one or more visual elements.

15. A non-transitory computer-readable storage medium storing instructions for providing rendered audiovisual communication, the instructions when executed by a processor causing the processor to:

obtain first image data including a first representation of at least one first portion of a user, the user being associated with a first computing device;

analyze at least a portion of the first image data to determine one or more features of the user;

obtain a model of the user based at least in part upon the one or more features;

obtain second image data including a second representation of at least one second portion of the user;

obtain audio data corresponding to the user, the audio data further corresponding to at least one of the first image data or the second image data;

generate rendered image data corresponding to the user based at least in part upon the model of the user and at least one of the first image data, the second image data, or the audio data; and

send the rendered image data to a second computing device in communication with the first computing device.

16. The non-transitory computer-readable storage medium of claim 15, wherein the instructions upon being executed further cause the processor to:

determine a display capability of the first computing device; and

send the rendered image data to the second computing device in a format that is supported by the display capability of the second computing device.

17. The non-transitory computer-readable storage medium of claim 15, wherein the audio data includes speech of the user and the instructions upon being executed further cause the processor to:

synthesize second audio data based at least in part upon (a) one of a speech synthesizer or a speech model and (b) the speech of the user; and

send the second audio data to the second computing device, the second audio data being at least substantially in sync with the rendered image data.

18. The non-transitory computer-readable storage medium of claim 17, wherein:

the speech comprises a first language;

the second audio data is based at least in part upon second speech, comprising a second language, that is a translation of the speech; and

the rendered image data is based at least in part upon the second audio data.