Method for teleconferencing using 3-D avatar

• US 9,424,678 B1
• Filed: 08/16/2013
• Issued: 08/23/2016
• Est. Priority Date: 08/21/2012
• Status: Active Grant

• Current Assignee: MidCap Financial Trust
• Original Assignee: Acronis International GmbH (Acronis AG)
• Inventors: Enakiev, Anton, Tsibrovski, Juri V., Tormasov, Alexander G., Beloussov, Serguei M., Protassov, Stanislav S.
• Primary Examiner(s): HAJNIK, DANIEL F
• Application Number:
  US13/968,489
• Time in Patent Office: 1,103 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G01B 11/24 for measuring contours or curvatures
• CPC Class Codes: G06T 13/205 driven by audio data
• CPC Class Codes: G06T 13/40 of characters, e.g. humans, animals or virtual beings
• CPC Class Codes: H04N 7/157 defining a virtual conference space and using avatars or agents computer conference optimisation or adaptation H04L12/1827

Abstract

A method for implementing teleconferences when at least one participant receives 3-D data. A data rendering device presents data in a 3-D format or in pseudo 3-D format. A 3-D image is formed on a user computer system. The 3-D presentation is calculated by a local computer system. A block of user avatars is formed on the local computer system for all teleconference participants (including a local participant). The participant avatar includes a kinetic model (joints, muscles, body parts, etc.). The avatar includes a participant'"'"'s behavior model (gestures, fingers, mimics, etc). The avatar also includes an avatar skin. The avatar skin includes a stable (unchangeable) part of the participant containing face and visible body parts, as well as modifiable parts (e.g., clothes, shoes, accessories, etc.).

Claims

1. A system for providing a virtual conference environment using 3-D avatars, the system comprising:

a plurality of remotely controlled video cameras configured to record image data of conference participants, respectively;

a plurality of directional microphones configured to record audio data of the conference participants, respectively;

a plurality of speakers for reproducing the recorded audio data of the conference participants;

at least one kinetic sensor configured to detect and record movements of at least one of the conference participants, respectively, in a form of a kinetic data;

a virtual reality projector configured to render 3-D virtual data to the participants, the 3-D virtual data providing the virtual conference environment that includes each of the conference participants illustrated as the 3-D avatars; and

a computer processor configured to generate the 3-D virtual data by combining the image data with recorded audio data and the kinetic data;

wherein the 3-D avatars are stored in at least one avatar database and accessed by the computer processor when creating the 3-D virtual data,

wherein the computer processor is further configured to;

(i) extract differential and behavior parameters from the recorded image data for a selected 3-D avatar, (ii) optimize the recorded image data for the selected 3-D avatar for a transmission channel based on capacity and latency of the transmission channel, (iii) transfer the optimized data to a client associated with the selected 3-D avatar based on the capacity and latency, and (iv) apply the differential and behavior parameters to the selected 3-D avatar illustrated in the virtual conference environment to the client associated with the selected 3-D avatar,

wherein, when at least one additional conference participant attempts to join the virtual conference environment, the computer processor accesses the at least one avatar database to identify a 3-D avatar associated with the at least one additional conference participant, and

wherein, if the at least one avatar database does not have the 3-D avatar associated with the at least one additional conference participant, the computer processor is configured to access a photograph or video of the at least one additional conference participant from a source other than the at least one avatar database and generate the 3-D avatar associated with the at least one additional conference participant that is displayed in the virtual conference environment.

2. The system of claim 1, wherein the avatar database is located in a cloud.

3. The system of claim 1 further comprising at least one Doppler sensor configured to acquire the kinetic data.

4. The system of claim 1, wherein the kinetic data is acquired using traceable markers attached to body parts of the conference participants.

5. The system of claim 1, wherein the 3-D avatars are provided to remote server over a cloud.

6. The system of claim 1, wherein the 3-D avatars are displayed using at least one of:

a hologram;

virtual reality glasses;

a 3-D monitor; and

steam cloud images.

7. A method for providing a virtual conference environment using 3-D avatars, the method comprising:

capturing, by a plurality of remotely controlled video cameras, a video data of conference participants, respectively;

capturing, by a plurality of directional microphones, audio data of the conference participants, respectively;

detecting and recording, by at least one kinetic sensor, movements of at least one of the conference participants, respectively, as kinetic data;

generating 3-D virtual data by combining the captured video data with the captured audio data and with the recorded kinetic data;

rendering, by a virtual reality projector, the 3-D virtual data providing the virtual conference environment including each of the conference participants illustrated as the 3-D avatars;

storing the 3-D avatars in at least one avatar database;

extracting differential and behavior parameters from the captured data for a selected 3-D avatar;

optimizing the captured data for the selected avatar for a transmission channel based on capacity and latency of the transmission channel;

transferring the optimized data to a client based on the capacity and latency;

applying the differential and behavior parameters to the avatar illustrated in the virtual conference environment to the client;

when at least one additional conference participant attempts to join the virtual conference environment, accessing the at least one avatar database to identify a 3-D avatar associated with the at least one additional conference participant;

if the at least one avatar database does not have the 3-D avatar associated with the at least one additional conference participant, accessing a photograph or video of the at least one additional conference participant from a source other than the at least one avatar database; and

generating the 3-D avatar associated with the at least one additional conference participant that is displayed in the virtual conference environment.

8. The method of claim 7, further comprising adding a skin to the 3-D avatars.

9. The method of claim 7, further comprising masking lags in data transmission through the channel by applying behavior patterns.

10. The method of claim 9, wherein the behavior patterns are predefined and calculated during teleconference to avoid artifacts.

11. The method of claim 9, wherein the behavior patterns are extracted from prior data transmission translation and applied to any combination of micro-movement of parts of the 3-D avatar.

12. The method of claim 11, wherein the behavior patterns are applied to unintended gestures and movements to keep the 3-D avatar'"'"'s appearance live.

13. The method of claim 7, further comprising comparing current images of the participants with the 3-D avatars and correcting the avatars if the differences are detected.

14. The method of claim 7 further comprising generating artificial intelligence models for the 3-D avatars.

15. The method of claim 7, further comprising authenticating the 3-D avatars and changing an appearance of the 3-D avatars based on a successful authentication.