A METHOD SYSTEM AND APPARATUS FOR TELEPRESENCE COMMUNICATIONS UTILIZING VIDEO AVATARS

• US 20060210045A1
• Filed: 06/06/2006
• Published: 09/21/2006
• Est. Priority Date: 12/30/2002
• Status: Active Grant

• Current Assignee: Google Technology Holdings LLC (Alphabet Inc.)
• Original Assignee: Motorola, Inc. (Motorola Solutions, Inc.)
• Inventors: Buhrke, Eric R., Tarlton, Mark A., Valliath, George T.
• Granted Patent: US 8,072,479 B2
• Time in Patent Office: Days
• US Class Current: 379/202.10
• CPC Class Codes: H04N 7/147 Communication arrangements, e.g. identifying the communication as a video-communication, intermediate storage of the signals selecting H04Q
• CPC Class Codes: H04N 7/157 defining a virtual conference space and using avatars or agents computer conference optimisation or adaptation H04L12/1827

Abstract

An apparatus, system and method for telepresence communications in an environment of a virtual location between two or more participants at multiple locations. First perspective data descriptive of the perspective of the virtual location environment experienced by a first participant at a first location and feature data extracted and/or otherwise captured from a second participant at a second location are processed to generate a first virtual representation of the second participant in the virtual environment from the perspective of the first participant. Likewise, second perspective data descriptive of the perspective of the virtual location environment experienced by the second participant and feature data extracted and/or otherwise captured from features of the first participant are processed to generate a second virtual representation of the first participant in the virtual environment from the perspective of the second participant. The first and second virtual representations are rendered and then displayed to the first and second participants, respectively. The first and second virtual representations are updated and redisplayed to the participants upon a change in one or more of the perspective data and feature data from which they are generated. The apparatus, system and method are scalable to two or more participants.

Claims

1. A method of teleconferencing in an environment of a virtual location between a first participant at a first location and a second participant at a second location, comprising:

processing a first plurality of user perspective modification data of a first perspective of the environment of the virtual location experienced by the first participant and a first encoded feature data from the second participant to generate a first virtual representation of the second participant in the environment of the virtual location from the perspective of the first participant;

processing a second plurality of user perspective modification data of a second perspective of an environment of the virtual location experienced by the second participant and a second encoded feature data from the first participant to generate a second virtual representation of the first participant in the environment of the virtual location from the perspective of the second participant;

displaying the first virtual representation of the second participant in the virtual location of the teleconference to the first participant at the first location; and

displaying the second virtual representation of the first participant in the virtual location of the teleconference to the second participant at the second location.

2. The method of claim 1, further comprising:

capturing the first plurality of user perspective modification data by a first tracking element under control of the first participant; and

capturing the second plurality of user perspective modification data by a second tracking element under control of the second participant.

3. The method of claim 1, further comprising:

updating the first virtual representation of the second participant upon a change in one or more of the first plurality of user perspective modification data and the first encoded feature data to generate an updated first virtual representation;

displaying the updated first virtual representation of the second participant in the virtual location of the teleconference to the first participant at the first location;

updating the second virtual representation of the second participant upon a change in one or more of the second plurality of user perspective modification data and the second encoded feature data to generate an updated second virtual representation; and

displaying the updated second virtual representation of the first participant in the virtual location of the teleconference to the second participant at the first location.

4. The method of claim 3, further comprising:

updating the first virtual representation of the second participant upon a change in the second plurality of user perspective modification data; and

updating the second virtual representation of the first participant upon a change in the first plurality of user perspective modification data.

5. A method of teleconferencing in an environment of a virtual location between a first participant at a first location and a second participant at a second location, comprising:

processing a first plurality of perspective data of a first perspective of the environment of the virtual location experienced by the first participant and a first encoded feature data from the second participant to generate a first virtual representation of the second participant in the environment of the virtual location from the perspective of the first participant;

processing a first plurality of user perspective modification data of a second perspective of an environment of the virtual location experienced by the second participant and a first extracted feature data extracted from a first plurality of cued data captured from the first participant to generate a second virtual representation of the first participant in the environment of the virtual location from the perspective of the second participant;

displaying the first virtual representation of the second participant in the virtual location of the teleconference to the first participant at the first location; and

displaying the second virtual representation of the first participant in the virtual location of the teleconference to the second participant at the second location.

6. The method of claim 5, further comprising:

updating the first virtual representation of the second participant upon a change in one or more of the first plurality of perspective data of the first perspective and the first encoded feature data to generate an updated first virtual representation;

displaying the updated first virtual representation of the second participant in the virtual location of the teleconference to the first participant at the first location;

updating the second virtual representation of the first participant upon a change in one or more of the first plurality of user perspective modification data of the second perspective and the first extracted feature data extracted from the first plurality of cued data captured from the first participant to generate an updated second virtual representation; and

displaying the updated second virtual representation of the first participant in the virtual location of the teleconference to the second participant at the first location.

7. The method of claim 5, wherein displaying the first virtual representation of the second participant to the first participant in the first location further comprises:

displaying the first virtual representation of the second participant to the first participant on a first head mounted display worn by the first participant.

8. A method of teleconferencing in an environment of a virtual location between a first participant at a first location and a second participant at a second location, comprising:

capturing a first plurality of feature data from the first participant and a first plurality of user perspective modification data of a first perspective of an environment of the virtual location experienced by the first participant;

capturing a second plurality of feature data from the second participant and a second plurality of user perspective modification data of a second perspective of the environment of the virtual location experienced by the second participant;

encoding the first plurality of feature data from the first participant to generate a first plurality of encoded feature data;

encoding the second plurality of feature data from the second participant to generate a second plurality of encoded feature data;

processing the first plurality of user perspective modification data, the second plurality of encoded feature data, and a first environment data of the environment of the virtual location to generate a first virtual representation of the second participant in the environment of the virtual location from the perspective of the first participant;

processing the second plurality of user perspective modification data, the first plurality of encoded feature data, and a second environment data of the environment of the virtual location to generate a second virtual representation of the first participant in the environment of the virtual location from the perspective of the second participant;

displaying the first virtual representation of the second participant in the virtual location of the teleconference to the first participant at the first location; and

displaying the second virtual representation of the first participant in the virtual location of the teleconference to the second participant at the second location.

9. The method of claim 8, further comprising:

capturing the first plurality of user perspective modification data by a first tracking element under control of the first participant; and

capturing the second plurality of user perspective modification data by a second tracking element under control of the second participant.

10. The method of claim 8, further comprising:

updating the first virtual representation of the second participant upon a change in one or more of the first plurality of user perspective modification data, the second plurality of encoded feature data and the first environment data to generate an updated first virtual representation;

displaying the updated first virtual representation of the second participant in the virtual location of the teleconference to the first participant at the first location;

updating the second virtual representation of the first participant upon a change in one or more of the second plurality of user perspective modification data, the first plurality of encoded feature data and the second environment data to generate an updated second virtual representation; and

displaying the updated second virtual representation of the first participant in the virtual location of the teleconference to the second participant at the first location.

11. The system of claim 10, further comprising:

updating the first virtual representation of the second participant upon a change in the second plurality of user perspective modification data; and

updating the second virtual representation of the first participant upon a change in the first plurality of user perspective modification data.

12. The method of claim 8, wherein capturing the first and second plurality of feature data and the first and second plurality of user perspective modification data, respectively, occurs in real time.

13. A method of teleconferencing in an environment of a virtual location between a first participant at a first location and a second participant at a second location, comprising:

capturing a first plurality of cued data generated by a plurality of features of the first participant and a first plurality of perspective data of a first perspective of the environment of the virtual location experienced by the first participant;

capturing a first plurality of feature data from the second participant and a first plurality of user perspective modification data of a second perspective of the environment of the virtual location experienced by the second participant;

extracting a first extracted feature data of the first participant from the first plurality of cued data;

encoding the first plurality of feature data from the second participant to generate a first plurality of encoded feature data;

processing the first plurality of perspective data of the first participant, the first plurality of encoded feature data of the second participant, and a first environment data of the environment of the virtual location to generate a first virtual representation of the second participant in the environment of the virtual location from the perspective of the first participant;

processing the first plurality of user perspective modification data of the second participant, the first extracted feature data of the first participant, and a second environment data of the environment of the virtual location to generate a second virtual representation of the first participant in the environment of the virtual location from the perspective of the second participant;

displaying the first virtual representation of the second participant in the virtual location of the teleconference to the first participant at the first location; and

displaying the second virtual representation of the first participant in the virtual location of the teleconference to the second participant at the second location.

14. The method of claim 13, further comprising:

updating the first virtual representation of the second participant upon a change in one or more of the first plurality of perspective data of the first perspective, the first encoded feature data, and the first environment data to generate an updated first virtual representation;

displaying the updated first virtual representation of the second participant in the virtual location of the teleconference to the first participant at the first location;

updating the second virtual representation of the first participant upon a change in one or more of the first plurality of user perspective modification data of the second participant, the first extracted feature data extracted the first participant, and the second environment data to generate an updated second virtual representation; and

displaying the updated second virtual representation of the first participant in the virtual location of the teleconference to the second participant at the first location.

15. The method of claim 13, wherein displaying the first virtual representation of the second participant to the first participant in the first location further comprises:

displaying the first virtual representation of the second participant to the first participant on a first head mounted display worn by the first participant.

16. A system that supports a teleconference in an environment of a virtual location between a first participant at a first location and a second participant at a second location, comprising:

a first processing element, comprising a first encoder/decoder element, that processes a first plurality of feature data captured from the first participant to generate first encoded feature data of the first participant;

a first tracking element operable to capture first user perspective modification data of the first participant that reflects a first perspective of the environment of the virtual location experienced by the first participant;

a first transmit element that transmits the first encoded feature data and the first user perspective modification data of the first participant;

a second processing element, comprising a second encoder/decoder element, that processes a second plurality of feature data captured from the second participant to generate second encoded feature data of the second participant;

a second tracking element operable to capture second user perspective modification data of the second participant that reflects to a second perspective of the environment of the virtual location experienced by the second participant;

a second transmit element that transmits the second encoded feature data and the second user perspective modification data of the second participant;

wherein the first processing element processes the first user perspective modification data, the second encoded feature data, and a first environment data of the environment of the virtual location to generate a first virtual representation of the second participant in the environment of the virtual location from the perspective of the first participant;

wherein the second processing element processes the second user perspective modification data, the first encoded feature data, and a second environment data of the environment of the virtual location to generate a second virtual representation of the first participant in the environment of the virtual location from the perspective of the second participant;

a first display element that displays the first virtual representation to the first participant at the first location; and

a second display element that displays the second virtual representation to the second participant at the second location.

17. The system of claim 16, wherein the first processing element updates the first virtual representation of the second participant upon a change in one or more of the first user perspective modification data, the second encoded feature data and the first environment data, and

wherein the second processing element updates the second virtual representation of the first participant upon a change in one or more of the second user perspective modification data, the first encoded feature data and the second environment data.

18. The system of claim 17, further comprising wherein the first processing element updates the first virtual representation of the second participant upon a change in the second user perspective modification data and wherein the second processing element updates the second virtual representation of the first participant upon a change in the first user perspective modification data.

19. The system of claim 16, wherein the first tracking element is operable to capture the first user perspective modification data of the first participant under control of the first participant and wherein the second tracking element is operable to capture the second user perspective modification data of the second participant under control of the second participant

20. The system of claim 19, wherein the first processing element, the first transmit element, and the first display element are coupled to the first location and the second processing element, the second transmit element, and the second display element are coupled to the second location.

21. The system of claim 16, further comprising:

a first capture element that captures the first plurality of feature data from the first participant;

a second capture element that captures the second plurality of feature data from the second participant;

22. The system of claim 16, further comprising:

a first receive element operable to receive the second plurality of user perspective modification data and the second encoded feature data of the second participant transmitted by the second transmit element; and

a second receive element operable to receive the first plurality of perspective data and the first encoded feature data of the first participant transmitted by the first transmit element.

23. The system of claim 16, further comprising:

a first rendering element coupled to the first processing element and the first display element that renders the first virtual representation of the second participant for display by the first display element; and

a second rendering element coupled to the second processing element and the second display element that renders the second virtual representation of the first participant for display by the second display element;

24. A system that supports a teleconference in an environment of a virtual location between a first participant at a first location and a second participant at a second location, comprising:

a first processing element that processes a plurality of cued data captured from a plurality of features of the first participant and extracts selected feature data recognized from the first plurality of cued data to generate extracted feature data of the first participant;

a first transmit element that transmits the extracted feature data of the first participant;

a second processing element, comprising an encoder/decoder element, that processes a first plurality of feature data captured from the second participant to generate encoded feature data of the second participant;

a tracking element operable to capture user perspective modification data of the second participant that reflects a second perspective of the environment of the virtual location experienced by the second participant;

a second transmit element that transmits the encoded feature data of the second participant;

wherein the first processing element processes a first plurality of perspective data that relate to a first perspective of the environment of the virtual located experienced by the first participant, the encoded feature data of the second participant, and a first environment data of the environment of the virtual location to generate a first virtual representation of the second participant in the environment of the virtual location from the perspective of the first participant;

wherein the second processing element processes the user perspective modification data of the second participant, the first extracted feature data of the first participant, and a second environment data of the environment of the virtual location to generate a second virtual representation of the first participant in the environment of the virtual location from the perspective of the second participant;

a first display element that displays the first virtual representation to the first participant at the first location; and

a second display element that displays the second virtual representation to the second participant at the second location.

25. The system of claim 24, further comprising:

a first receive element coupled to the first processing element that receives the encoded feature data of the second participant transmitted by the second transmit element; and

a second receive element coupled to the second processing element that receives the extracted feature data of the first participant transmitted by the first transmit element.

26. The system of claim 25, further comprising the first transmitter transmitting the first plurality of perspective data of the first participant that is received by the second receive element and the second transmitter transmitting the user perspective modification data of the second participant that is received by the first receive element.

27. The system of claim 26, wherein an updated first virtual representation of the second participant is generated by the first processing element upon a change in at least one of the first plurality of perspective data of the first participant, the user perspective modification data of the second participant, the encoded feature data of the second participant, and the first environment data of the environment of the virtual location, and wherein an updated second virtual representation of the first participant is generated by the second processing element upon a change in at least one of the user perspective modification data of the second participant, the first plurality of perspective data of the first participant, the extracted feature data of the first participant, and the second environment data of the environment of the virtual location.

28. The system of claim 24, wherein the first display element is a head mounted display worn by the first participant and further comprising:

a plurality of capture elements of the head mounted display that capture the plurality of cued data.

29. The system of claim 24, wherein the first processing element further comprises a view generation element that generates first and second views of the first virtual presentation of the second participant that are rendered by a rendering element and displayed by the first display element to at least one eye of the first participant.

30. An apparatus that supports a teleconference in an environment of a virtual location between a first participant at a first location and a second participant at a second location, comprising:

a receive element that receives a first plurality of user perspective modification data captured from the first participant and relating to a first perspective of the environment of the virtual location experienced by the first participant and a first plurality of encoded feature data captured from the second participant;

a processing element coupled to the receive element that processes the first plurality of user perspective modification data, the first plurality of encoded feature data and environment data about the environment of the virtual location to generate a virtual representation of the second participant in the environment of the virtual location from the perspective of the first participant;

a rendering element coupled to the processing element that renders the virtual representation of the second participant for display; and

a display element that displays the rendered virtual representation

wherein the processing element updates the virtual representation of the second participant upon a change in one or more of the first plurality of user perspective modification data, the first plurality of encoded feature data and the environment data.

31. The apparatus of claim 30, wherein the first plurality of user perspective modification data is captured from the first participant by a first tracking element under control of the first participant and wherein the second plurality of user perspective modification data is captured from the second participant by a second tracking element under control of the second participant

32. The apparatus of claim 30, wherein the processing element further updates the virtual representation of the second participant upon a change in a second plurality of user perspective modification data relating to a second perspective of an environment of the virtual location experienced by the second participant.

33. The apparatus of claim 30, wherein the processing element maps the first plurality of encoded feature data to a three-dimensional model of the second participant in accordance with the first plurality of user perspective modification data and the environment data to generate the virtual representation of the second participant.

34. The apparatus of claim 30, wherein the virtual representation of the second participant comprises a virtual image component generated by the processing element from the first plurality of encoded feature data of the second participant.

35. The apparatus of claim 34, wherein the virtual representation further comprises a virtual audio component generated by the processing element from one or more of the first plurality of user perspective modification data of the first participant and an audio representation of the environment of the virtual location.

36. The apparatus of claim 30, wherein the audio representation comprises one or more of a first audio component as experienced by the first participant in the environment, a second audio component as experienced by the second participant in the environment, and an ambient component of the environment in the virtual location.

37. The apparatus of claim 36, wherein the virtual audio component is changed by a change in one or more of the first audio component as experienced by the first participant in the environment, the second audio component as experienced by the second participant in the environment, and the ambient component of the environment in the virtual location.

38. The apparatus of claim 30, wherein the first plurality of user perspective modification data from the first participant comprises one or more of head movement of the first participant and a relative position of the first participant in the environment of the virtual location.

39. The apparatus of claim 30, further comprising:

a transmit element coupled to the processing element that transmits a second plurality of encoded feature data captured from the first participant.

40. The apparatus of claim 30, wherein the first location is a local location and the second location is a remote location.