Enhanced communication between remote participants using augmented and virtual reality

• US 9,524,588 B2
• Filed: 12/30/2014
• Issued: 12/20/2016
• Est. Priority Date: 01/24/2014
• Status: Active Grant

• Current Assignee: Avaya Incorporated
• Original Assignee: Avaya Incorporated
• Inventors: Cohen, Azaria, Barzuza, Tamar, Wiener, Yair, Modai, Ori
• Primary Examiner(s): Gauthier, Gerald
• Application Number:
  US14/585,399
• Publication Number:
  US 20150215581A1
• Time in Patent Office: 721 Days
• Field of Search: 345/7, 345/8, 345/157, 345/633, 345/427, 348/14.07, 348/14.09, 348/14.1, 348/14.08, 348/14.16, 348/143, 348/207.1, 359/630, 434/271, 434/350, 463/6, 704/275, 709/203, 709/204, 381/309, 382/103, 715/700, 715/757, 235/375, 370/254
• US Class Current: 1/1
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0178 Eyeglass type eyeglass details G02C
• CPC Class Codes: G02B 27/01 Head-up displays
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: H04L 12/18 for broadcast or conference , e.g. multicast
• CPC Class Codes: H04L 12/1827 Network arrangements for conference optimisation or adaptation
• CPC Class Codes: H04L 65/403 Arrangements for multi-party communication, e.g. for conferences data switching systems for conference H04L12/18; arrangements for connecting several subscribers to a common circuit, i.e. affording conference facilities H04M3/56; television conferencing systems H04N7/15
• CPC Class Codes: H04L 65/4038 with floor control
• CPC Class Codes: H04L 67/131 Protocols for games, networked simulations or virtual reality
• CPC Class Codes: H04N 13/332 Displays for viewing with the aid of special glasses or head-mounted displays [HMD]
• CPC Class Codes: H04N 7/15 Conference systems
• CPC Class Codes: H04N 7/157 defining a virtual conference space and using avatars or agents computer conference optimisation or adaptation H04L12/1827

Abstract

Embodiments disclosed herein provide methods, systems, and computer readable storage media for facilitating enhanced visual communication between remote participants. In a particular embodiment, a method provides, during a conference session, receiving first video captured of a first participant remote to a physical meeting location. The method further provides determining a first position for the first participant in the physical meeting location and determining a view of an Augmented Reality (AR) head mounted display worn by a second participant at the physical meeting location. From the first video, the method provides generating AR video that makes the first participant appear to the second participant at the first position when viewed through the AR head mounted display. The method also includes transferring the AR video to the AR head mounted display for presentation to the second participant.

Claims

1. A non-transitory computer readable storage medium having instructions stored thereon for facilitating enhanced visual communication between remote participants that, when executed by a conferencing system, direct the conferencing system to:

during a conference session, receive first video captured of a first participant remote to a physical meeting location;

identify one or more available physical participant positions in the physical meeting location, wherein an available physical participant position comprises a position at the physical meeting location where a participant at the physical meeting location could be positioned but is not when determining the first position;

determine a first position for the first participant from the one or more available physical participant positions in the physical meeting location;

determine a view of an Augmented Reality (AR) head mounted display worn by a second participant at the physical meeting location;

from the first video, generate AR video that makes the first participant appear to the second participant as though the first participant is physically at the first position when viewed through the AR head mounted display; and

transfer the AR video to the AR head mounted display for presentation to the second participant.

2. The non-transitory computer readable medium of claim 1, wherein the instructions further direct the conferencing system to:

receive second video captured of the physical meeting location;

receive a second position in the physical meeting location for a third participant remote to the physical meeting location;

from the first video and the second video, generate Virtual Reality (VR) video that presents the physical meeting room with the first participant at the first position from a view point of the second position; and

transfer the VR video to a VR head mounted display worn by the third participant for presentation to the third participant.

3. The non-transitory computer readable medium of claim 2, wherein the instructions that direct the conferencing system to generate the AR video include instructions that direct the conferencing system to make a representation of the third participant appear to the second participant at the second position when viewed through the AR head mounted display.

4. The non-transitory computer readable medium of claim 1, wherein the instructions further direct the conferencing system to:

receive first audio captured of the first participant in conjunction with the first video;

transfer the first audio to an audio device at the physical meeting location, wherein the audio device presents the first audio such that the second participant hears audio as coming substantially from the first position.

5. The non-transitory computer readable medium of claim 1, wherein the instructions that direct the conferencing system to determine the view of the AR head mounted display comprise instructions that direct the conferencing system to:

receive orientation information from the AR head mounted display;

receive position information about the second participant from sources external to the AR head mounted display;

identify the view of the AR head mounted display based on the orientation information and the position information.

6. The non-transitory computer readable medium of claim 5, wherein the instructions that direct the conferencing system to determine the view of the AR head mounted display further comprise instructions that direct the conferencing system to:

receive subsequent orientation information and subsequent position information; and

update the view of the AR head mounted display based on the subsequent orientation information and the subsequent position information.

7. The non-transitory computer readable medium of claim 1, wherein the instructions further direct the conferencing system to:

record the conference session from at least one of the view of the AR head mounted display and another view of the physical meeting location.

8. The non-transitory computer readable medium of claim 7, wherein the instructions further direct the conferencing system to:

translate the recording into a format viewable on a non-head mounted display.

9. The non-transitory computer readable medium of claim 1, wherein the instructions that direct the conferencing system to generate the AR video comprise instructions that direct the conferencing system to:

overlay additional visual information associated with the conference session such that the additional visual information appears to the second participant at a position within the physical meeting location when viewed through the AR head mounted display.

10. A conferencing system for facilitating enhanced visual communication between remote participants, the conferencing system comprising:

a communication interface configured to, during a conference session, receive first video captured of a first participant remote to a physical meeting location;

processing circuitry configured to identify one or more available physical participant positions in the physical meeting location, wherein an available physical participant position comprises a position at the physical meeting location where a participant at the physical meeting location could be positioned but is not when determining the first position, determine a first position for the first participant from the one or more available physical participant positions in the physical meeting location, determine a view of an Augmented Reality (AR) head mounted display worn by a second participant at the physical meeting location, and, from the first video, generate AR video that makes the first participant appear to the second participant as though the first participant is physically at the first position when viewed through the AR head mounted display; and

the communication interface configured to transfer the AR video to the AR head mounted display for presentation to the second participant.

11. The conferencing system of claim 10, further comprising:

the communication interface configured to receive second video captured of the physical meeting location;

the processing circuitry configured to determine a second position in the physical meeting location for a third participant remote to the physical meeting location and, from the first video and the second video, generate Virtual Reality (VR) video that presents the physical meeting room with the first participant at the first position from a view point of the second position; and

the communication interface configured to transfer the VR video to a VR head mounted display worn by the third participant for presentation to the third participant.

12. The conferencing system of claim 11, wherein the processing circuitry configured to generate the AR video comprises the processing circuitry configured to make a representation of the third participant appear to the second participant at the second position when viewed through the AR head mounted display.

13. The conferencing system of claim 10, further comprising:

the communication interface configured to receive first audio captured of the first participant in conjunction with the first video and transfer the first audio to an audio device at the physical meeting location, wherein the audio device presents the first audio such that the second participant hears audio as coming substantially from the first position.

14. The conferencing system of claim 10, wherein the processing circuitry configured to determine the view of the AR head mounted display comprises the processing circuitry configured to:

receive orientation information from the AR head mounted display via the communication interface;

receive position information about the second participant from sources external to the AR head mounted display via the communication interface;

identify the view of the AR head mounted display based on the orientation information and the position information.

15. The conferencing system of claim 14, wherein the processing circuitry configured to determine the view of the AR head mounted display comprises the processing circuitry configured to:

receive subsequent orientation information and subsequent position information via the communication interface; and

update the view of the AR head mounted display based on the subsequent orientation information and the subsequent position information.

16. The conferencing system of claim 10, further comprising:

a storage system configured to record the conference session from at least one of the view of the AR head mounted display and another view of the physical meeting location.

17. The conferencing system of claim 16, further comprising:

translating the recording into a format viewable on a non-head mounted display.

18. The conferencing system of claim 10, wherein the processing circuitry configured to generate the AR video comprises the processing circuitry configured to:

overlay additional visual information associated with the conference session such that the additional visual information appears to the second participant at a position within the physical meeting location when viewed through the AR head mounted display.