Control of enhanced communication between remote participants using augmented and virtual reality

• US 10,013,805 B2
• Filed: 01/21/2015
• Issued: 07/03/2018
• Est. Priority Date: 01/24/2014
• Status: Active Grant

• Current Assignee: Avaya Incorporated
• Original Assignee: Avaya Incorporated
• Inventors: Barzuza, Tamar, Wiener, Yair, Modai, Ori
• Primary Examiner(s): Patel, Hemant
• Application Number:
  US14/601,505
• Publication Number:
  US 20150215351A1
• Time in Patent Office: 1,259 Days
• Field of Search: 348 1401- 1416
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0178 Eyeglass type eyeglass details G02C
• CPC Class Codes: G02B 27/01 Head-up displays
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: H04L 12/18 for broadcast or conference , e.g. multicast
• CPC Class Codes: H04L 12/1827 Network arrangements for conference optimisation or adaptation
• CPC Class Codes: H04L 65/403 Arrangements for multi-party communication, e.g. for conferences data switching systems for conference H04L12/18; arrangements for connecting several subscribers to a common circuit, i.e. affording conference facilities H04M3/56; television conferencing systems H04N7/15
• CPC Class Codes: H04L 65/4038 with floor control
• CPC Class Codes: H04L 67/131 Protocols for games, networked simulations or virtual reality
• CPC Class Codes: H04N 13/332 Displays for viewing with the aid of special glasses or head-mounted displays [HMD]
• CPC Class Codes: H04N 7/15 Conference systems
• CPC Class Codes: H04N 7/157 defining a virtual conference space and using avatars or agents computer conference optimisation or adaptation H04L12/1827

Abstract

Embodiments disclosed herein provide methods, systems, and computer readable storage media for facilitating controlling a meeting between remote Augmented Reality (AR) and Virtual Reality (VR) participants. In a particular embodiment, a method provides, during a conference session wherein a plurality of head mounted displays (HMDs) are each worn by one of a plurality of participants, compiling first orientation information and first view information from a first HMD of the plurality of HMDs. The orientation information describes current orientation of the first HMD. Based on the first orientation information and the first view information, the method provides determining a first change to how video is presented the first HMD. Also, the method provides directing the first HMD to implement the first change.

Claims

1. A non-transitory computer readable storage medium having instructions stored thereon that, when executed by a conferencing system, direct the conferencing system to perform a method for controlling a meeting between remote Augmented Reality (AR) and Virtual Reality (VR) participants, the method comprising:

during a conference session wherein a plurality of head mounted displays (HMDs) are each worn by respective ones of a plurality of participants, compiling first orientation information and first view information from a first HMD of the plurality of HMDs, wherein the orientation information describes current orientation of the first HMD and the first HMD comprises an AR HMD with transparent lenses onto which AR information is displayed;

based on the first orientation information and the first view information, determining a first change to how video is presented by the first HMD, wherein determining the first change includes;

using the first view information to determine a first view of a first participant wearing the first HMD, wherein the first view includes what the first participant sees of a physical meeting room through the transparent lenses; and

determining where to place one or more elements, including video captured of one or more remote participants at locations remote to the first HMD, in the video such that, when the video is displayed onto the transparent lenses by the first HMD, the one or more elements appear to the first participant at positions in the meeting room as though the one or more elements are in the physical meeting room with the first participant; and

directing the first HMD to implement the first change.

2. The non-transitory computer readable medium of claim 1, further comprising:

determining the positions in the meeting room.

3. The non-transitory computer readable medium of claim 1, wherein determining the first change further comprises:

identifying a first object of interest in the first view;

determining one or more options for the first object;

receiving first input from the first participant selecting a first option of the one or more options; and

based on the first option and the first object, generating the one or more elements.

4. The non-transitory computer readable medium of claim 3, wherein the first object comprises a document and wherein the one or more elements comprises a video representation of the document.

5. The non-transitory computer readable medium of claim 3, wherein the first input includes an instruction to present the one or more elements in video presented by each other HMD of the plurality of HMDs other than the first HMD, and the method further comprising:

directing the other HMDs to present video of the one or more elements.

6. The non-transitory computer readable medium of claim 3, wherein the first input comprises a head nod identified from second orientation information.

7. The non-transitory computer readable medium of claim 3, wherein the first input comprises a hand gesture made by the first participant.

8. The non-transitory computer readable medium of claim 3, wherein the first input comprises input by the first participant into a peripheral device.

9. The non-transitory computer readable medium of claim 1, wherein the first orientation information indicates that a first participant wearing the first HMD is leaning forward and wherein determining the first change comprises:

determining a zoom level for the video corresponding to an amount in which the first participant is leaning forward.

10. A conferencing system for controlling a meeting between remote Augmented Reality (AR) and Virtual Reality (VR) participants, the conferencing system comprising:

a communication interface configured to, during a conference session wherein a plurality of head mounted displays (HMDs) are each worn by respective ones of a plurality of participants, compile first orientation information and first view information from a first HMD of the plurality of HMDs, wherein the orientation information describes current orientation of the first HMD and the first HMD comprises an AR HMD with transparent lenses onto which AR information is displayed;

a processing system configured to, based on the first orientation information and the first view information, determine a first change to how video is presented by the first HMD by;

using the first view information to determine a first view of a first participant wearing the first HMD, wherein the first view includes what the first participant sees of a physical meeting room through the transparent lenses; and

determining where to place one or more elements, including video captured of one or more remote participants at locations remote to the first HMD, in the video such that, when the video is displayed onto the transparent lenses by the first HMD, the one or more elements appear to the first participant at positions in the meeting room as though the one or more elements are in the physical meeting room with the first participant; and

direct the first HMD to implement the first change.

11. The conferencing system of claim 10, wherein the processing system is further configured to determine the positions in the meeting room of the first HMD for each of the one or more remote participants from the perspective of the first view.

12. The conferencing system of claim 10, wherein the processing system configured to determine the first change further comprises the processing system configured to:

identify a first object of interest in the first view;

determine one or more options for the first object;

receive first input from the first participant selecting a first option of the one or more options; and

based on the first option and the first object, generate the one or more elements.

13. The conferencing system of claim 12, wherein the first object comprises a document and wherein the one or more elements comprises a video representation of the document.

14. The conferencing system of claim 12, wherein the first input includes an instruction to present the one or more elements in video presented by each other HMD of the plurality of HMDs other than the first HMD, further comprising:

the processing system configured to direct the other HMDs to present video of the one or more elements.

15. The conferencing system of claim 12, wherein the first input comprises a head nod identified from second orientation information.

16. The conferencing system of claim 12, wherein the first input comprises a hand gesture made by the first participant.

17. The conferencing system of claim 12, wherein the first input comprises input by the first participant into a peripheral device.

18. The conferencing system of claim 10, wherein the first orientation information indicates that a first participant wearing the first HMD is leaning forward and wherein the processing system configured to determine the first change comprises:

the processing system configured to determine a zoom level for the video corresponding to an amount in which the first participant is leaning forward.