IDENTIFICATION OF AUGMENTED REALITY IMAGE DISPLAY POSITION

• US 20190324277A1
• Filed: 06/28/2019
• Published: 10/24/2019
• Est. Priority Date: 06/13/2016
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Kohler, Jeffrey, Wright, Shawn Crispin, McManus, Michael W., Ambrus, Anthony, Krauskopf, Korey
• Granted Patent: US 10,670,868 B2
• Time in Patent Office: Days
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/0093 with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking
• CPC Class Codes: G02B 27/0172 characterised by optical features
• CPC Class Codes: G06F 1/163 Wearable computers, e.g. on a belt
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883

Abstract

A head-mounted display (HMD) device may include an at least partially see-through display, one or more location sensors, a communication unit, and a processor. The HMD device may determine at least a location of the one or more location sensors in a physical environment. The communication unit may receive peer location data from at least one peer HMD device in the physical environment. The processor of the HMD device may determine a field of view of the HMD device, identify a display position at which the augmented reality image can be displayed and viewed in the physical environment by the HMD device and the at least one peer HMD device, and display the augmented reality image at the display position on the at least partially see-through display of the HMD device.

Claims

1. A head-mounted display (HMD) device comprising:

a display configured to display an image representing a virtual object, the virtual object having at least a first portion and a second portion;

one or more location sensors from which the HMD device determines at least a location of the HMD device in a physical environment;

a communication unit configured to receive peer location data from at least one peer HMD device in the physical environment, wherein the peer location data indicates a peer field of view of the at least one peer HMD device; and

a processor configured to;

identify a display position at which the virtual object is to be displayed and viewed in the physical environment by the HMD device and the at least one peer HMD device;

determine an absolute focal point that is world-locked with respect to the HMD device and the at least one peer HMD device based on the display position, the location of the HMD device, and the peer location data;

assign a first relative focal point to the HMD device based at least on the display position and the location of the HMD device;

assign a second relative focal point to the at least one peer HMD device based at least on the display position and the peer location data;

orient the first portion of the virtual object at the display position based on the absolute focal point with respect to the HMD device and the at least one peer HMD device;

orient the second portion of the virtual object at the display position based on the first and second relative focal points respectively with respect to the HMD device and the at least one peer HMD device; and

display the image of the virtual object at the display position on the display of the HMD device.

2. The HMD device of claim 1, wherein the peer location data includes at least one of surface data about surfaces and candidate positions at which the image of the virtual object is to be displayed and viewed in the physical environment by the HMD device.

3. The HMD device of claim 1, wherein the communication unit is further configured to transmit an enumeration request to determine if one or more proximal HMD devices are in the physical environment;

and

receive an enumeration response from the one or more proximal HMD devices indicating that the one or more proximal HMD devices are located in the physical environment.

4. The HMD device of claim 3, wherein the processor is further configured to identify the at least one peer HMD device from one or more proximal HMD devices based on one or more logical constraints.

5. The HMD device of claim 4, wherein the one or more logical constraints include at least one of:

the one or more proximal HMD devices being within a predetermined distance of the location of the HMD device, the one or more proximal HMD devices being within a predetermined area, and the one or more proximal HMD devices belonging to a predetermined group.

6. The HMD device of claim 1, wherein the communication unit is configured to transmit location data about the HMD device to the at least one peer HMD device.

7. The HMD device of claim 1, wherein the processor is further configured to:

identify candidate positions at which the image of the virtual object is to be displayed and viewed in the physical environment by the HMD device.

8. The HMD device of claim 7, wherein the peer location data comprises peer candidate positions at which the image of the virtual object is to be displayed and viewed in the physical environment by the at least one peer HMD device.

9. The HMD device of claim 8, wherein the processor is further configured to:

identify compatible positions at which the image of the virtual object is to be displayed and viewed in the physical environment by the HMD device and the at least one peer HMD device by comparing peer candidate positions to the candidate positions and determining overlapping positions visible by both the HMD device and the at least one peer HMD device; and

select the display position from the compatible positions based on one or more logical constraints.

10. The HMD device of claim 1, further comprising an optical sensor configured to acquire surface data about surfaces.

11. The HMD device of claim 10, wherein the peer location data comprises peer surface data about surfaces within the peer field of view of the at least one peer HMD device.

12. The HMD device of claim 11, wherein the processor is further configured to:

identify compatible positions at which the image of the virtual object is to be displayed and viewed in the physical environment by the HMD device and the at least one peer HMD device by creating a union of the surface data and the peer surface data and determining overlapping positions visible by both the HMD device and the at least one peer HMD device; and

select the display position from the compatible positions based on one or more logical constraints.

13. The HMD device of claim 1, wherein the image is a virtual presenter, and the processor is further configured to:

animate the virtual presenter to perform a script, wherein the script comprises gestures and focal point information about the absolute focal point and the first and the second relative focal points, and the gestures of the virtual presenter depend at least on the focal point information.

14. The HMD device of claim 1, wherein the image is a virtual presenter, and the processor is further configured to:

display a presentation image at a presentation position, the presentation position being based on the location of the HMD device, the display position, and an area occupied by the virtual presenter; and

determine a plurality of focal points based at least on one of;

the display position, the presentation position, the location of the HMD device, and the peer location data; and

animate the virtual presenter to perform a script, wherein the script comprises gestures and focal point information about the plurality of focal points, and the gestures of the virtual presenter depend at least on the focal point information.

15. The HMD device of claim 1, wherein the communication unit is further configured to determine that an additional HMD device has entered the physical environment, receive additional location data from the additional HMD device, and wherein the processor is further configured to transmit an instruction to the additional HMD device indicating an unoccupied position nearest to the additional HMD device at which the image of the virtual object displayed at the display position is to be viewed by the additional HMD device.

16. A method implemented at least in part by a head-mounted display (HMD) device, the method comprising:

determining a location of the HMD device, via one or more location sensors, in a physical environment;

receiving peer location data from at least one peer HMD device in the physical environment via a communication unit of the HMD device, wherein the peer location data indicates a peer field of view of the at least one peer HMD device;

with a processor;

identifying a display position at which an image representing a virtual object, the virtual object having at least a first portion and a second portion, is to be displayed and viewed in the physical environment by the HMD device and the at least one peer HMD device with the processor;

determining an absolute focal point that is world-locked with respect to the HMD device and the at least one peer HMD device based on the display position, the location of the HMD device, and the peer location data;

assigning a first relative focal point to the HMD device based at least on the display position and the location of the HMD device;

assigning a second relative focal point to the at least one peer HMD device based at least on the display position and the peer location data;

orienting the first portion of the virtual object at the display position based on the absolute focal point with respect to the HMD device and the at least one peer HMD device;

orienting the second portion of the virtual object at the display position based on the first and second relative focal points respectively with respect to the HMD device and the at least one peer HMD device; and

displaying the image of the virtual object at the display position on a display of the HMD device.

17. The method of claim 16, wherein the image is a virtual presenter, the method further comprising, with the processor:

animating the virtual presenter to perform a script, wherein the script comprises gestures and focal point information about the absolute focal point and the first and the second relative focal points, and the gestures of the virtual presenter depend at least on the focal point information.

18. The method of claim 16, wherein the image is a virtual presenter, the method further comprising, with the processor:

based at least on the location of the HMD device, the display position, and an area occupied by the virtual presenter, identifying a presentation position at which a presentation image is to be displayed and viewed in the physical environment by the HMD device with the virtual presenter;

determining a plurality of focal points based at least on one of;

the display position, the presentation position, the location of the HMD device, and the peer location data;

displaying the presentation image at the presentation position on the display of the HMD device; and

animating the virtual presenter to perform a script, wherein the script comprises gestures and focal point information about the plurality of focal points, and the gestures of the virtual presenter depend at least on the focal point information.

19. A server to control image localization for multiple head-mounted display (HMD) devices, comprising:

memory to store a representation of a physical environment;

a processor configured to;

receive location data from the HMD devices in the physical environment;

based at least on the location data, identify a display position at which an image representing a virtual object is to be displayed and viewed in the physical environment by all of the HMD devices, the virtual object having at least a first portion and a second portion;

determine an absolute focal point that is world-locked with respect to the HMD devices based on the display position and the location data including locations of the HMD devices;

assign a first relative focal point to a first HMD device of the multiple HMD devices based at least on the display position and a location of the first HMD device;

assign a second relative focal point to a second HMD device of the multiple HMD devices based at least on the display position and a location of the second HMD device;

orient the first portion of the virtual object at the display position based on the absolute focal point with respect to the HMD devices;

orient the second portion of the virtual object at the display position based on the first and second relative focal points respectively with respect to the first HMD device and the second HMD device; and

propagate the display position back to the HMD devices to cause display of the image of the virtual object at the display position on a display of each of the HMD devices.

20. The server of claim 19, wherein the image is a virtual presenter;

and the processor is further configured to;

based at least on the location data, the display position, and an area occupied by the virtual presenter, identify a plurality of presentation positions at which a presentation image can be displayed and viewed in the physical environment by each of the HMD devices with the virtual presenter;

determine a plurality of focal points based at least on one of;

the display position, the plurality of presentation positions and the location data;

create an animation of the virtual presenter to perform a script, wherein the script comprises gestures and focal point information about the plurality of focal points, and the gestures of the virtual presenter depend at least on the focal point information; and

propagate the plurality of presentation positions back to each of the HMD devices to cause display of the presentation image at one of the plurality of presentation positions on the display of each of the HMD devices; and

propagate the animation back to each of the HMD devices to cause animation of the virtual presenter on the display of each of the HMD devices.