Identification of augmented reality image display position

• US 10,338,392 B2
• Filed: 01/18/2017
• Issued: 07/02/2019
• Est. Priority Date: 06/13/2016
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Kohler, Jeffrey, Wright, Shawn Crispin, McManus, Michael W., Ambrus, Anthony, Krauskopf, Korey
• Primary Examiner(s): Sheng, Xin
• Application Number:
  US15/409,472
• Publication Number:
  US 20170358140A1
• Time in Patent Office: 895 Days
• Field of Search: None
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/0093 with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking
• CPC Class Codes: G02B 27/0172 characterised by optical features
• CPC Class Codes: G06F 1/163 Wearable computers, e.g. on a belt
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883

Abstract

A head-mounted display (HMD) device may include an at least partially see-through display, one or more location sensors, a communication unit, and a processor. The HMD device may determine at least a location of the one or more location sensors in a physical environment. The communication unit may receive peer location data from at least one peer HMD device in the physical environment. The processor of the HMD device may determine a field of view of the HMD device, identify a display position at which the augmented reality image can be displayed and viewed in the physical environment by the HMD device and the at least one peer HMD device, and display the augmented reality image at the display position on the at least partially see-through display of the HMD device.

Claims

1. A head-mounted display (HMD) device comprising:

an at least partially see-through display configured to display an augmented reality image representing a virtual object, the virtual object having at least a first portion, a second portion, and an associated script for animating the virtual object;

one or more location sensors from which the HMD device determines at least a location of the HMD device in a physical environment;

a communication unit configured to receive peer location data from at least one peer HMD device in the physical environment, wherein the peer location data indicates a peer field of view of the at least one peer HMD device, the peer field of view comprising a range within which augmented reality images may be presented; and

a processor configured to;

based at least on the received peer location data, identify a display position at which the virtual object is to be displayed and viewed in the physical environment by the HMD device and the at least one peer HMD device;

determine a shared focal point based on the display position, the location of the HMD device, and the peer location data;

determine a first orientation of the first portion of the virtual object at the display position based on the shared focal point and focal point information of the script, the first orientation being common to the HMD device and the at least one peer HMD device;

determine a second orientation of the second portion of the virtual object at the display position based on the focal point information and a device-specific focal point that corresponds to the location of the HMD device relative to the display position, the second orientation being specific to the location of the HMD device and differing from the at least one peer HMD device; and

display and animate the augmented reality image of the virtual object at the display position with the first portion at the first orientation and the second portion at the second orientation on the at least partially see-through display of the HMD device according to the script.

2. The HMD device of claim 1, wherein peer location data includes at least one of surface data about surfaces and candidate positions at which the augmented reality image can be displayed and viewed in the physical environment by the HMD device.

3. The HMD device of claim 1, wherein the communication unit is further configured to transmit an enumeration request to determine if one or more proximal HMD devices are in the physical environment;

and

receive an enumeration response from the one or more proximal HMD devices indicating that the one or more proximal HMD devices are located in the physical environment.

4. The HMD device of claim 3, wherein the processor is further configured to identify the at least one peer HMD device from one or more proximal HMD devices based on one or more logical constraints.

5. The HMD device of claim 4, wherein the one or more logical constraints include at least one of:

the one or more proximal HMD devices being within a predetermined distance of the determined location, the one or more proximal HMD devices being within a predetermined area, and the one or more proximal HMD devices belonging to a predetermined group.

6. The HMD device of claim 1, wherein the communication unit is configured to transmit location data about the HMD device to the at least one peer HMD device.

7. The HMD device of claim 1, wherein the processor is further configured to:

identify candidate positions at which the augmented reality image can be displayed and viewed in the physical environment by the HMD device.

8. The HMD device of claim 7, wherein the received peer location data comprises peer candidate positions at which the augmented reality image can be displayed and viewed in the physical environment by the at least one peer HMD device.

9. The HMD device of claim 8, wherein the processor is further configured to:

identify compatible positions at which the augmented reality image can be displayed and viewed in the physical environment by the HMD device and the at least one peer HMD device by comparing peer candidate positions to the candidate positions and determining overlapping positions visible by both the HMD device and the at least one peer HMD device; and

select the display position from the compatible positions based on one or more logical constraints.

10. The HMD device of claim 1, further comprising an optical sensor configured to acquire surface data about surfaces.

11. The HMD device of claim 10, wherein the received peer location data comprises peer surface data about surfaces within the peer field of view of the at least one peer HMD device.

12. The HMD device of claim 11, wherein the processor is further configured to:

identify compatible positions at which the augmented reality image can be displayed and viewed in the physical environment by the HMD device and the at least one peer HMD device by creating a union of the surface data and the received peer surface data and determining overlapping positions visible by both the HMD device and the at least one peer HMD device; and

select the display position from the compatible positions based on one or more logical constraints.

13. The HMD device of claim 1, wherein the augmented reality image is a virtual presenter, and the processor is further configured to:

determine a plurality of focal points based at least on one of;

the display position, the location of the HMD device, and the peer location data; and

animate the virtual presenter to perform the script, wherein the script comprises gestures and focal point information about the plurality of focal points, and the gestures of the virtual presenter depend at least on the focal point information.

14. The HMD device of claim 1, wherein the augmented reality image is a virtual presenter, and the processor is further configured to:

display a presentation image at a presentation position, the presentation position being based on the location of the HMD device, the display position, and an area occupied by the virtual presenter; and

determine a plurality of focal points based at least on one of;

the display position, the presentation position, the location of the HMD device, and the peer location data; and

animate the virtual presenter to perform the script, wherein the script comprises gestures and focal point information about the plurality of focal points, and the gestures of the virtual presenter depend at least on the focal point information.

15. The HMD device of claim 1, wherein the communication unit is further configured to determine that an additional HMD device has entered the physical environment, receive additional location data from the additional HMD device, and wherein the processor is further configured to transmit an instruction to the additional HMD device indicating an unoccupied position nearest to the additional HMD device at which the augmented reality image displayed at the display position can be viewed by the additional HMD device.

16. A method implemented at least in part by a head-mounted display (HMD) device, the method comprising:

determining a location of the HMD device in a physical environment;

receiving peer location data from at least one peer HMD device in the physical environment via a communication unit of the HMD device, wherein the peer location data indicates a peer field of view of the at least one peer HMD device, the peer field of view comprising a range within which augmented reality images may be presented;

with a processor of the HMD device;

based at least on the received peer location data, identifying a display position at which an augmented reality image representing a virtual object, the virtual object having at least a first portion, a second portion, and an associated script for animating the virtual object, is to be displayed and viewed in the physical environment by the HMD device and the at least one peer HMD device with the processor;

determining a shared focal point based on the display position, the location of the HMD device, and the peer location data;

determining a first orientation of the first portion of the virtual object at the display position based on the shared focal point and focal point information of the script, the first orientation being common to the HMD device and the at least one peer HMD device;

determining a second orientation of the second portion of the virtual object at the display position based on the focal point information and a device-specific focal point that corresponds to the location of the HMD device relative to the display position, the second orientation being specific to the location of the HMD device and differing from the at least one peer HMD device; and

displaying and animating the augmented reality image of the virtual object at the display position with the first portion at the first orientation and the second portion at the second orientation on an at least partially see-through display of the HMD device according to the script.

17. The method of claim 16, wherein the augmented reality image is a virtual presenter, the method further comprising:

determining, with the processor of the HMD device, a plurality of focal points based at least on one of;

the display position, the location of the HMD device, and the peer location data; and

animating the virtual presenter to perform the script, wherein the script comprises gestures and focal point information about the plurality of focal points, and the gestures of the virtual presenter depend at least on the focal point information.

18. The method of claim 16, wherein the augmented reality image is a virtual presenter, the method further comprising:

based at least on the location of the HMD device, the display position, and an area occupied by the virtual presenter, identifying a presentation position at which a presentation image can be displayed and viewed in the physical environment by the HMD device with the virtual presenter;

determining, with the processor of the HMD device, a plurality of focal points based at least on one of;

the display position, the presentation position, the location of the HMD device, and the peer location data;

displaying the presentation image at the presentation position on the at least partially see-through display of the HMD device; and

animating the virtual presenter to perform the script, wherein the script comprises gestures and focal point information about the plurality of focal points, and the gestures of the virtual presenter depend at least on the focal point information.

19. A server to control augmented reality image localization for multiple head-mounted display (HMD) devices, comprising:

memory to store a representation of a physical environment;

a processor configured to;

receive location data from the HMD devices in the physical environment, the location data comprising a range within which augmented reality images may be presented;

based at least on the received location data, identify a display position at which an augmented reality image representing a virtual object is to be displayed and viewed in the physical environment by all of the HMD devices, the virtual object having at least a first portion, a second portion, and an associated script for animating the virtual object;

determine a shared focal point of the HMD devices based on the display position and the location data;

determine a first orientation of the first portion of the virtual object at the display position based on the shared focal point and focal point information of the script, the first orientation being common to the HMD devices;

determine a second orientation of the second portion of the virtual object at the display position based on the focal point information and a device-specific focal point that corresponds to the location of a first HMD device relative to the display position, the second orientation being specific to the location of the first HMD device and differing from other HMD devices; and

propagate the display position back to the HMD devices to cause display and animation of the augmented reality image of the virtual object at the display position with the first portion at the first orientation and the second portion at the second orientation on an at least partially see-through display of the first HMD device according to the script.

20. The server of claim 19, wherein the augmented reality image is a virtual presenter;

and

the processor is further configured to;

based at least on the location data, the display position, and an area occupied by the virtual presenter, identify a plurality of presentation positions at which a presentation image can be displayed and viewed in the physical environment by each of the HMD devices with the virtual presenter;

determine a plurality of focal points based at least on one of;

the display position, the plurality of presentation positions and the location data;

create an animation of the virtual presenter to perform the script, wherein the script comprises gestures and focal point information about the plurality of focal points, and the gestures of the virtual presenter depend at least on the focal point information; and

propagate the plurality of presentation positions back to each of the HMD devices to cause display of the presentation image at one of the plurality of presentation positions on the at least partially see-through display of each of the HMD devices; and

propagate the animation back to each of the HMD devices to cause animation of the virtual presenter on the at least partially see-through display of each of the HMD devices.