Gesture based region identification for holograms

• US 9,116,666 B2
• Filed: 06/01/2012
• Issued: 08/25/2015
• Est. Priority Date: 06/01/2012
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Salter, Tom G., Kipman, Alex Aben-Athar, Sugden, Ben J., Crocco, Robert L. Jr., Keane, Brian E., Miles, Christopher E., Perez, Kathryn Stone, Massey, Laura K., Lamb, Mathew J.
• Primary Examiner(s): Johnson, M Good
• Application Number:
  US13/487,079
• Publication Number:
  US 20130321462A1
• Time in Patent Office: 1,180 Days
• Field of Search: 345/633
• US Class Current: 1/1
• CPC Class Codes: G06F 1/163 Wearable computers, e.g. on a belt
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/147 using display panels

Abstract

Techniques are provided for allowing a user to select a region within virtual imagery, such as a hologram, being presented in an HMD. The user could select the region by using their hands to form a closed loop such that from the perspective of the user, the closed loop corresponds to the region the user wishes to select. The user could select the region by using a prop, such as a picture frame. In response to the selection, the selected region could be presented using a different rendering technique than other regions of the virtual imagery. Various rendering techniques such as zooming, filtering, etc. could be applied to the selected region. The identification of the region by the user could also serve as a selection of an element in that portion of the virtual image.

Claims

1. A method comprising:

presenting a hologram in a see-through, near-eye display device;

accessing, by the see-through, near-eye display device, depth image data of an environment that coincides with the hologram from the perspective of a user wearing the see-through, near-eye display device;

analyzing, by the see-through, near-eye display device, the depth image data to identify a region select symbol being made by the user, the region select symbol being an enclosed loop;

identifying, by the see-through, near-eye display device, a region in the depth image data that corresponds to the enclosed loop;

determining, by the see-through, near-eye display device, a portion of the hologram presented in the see-through, near-eye display device that corresponds to the enclosed loop in the depth image data from the perspective of the user; and

displaying, in the see-through, near-eye display device, the portion of the hologram that corresponds to the enclosed loop from the perspective of the user with a first display property and other portions of the hologram with a second display property.

2. The method of claim 1, further comprising:

displaying, in the see-through, near-eye display device, the entire hologram with the second display property;

receiving, by the see-through, near-eye display device while the entire hologram is being displayed with the second display property, a selection of a modification to perform to the portion of the hologram that corresponds to the enclosed loop, wherein the modification is the first display property; and

modifying, in the see-through, near-eye display device, the portion of the hologram that corresponds to the enclosed loop in accordance with the selected modification, including changing from displaying the portion of the hologram that corresponds to the enclosed loop with the second display property to displaying the portion of the hologram that corresponds to the enclosed loop with the first display property while continuing to display the rest of the hologram with the second display property.

3. The method of claim 1, further comprising:

initiating, by the see-through, near-eye display device, a select event of an element within the portion of the hologram that corresponds to the enclosed loop from the perspective of the user; and

taking an action, by the see-through, near-eye display device, based on the element being selected.

4. The method of claim 1, wherein the analyzing the depth image data to identify a region select symbol being made by the user and the analyzing the depth image data to identify a region select symbol being made by the user include:

identifying, by the see-through, near-eye display device, at least one of the user'"'"'s hands; and

determining, by the see-through, near-eye display device, that the at least one hand forms the enclosed loop.

5. The method of claim 4, wherein the identifying at least one of the user'"'"'s hands includes performing skeletal tracking based on the depth image data.

6. The method of claim 1, wherein the analyzing the depth image data to identify a region select symbol being made by the user includes:

identifying, by the see-through, near-eye display device, a prop other than a body part that forms the enclosed loop.

7. The method of claim 1, wherein the displaying, in the see-through, near-eye display device, the portion of the hologram that corresponds to the enclosed loop from the perspective of the user with the first display property comprises:

presenting an x-ray view in the see-through, near-eye display device within the enclosed loop that allows the user to see through an otherwise opaque portion of the hologram being displayed in the see-through, near-eye display device.

8. A display system comprising:

a see-through, near-eye display device;

an image sensor;

logic in communication with the display device and the image sensor, the logic;

presents holographic imagery in the see-through, near-eye display device;

accesses depth image data of an environment that coincides with the holographic imagery from the perspective of a user wearing the see-through, near-eye display device;

analyzes the depth image data to identify a region select symbol being made by the user, the region select symbol being an enclosed loop;

correlates the enclosed loop in the depth image data to a portion of the holographic imagery;

displays, in the see-through, near-eye display device, the portion of the holographic imagery that corresponds to the enclosed loop from the perspective of the user with a first display property and other portions of the holographic imagery with a second display property.

9. The display system of claim 8, wherein the logic initiates a select event in response to identifying the region select symbol being made by the user, the portion of the holographic imagery that corresponds to the enclosed loop from the perspective of the user includes an element, the select event is a selection of the element, the logic further takes an action based on the user selecting the element.

10. The display system of claim 8, wherein the logic that analyzes the depth image data to identify a region select symbol being made by the user includes logic that identifies at least one of the user'"'"'s hands forming the enclosed loop when analyzing the image data to identify the enclosed loop.

11. The display system of claim 10, wherein the logic that identifies at least one of the user'"'"'s hands forming the enclosed loop includes logic that performs skeletal tracking based on the depth image data.

12. The display system of claim 8, wherein the logic that analyzes the depth image data to identify a region select symbol being made by the user includes logic that:

identifies a prop other than a body part that forms the enclosed loop.

13. The display system of claim 8, wherein at least a portion of the logic communicates with the see-through, near-eye display device over a network.

14. The display system of claim 8, wherein the logic:

displays the entire hologram in the see-through, near-eye display device with the second display property;

receives, while the entire hologram is being displayed with the second display property, a selection of a modification to perform to the portion of the hologram that corresponds to the enclosed loop, wherein the modification is the first display property; and

modifies the display of the portion of the hologram that corresponds to the enclosed loop in accordance with the selected modification, wherein the logic changes the portion of the hologram that corresponds to the enclosed loop from the second display property to the first display property.

15. The display system of claim 8, wherein, when the logic correlates the enclosed loop in the depth image data to the portion of the holographic imagery, the logic:

accesses a location of the image sensor;

accesses a location of an eye of the user; and

reprojects the region select symbol from a perspective of the image sensor to a perspective of the eye of the user to accurately represent the user'"'"'s perspective.

16. A computer storage device having instructions stored thereon which, when executed on a processor, cause the processor to:

cause a see-through, near-eye display device to present holographic imagery, wherein the entire holographic imagery is presented with a first display property;

access depth image data of an environment that coincides with the holographic imagery from the perspective of a user wearing the see-through, near-eye display device;

analyze the depth image data to identify a region select symbol being made by the user, the region select symbol being an enclosed loop;

form an intermediate image that comprises foreground pixels and background pixels, the foreground pixels corresponding to the region select symbol;

perform a floodfill of the enclosed loop in the intermediate image to identify a region in the intermediate image that corresponds to the enclosed loop;

correlate the enclosed loop in the intermediate image to a portion of the holographic imagery;

initiate a select event in response to identifying the region select symbol being made by the user;

receive a selection of a second display property to apply to the portion of the holographic imagery; and

cause the see-through, near-eye display device to present the portion of the holographic imagery that corresponds to inside of the enclosed loop with the second display property and other portions of the holographic imagery that correspond to outside of the enclosed loop with the first display property.

17. The computer storage device of claim 16, wherein the second display property is selected from the group including:

zooming, presenting an x-ray, and presenting a view into a different world.

18. The computer storage device of claim 16, wherein the instructions that cause the processor to analyze the depth image data to identify a region select symbol being made by the user and to identify a region in the intermediate image that corresponds to the enclosed loop cause the processor to:

identify at least one of the user'"'"'s hands; and

determine that at least one hand forms the enclosed loop.

19. The computer storage device of claim 16, wherein the instructions that cause the processor to identify at least one of the user'"'"'s hands cause the processor to:

perform skeletal tracking based on the depth image data.

20. The computer storage device of claim 16, wherein the instructions that cause the processor to analyze the depth image data to identify a region select symbol being made by the user cause the processor to:

identify a prop other than a body part that forms the enclosed loop.