DIRECT HOLOGRAM MANIPULATION USING IMU

• US 20140152558A1
• Filed: 11/30/2012
• Published: 06/05/2014
• Est. Priority Date: 11/30/2012
• Status: Abandoned Application

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Robert L. Crocco Jr
• Inventors: Sugden, Ben J., Crocco, Robert L. Jr., Keane, Brian E., Miles, Christopher E., Massey, Laura K., Kipman, Alex Aben-Athar, Salter, Tom, Deptford, Daniel
• Application Number:
  US13/691,445
• Publication Number:
  US 20140152558A1
• Time in Patent Office: Days
• US Class Current: 345/157
• CPC Class Codes: G02B 2027/0178 Eyeglass type eyeglass details G02C
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G09G 5/377 for mixing or overlaying two or more graphic patterns G09G5/02, G09G5/397 take precedence

Abstract

Methods for controlling an augmented reality environment associated with a head-mounted display device (HMD) are described. In some embodiments, a virtual pointer may be displayed to an end user of the HMD and controlled by the end user using motion and/or orientation information associated with a secondary device (e.g., a mobile phone). Using the virtual pointer, the end user may select and manipulate virtual objects within the augmented reality environment, select real-world objects within the augmented reality environment, and/or control a graphical user interface of the HMD. In some cases, the initial position of the virtual pointer within the augmented reality environment may be determined based on a particular direction in which the end user is gazing and/or a particular object at which the end user is currently focusing on or has recently focused on.

Claims

1. A method for controlling an augmented reality environment associated with an HMD, comprising:

detecting a triggering event corresponding with a virtual pointer mode of the HMD;

determining an initial virtual pointer location in response to the detecting a triggering event;

acquiring orientation information from a secondary device in communication with the HMD;

updating the virtual pointer location based on the orientation information; and

displaying a virtual pointer within the augmented reality environment corresponding with the virtual pointer location.

2. The method of claim 1, wherein:

the determining an initial virtual pointer location includes determining a gaze direction associated with an end user of the HMD and setting the initial virtual pointer location based on the gaze direction.

3. The method of claim 1, wherein:

the determining an initial virtual pointer location includes determining a gaze direction associated with an end user of the HMD, identifying one or more selectable objects within a field of view of the HMD, determining a selectable object of the one or more selectable objects closest to the gaze direction, and setting the initial virtual pointer location based on a location of the selectable object within the augmented reality environment.

4. The method of claim 3, further comprising:

providing feedback to the end user if the virtual pointer location corresponds with one or more regions within the augmented reality environment associated with the one or more selectable objects.

5. The method of claim 4, wherein:

the feedback includes a vibration of the secondary device.

6. The method of claim 1, further comprising:

determining whether a change in orientation of the secondary device is within a threshold range based on the orientation information, the updating the virtual pointer location is performed in response to the change in orientation of the secondary device being within the threshold range.

7. The method of claim 1, further comprising:

determining an initial orientation associated with the secondary device prior to the acquiring orientation information, the acquiring orientation information includes receiving relative orientation information relative to the initial orientation associated with the secondary device.

8. The method of claim 1, wherein:

the detecting a triggering event includes detecting a hand gesture associated with the virtual pointer mode performed by an end user of the HMD.

9. The method of claim 1, wherein:

the secondary device comprises a mobile phone.

10. An electronic device for displaying an augmented reality environment, comprising:

a memory, the memory stores an initial orientation associated with a secondary device in communication with the electronic device;

one or more processors in communication with the memory, the one or more processors detect a triggering event corresponding with a virtual pointer mode and determine an initial virtual pointer location in response to detecting the triggering event, the one or more processors acquire orientation information from the secondary device and update the virtual pointer location based on the orientation information and the initial orientation; and

a see-through display in communication with the one or more processors, the see-through display displays the augmented reality environment including a virtual pointer corresponding with the virtual pointer location.

11. The electronic device of claim 10, wherein:

the one or more processors determine the initial virtual pointer location by determining a gaze direction associated with an end user of the electronic device and setting the initial virtual pointer location based on the gaze direction.

12. The electronic device of claim 10, wherein:

the one or more processors determine the initial virtual pointer location by determining a gaze direction associated with an end user of the electronic device, identifying one or more selectable objects within a field of view of the electronic device, determining a selectable object of the one or more selectable objects closest to the gaze direction, and setting the initial virtual pointer location based on a location of the selectable object within the augmented reality environment.

13. The electronic device of claim 12, wherein:

the one or more processors provide feedback to the end user if the virtual pointer location corresponds with one or more regions within the augmented reality environment associated with the one or more selectable objects.

14. The electronic device of claim 13, wherein:

the one or more processors highlight the selectable object within the augmented reality environment if the virtual pointer location corresponds with the location of the selectable object.

15. The electronic device of claim 10, wherein:

the one or more processors detect a triggering event by detecting a hand gesture associated with the virtual pointer mode performed by an end user of the electronic device.

16. The electronic device of claim 10, wherein:

the electronic device comprises an HMD; and

the secondary device comprises a mobile phone.

17. One or more storage devices containing processor readable code for programming one or more processors to perform a method for controlling an augmented reality environment associated with an HMD comprising the steps of:

detecting a triggering event corresponding with a virtual pointer mode of the HMD;

determining a gaze direction associated with an end user of the HMD;

determining an initial virtual pointer location based on the gaze direction;

acquiring updated orientation information from a secondary device in communication with the HMD;

updating the virtual pointer location based on the updated orientation information;

displaying a virtual pointer within the augmented reality environment corresponding with the virtual pointer location;

determining that a selection criterion has been satisfied; and

displaying an updated augmented reality environment based on the selection criterion and the virtual pointer location.

18. The one or more storage devices of claim 17, wherein:

the determining an initial virtual pointer location includes identifying one or more selectable objects within a field of view of the HMD, determining a selectable object of the one or more selectable objects closest to the gaze direction, and setting the initial virtual pointer location based on a location of the selectable object within the augmented reality environment.

19. The one or more storage devices of claim 17, wherein:

the determining that a selection criterion has been satisfied includes determining that a change in orientation satisfies the selection criterion.

20. The one or more storage devices of claim 17, wherein:

the secondary device comprises a mobile phone.