HOLOGRAPHIC OBJECT FEEDBACK

• US 20140306891A1
• Filed: 04/12/2013
• Published: 10/16/2014
• Est. Priority Date: 04/12/2013
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Latta, Stephen G., Poulos, Adam G., Brown, Cameron G., McCulloch, Daniel J., Kaplan, Matthew, Navratil, Arnulfo Zepeda, Tsunoda, Kudo, Paulovich, Jon
• Granted Patent: US 9,367,136 B2
• Time in Patent Office: Days
• US Class Current: 345/158
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0178 Eyeglass type eyeglass details G02C
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/016 Input arrangements with force or tactile feedback as computer generated output to the user
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883

Abstract

Methods for providing real-time feedback to an end user of a mobile device as they are interacting with or manipulating one or more virtual objects within an augmented reality environment are described. The real-time feedback may comprise visual feedback, audio feedback, and/or haptic feedback. In some embodiments, a mobile device, such as a head-mounted display device (HMD), may determine an object classification associated with a virtual object within an augmented reality environment, detect an object manipulation gesture performed by an end user of the mobile device, detect an interaction with the virtual object based on the object manipulation gesture, determine a magnitude of a virtual force associated with the interaction, and provide real-time feedback to the end user of the mobile device based on the interaction, the magnitude of the virtual force applied to the virtual object, and the object classification associated with the virtual object.

Claims

1. An electronic device for providing feedback associated with virtual object interactions, comprising:

one or more processors, the one or more processors acquire a first virtual object and determine an object classification associated with the first virtual object, the one or more processors detect an object manipulation gesture performed by an end user of the electronic device and detect an interaction with the first virtual object based on the object manipulation gesture, the one or more processors determine a magnitude of a virtual force associated with the interaction, the one or more processors determine a feedback response to the interaction based on the magnitude of the virtual force and the object classification, the one or more processors output the feedback response; and

a see-through display in communication with the one or more processors, the see-through display displays one or more images of the first virtual object such that the first virtual object is perceived to exist within the augmented reality environment.

2. The electronic device of claim 1, wherein:

the one or more processors determine a material property associated with the first virtual object, the one or more processors determine the feedback response based on the magnitude of the virtual force and the material property.

3. The electronic device of claim 1, wherein:

the one or more processors determine a region within the augmented reality environment associated with the first virtual object, the one or more processors detect the interaction with the first virtual object by detecting the object manipulation gesture being performed within the region, the region extends beyond the first virtual object by a particular distance.

4. The electronic device of claim 1, wherein:

the feedback response includes a haptic response, the haptic response comprises a vibrating of the electronic device.

5. The electronic device of claim 1, wherein:

the one or more processors determine the feedback response by determining whether a hand of the end user of the electronic device is facing away from the electronic device, the feedback response includes a visual response, the visual response includes displaying a first portion of the first virtual object and not displaying a second portion of the first virtual object occluded by the hand of the end user.

6. The electronic device of claim 5, wherein:

the visual response includes shifting a position of the first virtual object within the augmented reality environment such that the first portion appears between a thumb and index finger of the end user'"'"'s hand.

7. The electronic device of claim 1, wherein:

the feedback response includes a haptic response, the haptic response comprises a vibrating of a second electronic device different from the electronic device at a particular magnitude that is proportional to the magnitude of the virtual force.

8. A method for providing feedback associated with virtual object interactions, comprising:

acquiring one or more virtual objects associated with an augmented reality environment, the one or more virtual objects include a first virtual object;

determining an object classification associated with the first virtual object;

detecting an object manipulation gesture;

detecting an interaction with the first virtual object based on the object manipulation gesture;

determining a magnitude of a virtual force associated with the interaction;

determining a feedback response to the interaction based on the magnitude of the virtual force and the object classification;

outputting the feedback response;

rendering one or more images corresponding with the first virtual object; and

displaying the one or more images on a mobile device.

9. The method of claim 8, further comprising:

determining a material property associated with the first virtual object, the determining a feedback response to the interaction includes determining the feedback response based on the magnitude of the virtual force and the material property.

10. The method of claim 8, further comprising:

determining a region within the augmented reality environment associated with the first virtual object, the detecting an interaction with the first virtual object includes detecting the object manipulation gesture being performed within the region.

11. The method of claim 10, wherein:

the region extends beyond the first virtual object by a particular distance.

12. The method of claim 8, wherein:

the determining a feedback response includes determining whether a hand of an end user of the mobile device is facing away from the mobile device, the feedback response includes a visual response, the visual response includes displaying a first portion of the first virtual object and not displaying a second portion of the first virtual object that is occluded by the hand of the end user.

13. The method of claim 12, wherein:

the visual response includes shifting a position of the first virtual object within the augmented reality environment such that the first portion appears between a thumb and index finger of the end user'"'"'s hand.

14. The method of claim 8, wherein:

the feedback response includes a haptic response, the haptic response comprises a vibrating of an electronic device at a particular magnitude that is proportional to the magnitude of the virtual force.

15. The method of claim 8, wherein:

the object classification is associated with metallic virtual objects.

16. The method of claim 8, wherein:

the mobile device comprises an HMD;

the acquiring one or more virtual objects is performed by the HMD;

the detecting an interaction with the first virtual object is performed by the HMD; and

the determining a feedback response to the interaction is performed by the HMD.

17. One or more storage devices containing processor readable code for programming one or more processors to perform a method for providing feedback associated with virtual object interactions comprising the steps of:

acquiring a first virtual object and a second virtual object associated with an augmented reality environment;

determining an object classification associated with the first virtual object and the second virtual object;

detecting an object manipulation gesture within the augmented reality environment;

detecting an interaction with the first virtual object based on the object manipulation gesture;

determining a magnitude of a virtual force associated with the interaction;

determining a feedback response to the interaction based on the magnitude of the virtual force and the object classification;

outputting the feedback response, the feedback response includes at least one of a visual response, an audio response, or a haptic response;

rendering one or more images corresponding with the first virtual object; and

displaying the one or more images on a mobile device.

18. The one or more storage devices of claim 17, further comprising:

determining a material property associated with the first virtual object, the determining a feedback response to the interaction includes determining the feedback response based on the material property, the audio response comprises a sound effect associated with the material property.

19. The one or more storage devices of claim 17, further comprising:

determining a region within the augmented reality environment associated with the first virtual object, the detecting an interaction with the first virtual object includes detecting the object manipulation gesture being performed within the region, the region extends beyond the first virtual object by a particular distance.

20. The one or more storage devices of claim 17, wherein:

the determining a feedback response includes determining whether a hand of an end user of the mobile device is facing away from the mobile device, the visual response includes displaying a first portion of the first virtual object and not displaying a second portion of the first virtual object that is occluded by the hand of the end user.