DYNAMIC HAPTIC RETARGETING5

• US 20170309071A1
• Filed: 04/21/2016
• Published: 10/26/2017
• Est. Priority Date: 04/21/2016
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Benko, Hrvoje, Hancock, Mark, Wilson, Andrew D., Ofek, Eyal, Azmandian, Mahdi
• Granted Patent: US 9,805,514 B1
• Time in Patent Office: Days
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00

Abstract

Dynamic haptic retargeting can be implemented using world warping techniques and body warping techniques. World warping is applied to improve an alignment between a virtual object and a physical object, while body warping is applied to redirect a user'"'"'s motion to increase a likelihood that a physical hand will reach the physical object at the same time a virtual representation of the hand reaches the virtual object. Threshold values and/or a combination of world warping a body warping can be used to mitigate negative impacts that may be caused by using either technique excessively or independently.

Claims

1. A method comprising:

mapping a virtual environment to a physical environment to establish an alignment between the virtual environment and the physical environment;

determining, within the physical environment, a physical location of a physical object;

determining, within the virtual environment, a virtual location of a virtual object;

determining that a user is reaching toward the virtual object;

rendering within the virtual environment, a virtual hand that represents at least a portion of the user'"'"'s hand while the user is reaching toward the virtual object;

determining a location of the user'"'"'s physical hand and a corresponding location of the virtual hand while the user is reaching toward the virtual object; and

based at least in part on a difference between the physical location of the physical object and the virtual location of the virtual object;

dynamically adjusting the alignment between the virtual environment and the physical environment to reduce the difference between the physical location and the virtual location; and

dynamically adjusting the virtual representation of the user'"'"'s hand to cause the user to physically reach for the physical object while it appears that the virtual representation of the user'"'"'s hand is reaching for the virtual object, wherein dynamically adjusting the virtual representation of the user'"'"'s hand includes dynamically applying a body warping to adjust the location of the virtual hand within the virtual environment, wherein applying the body warping includes;

calculating a warping ratio based on the physical location of the physical object, an initial location of the user'"'"'s physical hand, and a current location of the user'"'"'s physical hand; and

adjusting the location of the virtual hand within the virtual environment based, at least in part, on the warping ratio.

2. A method as recited in claim 1, further comprising:

determining a first position of the user'"'"'s head while the user is reaching toward the virtual object;

determining a second position of the user'"'"'s head while the user is reaching toward the virtual object;

calculating a difference between the first position of the user'"'"'s head and the second position of the user'"'"'s head, wherein the difference indicates a vertical rotation; and

dynamically adjusting the alignment between the virtual environment and the physical environment to reduce a vertical distance between the physical location and the virtual location.

3. A method as recited in claim 1, wherein applying the body warping to adjust the location of the virtual hand within the virtual environment further includes:

determining a virtual vector that represents a distance and direction between the location of the virtual hand and the virtual location of the virtual object;

determining a physical vector that represents a distance and direction between the location of the user'"'"'s physical hand and the physical location of the physical object; and

dynamically applying the body warping to adjust the location of the virtual hand within the virtual environment based at least in part on a difference between the virtual vector and the physical vector.

5. A method as recited in claim 1, wherein applying the body warping further includes:

applying a translation to the virtual hand to adjust the location of the virtual hand within the virtual environment; and

applying a rotation to the virtual hand.

6. A method as recited in claim 1, further comprising:

repeatedly applying a world warping as the user reaches toward the virtual object.

7. A method as recited in claim 1, further comprising:

repeatedly applying a body warping as the user reaches toward the virtual object such that a position of the virtual hand intersects with the virtual location at substantially the same time that a position of the user'"'"'s physical hand intersects with the physical location.

4. (canceled)

8. A method comprising:

mapping a virtual environment to a physical environment to establish an alignment between the virtual environment and the physical environment;

determining, within the physical environment, a physical location of a physical object and a physical location of a user'"'"'s physical hand;

determining, within the virtual environment, a virtual location of a virtual object and a virtual location of a virtual representation of the user'"'"'s hand;

determining that a user is reaching toward the virtual object;

determining that the virtual object and the physical object are not aligned, so that, based on a current trajectory, when the virtual representation of the user'"'"'s hand reaches the virtual object, the user'"'"'s physical hand will not reach the physical object;

calculating a warping ratio based on the physical location of the physical object, an initial physical location of the user'"'"'s hand, and a current physical location of the user'"'"'s hand; and

dynamically adjusting the virtual location of the virtual representation of the user'"'"'s hand based, at least in part, on the warping ratio, to reduce a difference between a vector between the physical location of the physical object and the physical location of the user'"'"'s physical hand and a vector between the virtual location of the virtual object and the virtual location of the virtual representation of the user'"'"'s hand.

9. A method as recited in claim 8, further comprising:

repeatedly adjusting the virtual location of the virtual representation of the user'"'"'s hand as the user reaches toward the virtual object such that the virtual location of the virtual representation of the user'"'"'s hand intersects with the virtual location of the virtual object at substantially the same time that the physical location the user'"'"'s hand intersects with the physical location of the physical object.

10. A method as recited in claim 8, wherein dynamically adjusting the virtual location of the virtual representation of the user'"'"'s hand comprises:

applying a translation to the virtual representation of the user'"'"'s hand to adjust the virtual location of the virtual representation of the user'"'"'s hand within the virtual environment; and

applying a rotation to the virtual representation of the user'"'"'s hand.

12. A method as recited in claim 8, further comprising:

repeatedly calculating a warping ratio and adjusting the virtual location of the virtual representation of the user'"'"'s hand based, at least in part, on the warping ratio as the user reaches toward the virtual object such that the virtual location of the virtual representation of the user'"'"'s hand intersects with the virtual location of the virtual object at substantially the same time that the physical location the user'"'"'s hand intersects with the physical location of the physical object.

11. (canceled)

13. One or more computer readable media having computer-executable instructions stored thereon, which, when executed by a computing device, cause the computing device to perform operations comprising:

mapping a virtual environment to a physical environment to establish an alignment between the virtual environment and the physical environment;

determining, within the physical environment, a physical location of a physical object;

determining, within the virtual environment, a virtual location of a virtual object;

determining that a user is reaching toward the virtual object;

calculating a vertical distance between the physical location of the physical object and the virtual location of the virtual object;

determining a vertical rotation of the user'"'"'s head;

based on the vertical rotation of the user'"'"'s head, calculating a maximum adjustment value; and

dynamically adjusting the alignment between the virtual environment and the physical environment to reduce a difference between the physical location of the physical object and the virtual location of the virtual object, wherein adjusting the alignment includes;

dynamically adjusting a vertical alignment between the virtual environment and the physical environment based, at least in part on the maximum adjustment value and the vertical distance between the physical location of the physical object and the virtual location of the virtual object.

14. One or more computer readable media as recited in claim 21, wherein dynamically adjusting a virtual location of a virtual representation of the user'"'"'s hand to increase a likelihood that a physical hand of the user will reach the physical location of the physical object at substantially the same time that a virtual representation of the user'"'"'s hand will reach the virtual location of the virtual object includes:

dynamically adjusting a virtual location of a virtual representation of the user'"'"'s hand to reduce a difference between a vector between the physical location of the physical object and a physical location of the user'"'"'s physical hand and a vector between the virtual location of the virtual object and a virtual location of the virtual representation of the user'"'"'s hand.

15. One or more computer readable media as recited in claim 21, wherein dynamically adjusting a virtual location of a virtual representation of the user'"'"'s hand comprises:

translating the virtual representation of the user'"'"'s hand within the virtual environment; and

rotating the virtual representation of the user'"'"'s hand within the virtual environment.

16. One or more computer-readable media as recited in claim 13, wherein dynamically adjusting the alignment between the virtual environment and the physical environment to reduce a difference between the physical location of the physical object and the virtual location of the virtual object comprises:

determining a change in a position of the user'"'"'s head; and

dynamically adjusting the alignment between the virtual environment and the physical environment based, at least in part, on the determined change in the position of the user'"'"'s head.

17. One or more computer-readable media as recited in claim 16, wherein dynamically adjusting the alignment between the virtual environment and the physical environment to reduce a difference between the physical location of the physical object and the virtual location of the virtual object further comprises:

calculating distance between the physical location of the physical object and the virtual location of the virtual object; and

dynamically adjusting the alignment between the virtual environment and the physical environment further based, at least in part, on the distance between the physical location of the physical object and the virtual location of the virtual object.

18. One or more computer-readable media as recited in claim 13, wherein dynamically adjusting the alignment between the virtual environment and the physical environment to reduce a difference between the physical location of the physical object and the virtual location of the virtual object comprises:

determining a change in a position of the user'"'"'s head;

based, at least in part, on the change in the position of the user'"'"'s head, calculating a maximum adjustment value; and

adjusting the alignment between the virtual environment and the physical environment based, at least in part, on the maximum adjustment value.

19. One or more computer-readable media as recited in claim 18, wherein dynamically adjusting the alignment between the virtual environment and the physical environment to reduce a difference between the physical location of the physical object and the virtual location of the virtual object further comprises:

calculating a distance between the physical location of the physical object and the virtual location of the virtual object; and

dynamically adjusting the alignment between the virtual environment and the physical environment further based, at least in part, on the distance between the physical location of the physical object and the virtual location of the virtual object.

21. One or more computer readable media as recited in claim 13, further comprising:

dynamically adjusting a virtual location of a virtual representation of the user'"'"'s hand to increase a likelihood that a physical hand of the user will reach the physical location of the physical object at substantially the same time that a virtual representation of the user'"'"'s hand will reach the virtual location of the virtual object.

20. (canceled)