Physical haptic feedback system with spatial warping

• US 10,416,769 B2
• Filed: 02/14/2017
• Issued: 09/17/2019
• Est. Priority Date: 02/14/2017
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Ofek, Eyal, Wilson, Andrew, Benko, Hrvoje, Holz, Christian, Cheng, Lung-Pan
• Primary Examiner(s): Dicke, Chad M
• Application Number:
  US15/432,878
• Publication Number:
  US 20180232050A1
• Time in Patent Office: 945 Days
• Field of Search: None
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G06F 2203/014 Force feedback applied to GUI
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/016 Input arrangements with force or tactile feedback as computer generated output to the user
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06T 3/18 Image warping, e.g. rearranging pixels individually

Abstract

A computing system including a head mounted display device with a processor and an associated display is provided. A sensor in communication with the processor is configured to detect a movable body part of a user. A plurality of physical haptic feedback structures are configured to be contacted by the movable body part. The processor is configured to operate the display device, receive data from the sensor, and determine an intended virtual target of the movable body part and a target physical structure having haptic characteristics corresponding to the intended virtual target. Also, the processor is configured to compute a path in real three-dimensional space from the movable body part to the target physical structure, compute a spatial warping pattern, and display via the display the virtual space and the virtual reality representation according to the spatial warping pattern.

Claims

1. A computing system, comprising:

a head mounted display device including a processor and an associated display;

a sensor in communication with the processor, the sensor being configured to detect a movable body part of a user; and

a plurality of physical haptic feedback structures configured to be contacted by the movable body part, the physical haptic feedback structures having physical haptic characteristics that are differentiable from each other and positioned at different respective positions in real three-dimensional space;

the processor configured to;

operate the display device to display a virtual three-dimensional space corresponding to real three-dimensional space;

receive from the sensor data indicating a detected location of the movable body part within real three-dimensional space;

operate the display device to display a virtual reality representation of the movable body part, a position of the virtual reality representation of the movable body part being displayed so as to appear to be positioned in a virtual location within the virtual space corresponding to the detected location in real three-dimensional space;

determine, from among a plurality of virtual targets in the virtual space and a detected motion of the movable body part, an estimated intended virtual target of the movable body part;

determine a score for each of the plurality of physical haptic feedback structures based on the physical haptic characteristics and the position of each of the physical haptic feedback structures with respect to the estimated intended virtual target;

determine a target physical haptic feedback structure from among the plurality of physical haptic feedback structures based on a comparison of the scores calculated for the plurality of physical haptic feedback structures;

compute a path in the real three-dimensional space from the movable body part to the target physical haptic feedback structure;

compute a spatial warping pattern to warp an image displayed on the display based on the computed path; and

display via the display the virtual space and the virtual reality representation according to the spatial warping pattern in order to redirect the movable body part along the computed path from the estimated intended virtual target to the target physical haptic feedback structure.

2. The system of claim 1, wherein the processor is configured to determine to which of the plurality of physical haptic feedback structures the movable body part is to be directed based upon at least one parameter selected from the group consisting of a distance between a current location of the movable body part and the target physical haptic feedback structure, an orientation of the target physical haptic feedback structure and the virtual target in the virtual space, and a type of haptic feedback mechanism in the target physical haptic feedback structure.

3. The system of claim 1, wherein the spatial warping pattern is computed to redirect the movable body part along the computed path to the target physical haptic feedback structure and the image warped by the spatial warping pattern is at least one of the group consisting of an image of the virtual space and an image of the virtual reality representation of the movable body part.

4. The system of claim 1, wherein the spatial warping pattern is computed to redirect the movable body part along the computed path to the target physical haptic feedback structure, and wherein the processor is further configured to dynamically recalculate the spatial warping pattern in a series of time steps based on dynamic determination of the estimated intended virtual target of the movable body part, therefore causing redirection of the movable body part to the target physical haptic feedback structure to be dynamic and the movable body part to contact the target physical haptic feedback structure concurrently with the virtual reality representation of the movable body part appears to contact the estimated intended virtual target.

5. The system of claim 4, wherein the path is one of a plurality of possible paths to the target physical haptic feedback structure, and wherein computation of the spatial warping pattern includes computing a minimized spatial warping pattern that minimizes an amount by which the image displayed is warped.

6. The system of claim 1, wherein the processor is further configured to determine application of the spatial warping pattern based upon a threshold distance between the estimated intended virtual target and the target physical haptic feedback structure.

7. The system of claim 1, wherein at least one of the plurality of physical haptic feedback structures is dynamically mapped to the plurality of virtual targets in the virtual space and the movable body part is directed to the target physical haptic feedback structure based on the determination by the processor, from among the plurality of virtual targets in the virtual space and the detected motion of the movable body part, of the estimated intended virtual target of the movable body part.

8. The system of claim 1, further comprising a dynamic haptic adjustment mechanism that adjusts at least a first haptic characteristic of the physical haptic feedback structures, the first haptic characteristic being at least one of the group consisting of applied force, pressure, rotation, rotatability, mechanical resistance, vibration, deformability, elasticity, texture, temperature, electrical charge, electrical resistance, pressure from vented air (non-contact), and emitted ultrasound (non-contact).

9. The system of claim 1, wherein the target physical haptic feedback structure is selected from the group consisting of a handle, a dial, a knob, a button, a switch, a toggle, a wheel, a lever, a pedal, a pull, a key, and a joystick.

10. The system of claim 1, wherein the physical haptic feedback structures include a first surface and a second surface formed as regions on a continuous surface of a base material.

11. A method for use with a computing device, comprising:

at a processor;

operating a head mounted display device and an associated display to display a virtual three-dimensional space corresponding to real three-dimensional space, the display device including a sensor in communication with the processor, the sensor being configured to detect a movable body part of a user;

receiving from the sensor data indicating a detected location of the movable body part within real three-dimensional space;

operating the display device to display a virtual reality representation of the movable body part, a position of the virtual reality representation of the movable body part being displayed so as to appear to be positioned in a virtual location within the virtual space corresponding to the detected location in real three-dimensional space;

determining, from among a plurality of virtual targets in the virtual space and a detected motion of the movable body part, an estimated intended virtual target of the movable body part;

determining a score for each of a plurality of physical haptic feedback structures based on physical haptic characteristics of each of the physical haptic feedback structures and the position of each of the physical haptic feedback structures with respect to the estimated intended virtual target;

determining a target physical haptic feedback structure from among the plurality of physical haptic feedback structures based on a comparison of the scores calculated for the plurality of physical haptic feedback structures, the target physical haptic feedback structure being selected from among the plurality of physical haptic feedback structures configured to be contacted by the movable body part, the physical haptic feedback structures having physical haptic characteristics that are differentiable from each other and positioned at different respective positions in real three-dimensional space;

computing a path in the real three-dimensional space from the movable body part to the target physical haptic feedback structure;

computing a spatial warping pattern to warp an image displayed on the display based on the computed path; and

displaying via the display the virtual space and the virtual reality representation according to the spatial warping pattern in order to redirect the movable body part along the computed path from the estimated intended virtual target to the target physical haptic feedback structure.

12. The method of claim 11, wherein the processor is configured to determine to which of the plurality of physical haptic feedback structures the movable body part is to be directed based upon at least one parameter selected from the group consisting of a distance between a current location of the movable body part and the target physical haptic feedback structure, an orientation of the target physical haptic feedback structure and the virtual target in the virtual space, and a type of haptic feedback mechanism in the target physical haptic feedback structure.

13. The method of claim 11, wherein the spatial warping pattern is computed to redirect the movable body part along the computed path to the target physical haptic feedback structure and the image warped by the spatial warping pattern is at least one of the group consisting of an image of the virtual space and an image of the virtual reality representation of the movable body part.

14. The method of claim 11, wherein the spatial warping pattern is computed to redirect the movable body part along the computed path to the target physical haptic feedback structure, and wherein the processor is further configured to dynamically recalculate the spatial warping pattern in a series of time steps based on dynamic determination of the estimated intended virtual target of the movable body part, therefore causing redirection of the movable body part to the target physical haptic feedback structure to be dynamic and the movable body part to contact the target physical haptic feedback structure concurrently with the virtual reality representation of the movable body part appears to contact the estimated intended virtual target.

15. The method of claim 14, wherein the path is one of a plurality of possible paths to the target physical haptic feedback structure, and wherein computation of the spatial warping pattern includes computing a minimized spatial warping pattern that minimizes an amount by which the image displayed is warped.

16. The method of claim 11, further comprising, via the processor, determining application of the spatial warping pattern based upon a threshold distance between the estimated intended virtual target and the target physical haptic feedback structure.

17. The method of claim 11, wherein at least one of the plurality of physical haptic feedback structures is dynamically mapped to the plurality of virtual targets in the virtual space and the movable body part is directed to the target physical haptic feedback structure based on the determination by the processor, from among the plurality of virtual targets in the virtual space and the detected motion of the movable body part, of the estimated intended virtual target of the movable body part.

18. The method of claim 11, further comprising adjusting, via a dynamic haptic adjustment mechanism, at least a first haptic characteristic of the physical haptic feedback structures, the first haptic characteristic being at least one of the group consisting of applied force, pressure, rotation, rotatability, mechanical resistance, vibration, deformability, elasticity, texture, temperature, electrical charge, electrical resistance, pressure from vented air (non-contact), and emitted ultrasound (non-contact).

19. The method of claim 11, wherein the physical haptic feedback structures include a first surface and a second surface formed as regions on a continuous surface of a base material.

20. A computing system, comprising:

a head mounted display device including a processor and an associated display;

a sensor in communication with the processor, the sensor being configured to detect a movable physical object under direct control of a user; and

a plurality of physical haptic feedback structures configured to be contacted by the movable object, the physical haptic feedback structures having physical haptic characteristics that are differentiable from each other and positioned at different respective positions in a real three-dimensional space;

the processor configured to;

operate the display device to display a virtual three-dimensional space corresponding to real three-dimensional space;

receive from the sensor data indicating a detected location of the movable object within real three-dimensional space;

operate the display device to display a virtual reality representation of the movable object, a position of the virtual reality representation of the movable object being displayed so as to appear to be positioned in a virtual location within the virtual space corresponding to the detected location in real three-dimensional space;

determine, from among a plurality of virtual targets in the virtual space and a detected motion of the movable object, an estimated intended virtual target of the movable object;

determine a score for each of the plurality of physical haptic feedback structures based on the physical haptic characteristics and the position of each of the physical haptic feedback structures with respect to the estimated intended virtual target;

determine a target physical haptic feedback structure from among the plurality of physical haptic feedback structures based on a comparison of the scores calculated for the plurality of physical haptic feedback structures;

compute a path in the real three-dimensional space from the movable object to the target physical haptic feedback structure;

compute a spatial warping pattern to warp an image displayed on the display based on the computed path; and

display via the display the virtual space and the virtual reality representation according to the spatial warping pattern in order to redirect the movable object along the computed path from the estimated intended virtual target to the target physical haptic feedback structure.