Dynamic gloves to convey sense of touch and movement for virtual objects in HMD rendered environments

• US 10,296,086 B2
• Filed: 03/20/2015
• Issued: 05/21/2019
• Est. Priority Date: 03/20/2015
• Status: Active Grant

• Current Assignee: Sony Interactive Entertainment Inc. (Sony Group Corp.)
• Original Assignee: Sony Interactive Entertainment Inc. (Sony Group Corp.)
• Inventors: Rimon, Noam, Mao, Xiaodong (Crusoe), Ye, Xiaoyong
• Primary Examiner(s): Osordio, Ricardo
• Application Number:
  US14/664,740
• Publication Number:
  US 20160274662A1
• Time in Patent Office: 1,523 Days
• CPC Class Codes: A63F 13/285 Generating tactile feedback signals via the game input device, e.g. force feedback
• CPC Class Codes: A63F 13/355 Performing operations on behalf of clients with restricted processing capabilities, e.g. servers transform changing game scene into an encoded video stream for transmitting to a mobile phone or a thin client
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/0141 characterised by the informative content of the display
• CPC Class Codes: G02B 27/0172 characterised by optical features
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/014 Hand-worn input/output arrangements, e.g. data gloves
• CPC Class Codes: G06F 3/016 Input arrangements with force or tactile feedback as computer generated output to the user
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/16 Sound input; Sound output speech processing G10L
• CPC Class Codes: G06T 7/20 Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51

Abstract

A system and method of using a peripheral device for interfacing with a virtual reality scene generated by a computer for presentation on a head mounted display. The peripheral device includes a haptic device capable of being placed in contact with a user and a haptic feedback controller for processing instructions for outputting a haptic signal to the haptic device. The haptic feedback controller receiving the instructions from the computer so that haptic feedback of the haptic device changes to correspond to a user'"'"'s virtual interactions with a virtual object in the virtual reality scene as presented on the head mounted display.

Claims

1. A system for processing content for a head mounted display (HMD), comprising,

a user interface device having a haptic device for imparting a haptic feedback to a user, the user interface device configured to receive haptic profiles of virtual objects;

and

a computing device interfaced with the user interface device and the HMD, the computing device including,

an input device interface for receiving input from the user interface device or the HMD;

a virtual reality space generation module for rendering a virtual reality scene for the HMD;

an object configuration matrix rendering module for identifying a virtual object in the virtual reality scene, the object configuration matrix rendering module accessing a haptic profile for the virtual object; and

a haptic response processing module for identifying interactions between a virtual representation associated with the user interface device and the virtual object and sending the haptic feedback to the user interface device;

wherein the user interface device is operable to use the haptic profile of the virtual object for imparting the haptic feedback to the user, the haptic feedback drives activation of multiple points of pressure that vary in magnitude across a portion of a finger of the user that contacts the user interface device, the user interface device is configured to maintain the multiple points of pressure with continuous pressure for a duration depending upon said interactions between the virtual representation and the virtual object.

2. The system of claim 1, wherein the user interface is a glove device that includes the haptic device, the glove envelopes said finger of the user that contacts the user interface device, and the haptic response processing module uses the haptic profile to impart changes to the haptic feedback based on a type of interaction detected.

3. The system of claim 2, further comprising,

a camera for detecting a location of the glove device in real space and a location of the HMD,

wherein the location of the glove device relative to the HMD in real space identifies a location of the glove device relative to the virtual object in the virtual reality scene;

wherein the relative location of the glove device and the HMD is used to identify interactions between the glove device and the virtual object and define a magnitude or type of the haptic feedback provided to at least one portion of the glove device.

4. The system of claim 2, further comprising,

at least one inertial sensor disposed on the glove device, the inertial sensor generating motion data usable by the computer to determine a position of the glove device or changes in position of the glove device.

5. The system of claim 1, wherein the content selected for display in the HMD is accessed from one or both of a local storage of the computing device or over a network.

6. The system of claim 1, wherein each of a plurality of virtual objects have a respective haptic profile, and each haptic profile defines attributes of the virtual object, and attributes of the virtual object can include one or more of a response to being held, a response to being moved, a response to being crushed, a response to being tossed, a response to being dropped, a response to being felt, a response to being rubbed, a response to being squeezed, a response to being pressed, a response to being smashed, a response to being bumped, a definition of size, a definition of temperature, a definition of weight, a definition associated with audio feedback, a definition associated with pressure or texture, a vibration definition, a surface smoothness, a surface roughness, a definition of resistance, of gradual pressure, or combinations of two or more thereof.

7. The system of claim 1, wherein the virtual reality space generation module is configured to continually generate additional content for the HMD depending at least on a location that the HMD is moved toward while rendering the virtual reality scene, the additional content further including one or more additional virtual objects;

wherein each additional virtual object includes a corresponding haptic profile.

8. The system of claim 1, wherein a pressure gradient defines how to drive activation of the multiple points of pressure across the finger of the user such that the multiple points of pressure follow a nonlinear curve of pressure magnitude.

9. The system of claim 1, wherein the haptic profile defines the haptic feedback following a linear or nonlinear curve based on characteristics of the virtual object which drives activation of the multiple points of pressure.

10. A peripheral device for use in interfacing with a virtual reality scene generated by a computer for presentation on a head mounted display, comprising,

a haptic device capable of imparting haptic feedback to a user, the haptic feedback includes a pressure gradient applied across a portion of a finger of the user that contacts the peripheral device and

a haptic feedback controller for processing instructions for outputting a haptic signal to the haptic device, the haptic feedback controller using a haptic profile of a virtual object for outputting the haptic signal, the haptic feedback controller configured to receive the instructions from the computer and to receive the haptic profile of the virtual object from the computer so that the haptic feedback of the haptic device drives activation of multiple points of pressure that vary in magnitude for the pressure gradient that is applied across the portion of the finger of the user in response to an interaction by a virtual representation of the peripheral device with the virtual object in the virtual reality scene when presented on the head mounted display, the haptic feedback controller is configured to maintain the pressure gradient with continuous pressure for a duration depending upon the interaction by the virtual representation with the virtual object in the virtual reality scene;

wherein the haptic feedback of the haptic device is modified by the haptic feedback controller in response to an additional interaction with the virtual object using the peripheral device.

11. The peripheral device of claim 10, further comprising a motion sensor coupled to the haptic feedback controller, the motion sensor being capable of detecting a motion of the peripheral device relative to the virtual object and wherein the haptic feedback controller is capable of further modifying the haptic feedback to correspond with the detected motion.

12. The peripheral device of claim 10, wherein the haptic profile includes at least one of a characterization of a shape of the virtual object, or a characterization of a texture of the virtual object, or a characterization of a motion of the virtual object, or a characterization of size of the virtual object, or a characterization of physics of the virtual object, or a characterization of sound of the virtual object, or a characterization of a vibration of the virtual object, or a characterization of an amount of pressure virtually exerted on the user by the virtual object, or a characterization of a proximity of the virtual object to a second virtual object, or combinations of two or more thereof.

13. The peripheral device of claim 10, wherein the haptic device includes at least one of:

a compressed gas;

a liquid;

an electric motor;

a linear motor;

an electromagnet;

a piezo-electric device;

a heating element;

a cooling element;

one or more electroactive polymers;

or

one or more phase changing materials.

14. The peripheral device of claim 10, wherein the haptic device includes a fluid.

15. The peripheral device of claim 10, further comprising at least one motion sensor coupled to the haptic feedback controller.

16. The peripheral device of claim 10, wherein the haptic device is disposed in a glove capable of being worn by a user, the glove envelopes the finger of the user that contacts the peripheral device.

17. The peripheral device of claim 10, wherein the haptic device is disposed in an article of clothing capable of being worn by the user, the article of clothing envelopes the finger of the user that contacts the peripheral device.

18. A method of simulating a feeling of contact with a virtual object in a virtual scene presented in a head mounted display (HMD), comprising:

tracking a peripheral device when interfacing with the virtual scene, the peripheral device configured to receive haptic profiles of virtual objects;

identifying the virtual object in the virtual scene;

accessing a haptic profile for the virtual object;

sending the haptic profile for the virtual object to the peripheral device; and

detecting a type of interaction between a virtual representation of the peripheral device and the virtual object, and in response to the detected interaction, applying a haptic feedback to the peripheral device as defined by a pressure gradient, the haptic feedback simulates a feeling of contact with the virtual object;

wherein the peripheral device is operable to use the haptic profile to drive application of multiple points of pressure that vary in magnitude across a portion of a finger of the user for the pressure gradient, the peripheral device is configured to maintain the pressure gradient with continuous pressure for a duration depending upon the type of interaction between the virtual representation and the virtual object.

19. The method of claim 18, wherein tracking the peripheral device enables tracking of position of the peripheral device relative to the virtual object in the virtual scene, the tracked position is used to determine the type of interaction and a corresponding type of the haptic feedback based on the haptic profile.

20. The method of claim 18, wherein the peripheral device is a glove device that envelopes said finger of the user and the glove device includes one or more haptic devices, the haptic devices configured to impart the haptic feedback to a hand of a user including said finger, the method further comprising,

detecting movements in the HMD;

changing the virtual scene in response to the detected movements in the HMD, the changes in the virtual scene including rendering a second virtual object, the second virtual object being associated with a second haptic profile.

21. The method of claim 20, wherein the virtual object and the second virtual object are part of a plurality of virtual objects,

wherein each of a plurality of virtual objects have a respective haptic profile, and each haptic profile defines attributes of the virtual object, and attributes of the virtual object can include one or more of a response to being held, a response to being moved, a response to being crushed, a response to being tossed, a response to being dropped, a response to being felt, a response to being rubbed, a response to being squeezed, a response to being pressed, a response to being smashed, a response to being bumped, a definition of size, a definition of weight, a definition associated with audio feedback, a definition associated with pressure or texture, a definition of temperature, a vibration definition, a surface smoothness, a surface roughness, a definition of resistance, of gradual pressure, or combinations of two or more thereof.

22. The method of claim 18, wherein the peripheral device is a glove device and the glove device includes one or more haptic devices, the haptic devices configured to impart the haptic feedback to a hand of a user including said finger.

23. The method of claim 22, wherein at least one of the haptic devices is defined by:

a compressed gas;

a liquid;

an electric motor;

a linear motor;

an electromagnet;

a piezo-electric device;

a heating element;

a cooling element;

one or more electroactive polymers;

or

one or more phase changing materials.