DYNAMIC GLOVES TO CONVEY SENSE OF TOUCH AND MOVEMENT FOR VIRTUAL OBJECTS IN HMD RENDERED ENVIRONMENTS

• US 20160274662A1
• Filed: 03/20/2015
• Published: 09/22/2016
• Est. Priority Date: 03/20/2015
• Status: Active Grant

• Current Assignee: Sony Interactive Entertainment Inc. (Sony Group Corp.)
• Original Assignee: Sony Computer Entertainment Incorporated (Sony Group Corp.)
• Inventors: Mao, Xiaodong (Crusoe), Rimon, Noam, Ye, Xiaoyong
• Granted Patent: US 10,296,086 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: A63F 13/285 Generating tactile feedback signals via the game input device, e.g. force feedback
• CPC Class Codes: A63F 13/355 Performing operations on behalf of clients with restricted processing capabilities, e.g. servers transform changing game scene into an encoded video stream for transmitting to a mobile phone or a thin client
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/0141 characterised by the informative content of the display
• CPC Class Codes: G02B 27/0172 characterised by optical features
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/014 Hand-worn input/output arrangements, e.g. data gloves
• CPC Class Codes: G06F 3/016 Input arrangements with force or tactile feedback as computer generated output to the user
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/16 Sound input; Sound output speech processing G10L
• CPC Class Codes: G06T 7/20 Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51

Abstract

A system and method of using a peripheral device for interfacing with a virtual reality scene generated by a computer for presentation on a head mounted display. The peripheral device includes a haptic device capable of being placed in contact with a user and a haptic feedback controller for processing instructions for outputting a haptic signal to the haptic device. The haptic feedback controller receiving the instructions from the computer so that haptic feedback of the haptic device changes to correspond to a user'"'"'s virtual interactions with a virtual object in the virtual reality scene as presented on the head mounted display.

Claims

1. A system for processing content for a head mounted display (HMD), comprising,

a user interface device having a haptic device for imparting a haptic feedback to a user;

a computing device interfaced with the user interface and the HMD, the computing device including,

an input device interface for receiving input from the user interface device or the HMD, the user device interface processes instructions to select content for display in the HMD;

a virtual reality space generation module for rendering the selected content in the HMD in the form of a virtual reality scene;

an object configuration matrix rendering module for identifying a virtual object in the virtual reality scene, the object configuration matrix rendering module accessing a haptic profile for the virtual object;

a haptic response processing module for identifying interactions between the user interface device and the virtual object and sending the haptic feedback to the user interface device in accordance with the haptic profile of the virtual object.

2. The system of claim 1, wherein the user interface is a glove device that includes the haptic device and the haptic response processing module uses the haptic profile to impart changes to the haptic feedback based on a type of interaction detected.

3. The system of claim 2, further comprising,

a camera for detecting a location of the glove device in real space and a location of the HMD,

wherein the location of the glove device relative to the HMD in real space identifies a location of the glove device relative to the virtual object in the virtual reality scene;

wherein the relative location of the glove device and the HMD is used to identify interactions between the glove device and the virtual object and define a magnitude or type of the haptic feedback provided to at least one portion of the glove device.

4. The system of claim 2, further comprising,

at least one inertial sensor disposed on the glove device, the inertial sensor generating motion data usable by the computer to determine a position of the glove device or changes in position of the glove device.

5. The system of claim 1, wherein the content selected for display in the HMD is accessed from one or both of a local storage of the computing device or over a network.

6. The system of claim 1, wherein each of a plurality of virtual objects have a respective haptic profile, and each haptic profile defines attributes of the virtual object, and attributes of the virtual object can include one or more of a response to being held, a response to being moved, a response to being crushed, a response to being tossed, a response to being dropped, a response to being felt, a response to being rubbed, a response to being squeezed, a response to being pressed, a response to being smashed, a response to being bumped, a definition of size, a definition of temperature, a definition of weight, a definition associated with audio feedback, a definition associated with pressure or texture, a vibration definition, a surface smoothness, a surface roughness, a definition of resistance, of gradual pressure, or combinations of two or more thereof.

7. The system of claim 1, wherein the virtual reality space generation module is configured to continually generate additional content for the HMD depending at least on a location that the HMD is moved toward while rendering the virtual reality scene, the additional content further including one or more additional virtual objects;

wherein each additional virtual object includes a corresponding haptic profile.

8. A peripheral device for use in interfacing with a virtual reality scene generated by a computer for presentation on a head mounted display, comprising,

a haptic device capable of imparting haptic feedback to a user;

and

a haptic feedback controller for processing instructions for outputting a haptic signal to the haptic device, the haptic feedback controller using a haptic profile of a virtual object, the haptic feedback controller configured to receive the instructions from the computer so that the haptic feedback of the haptic device is defined by the haptic profile in response to an interaction by the peripheral device with a virtual object in the virtual reality scene when presented on the head mounted display;

wherein the haptic feedback of the haptic device is modified by the haptic feedback controller in response to an additional interaction with the virtual object using the peripheral device.

9. The peripheral device of claim 8, further comprising a motion sensor coupled to the haptic feedback controller, the motion sensor being capable of detecting a motion of the peripheral device relative to the virtual object and wherein the haptic feedback controller is capable of further modifying the haptic feedback to correspond with the detected motion.

10. The peripheral device of claim 8, wherein the haptic profile includes at least one of a characterization of a shape of the virtual object, or a characterization of a texture of the virtual object, or a characterization of a motion of the virtual object, or a characterization of size of the virtual object, or a characterization of physics of the virtual object, or a characterization of sound of the virtual object, or a characterization of a vibration of the virtual object, or a characterization of an amount of pressure virtually exerted on the user by the virtual object, or a characterization of a proximity of the virtual object to a second virtual object, or combinations of two or more thereof.

11. The peripheral device of claim 8, wherein the haptic device includes at least one of:

a compressed gas;

a liquid;

an electric motor;

a linear motor;

an electromagnet;

a piezo-electric device;

a heating element;

a cooling element;

one or more electroactive polymers;

or

one or more phase changing materials.

12. The peripheral device of claim 8, wherein the haptic device includes a fluid.

13. The peripheral device of claim 8, further comprising at least one motion sensor coupled to the haptic feedback controller.

14. The peripheral device of claim 8, wherein the haptic device is disposed in a glove capable of being worn by a user.

15. The peripheral device of claim 8, wherein the haptic device is disposed in the head mounted display and wherein the haptic device is capable of imparting the haptic feedback to a portion of the user'"'"'s head.

16. The peripheral device of claim 8, wherein the haptic device is disposed in an article of clothing capable of being worn by the user.

17. Method of simulating a feeling of contact with a virtual object in a virtual scene presented in a head mounted display (HMD), comprising:

tracking a peripheral device when interfacing with the virtual scene;

identifying the virtual object in the virtual scene;

accessing a haptic profile for the virtual object;

detecting a type of interaction between the peripheral device and the virtual object, and in response to the detected interaction, applying a haptic feedback to the peripheral device as defined by the haptic profile and the type of interaction, the haptic feedback simulates a feeling of contact with the virtual object.

18. The method of claim 17, wherein tracking the peripheral device enables tracking of position of the peripheral device relative to the virtual object in the virtual scene, the tracked position is used to determine the type of interaction and a corresponding type of the haptic feedback based on the haptic profile.

19. The method of claim 17, wherein the peripheral device is a glove device and the glove device includes one or more haptic devices, the haptic devices configured to impart the haptic feedback to a hand of a user, the method further comprising,

detecting movements in the HMD;

changing the virtual scene in response to the detected movements in the HMD, the changes in the virtual scene including rendering a second virtual object, the second virtual object being associated with a second haptic profile.

20. The method of claim 17, wherein the peripheral device is a glove device and the glove device includes one or more haptic devices, the haptic devices configured to impart the haptic feedback to a hand of a user.

21. The method of claim 19, wherein the virtual object and the second virtual object are part of a plurality of virtual objects,

wherein each of a plurality of virtual objects have a respective haptic profile, and each haptic profile defines attributes of the virtual object, and attributes of the virtual object can include one or more of a response to being held, a response to being moved, a response to being crushed, a response to being tossed, a response to being dropped, a response to being felt, a response to being rubbed, a response to being squeezed, a response to being pressed, a response to being smashed, a response to being bumped, a definition of size, a definition of weight, a definition associated with audio feedback, a definition associated with pressure or texture, a definition of temperature, a vibration definition, a surface smoothness, a surface roughness, a definition of resistance, of gradual pressure, or combinations of two or more thereof.

22. The method of claim 17, wherein at least one of the haptic devices is defined by:

a compressed gas;

a liquid;

an electric motor;

a linear motor;

an electromagnet;

a piezo-electric device;

a heating element;

a cooling element;

one or more electroactive polymers;

or

one or more phase changing materials.