GAMING DEVICE WITH VOLUMETRIC SENSING

• US 20150258432A1
• Filed: 03/13/2015
• Published: 09/17/2015
• Est. Priority Date: 03/14/2014
• Status: Active Grant

• Current Assignee: Sony Interactive Entertainment Inc. (Sony Group Corp.)
• Original Assignee: Sony Computer Entertainment Incorporated (Sony Group Corp.)
• Inventors: Stafford, Jeffrey Roger, Tokubo, Todd
• Granted Patent: US 9,987,554 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: A63F 13/213 comprising photodetecting means, e.g. cameras, photodiodes or infrared cells A63F13/219, A63F13/655 take precedence
• CPC Class Codes: A63F 13/215 comprising means for detecting acoustic signals, e.g. using a microphone
• CPC Class Codes: A63F 13/23 for interfacing with the game device, e.g. specific interfaces between game controller and console
• CPC Class Codes: A63F 13/24 Constructional details thereof, e.g. game controllers with detachable joystick handles
• CPC Class Codes: A63F 13/25 Output arrangements for video game devices
• CPC Class Codes: A63F 13/26 having at least one additional display device, e.g. on the game controller or outside a game booth
• CPC Class Codes: A63F 13/426 involving on-screen location information, e.g. screen coordinates of an area at which the player is aiming with a light gun
• CPC Class Codes: A63F 13/428 involving motion or position input signals, e.g. signals representing the rotation of an input controller or a player's arm motions sensed by accelerometers or gyroscopes
• CPC Class Codes: A63F 13/5255 according to dedicated instructions from a player, e.g. using a secondary joystick to rotate the camera around a player's character
• CPC Class Codes: A63F 13/98 Accessories, i.e. detachable arrangements optional for the use of the video game device, e.g. grip supports of game controllers
• CPC Class Codes: A63F 2300/538 for performing operations on behalf of the game client, e.g. rendering
• CPC Class Codes: G06F 2203/04101 2.5D-digitiser, i.e. digitiser detecting the X/Y position of the input means, finger or stylus, also when it does not touch, but is proximate to the digitiser's interaction surface and also measures the distance of the input means within a short range in the Z direction, possibly with a separate measurement setup
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text

Abstract

A method for enhancing detection of a user'"'"'s hand relative to a head-mounted display (HMD) is described. The method includes sensing a disrupted portion of energy by sensors integrated within a pad device. The disrupted portion of the energy is generated when the hand of the user interferes with the energy. The plurality of sensors that sense the disrupted portion of the energy produce an energy image that mirrors a current position of the hand. The method includes repeating the sensing continuously to produce a stream of energy images. The method includes communicating the stream of energy images to a game console for processing each of the energy images to produce a model of the hand and movement of the model of the hand. The model of the hand is at least partially rendered as a virtual hand in a virtual environment that is displayed in the HMD.

Claims

1. A method for enhancing detection of a user'"'"'s hand relative to a head-mounted display (HMD), comprising:

sensing a disrupted portion of energy, when disrupted, by a plurality of sensors integrated within a pad device and arranged in an array, wherein the disrupted portion of the energy is generated when the hand of the user is placed proximate to the pad device, wherein the plurality of sensors that sense the disrupted portion of the energy produce an energy image that mirrors a current position of the hand;

repeating the sensing continuously to produce a stream of energy images, the stream of energy images showing changes of the position of the hand while the sensing is being repeated; and

communicating the stream of energy images to a game console for processing each of the energy images to produce a model of the hand and movement of the model of the hand, wherein the model of the hand is at least partially rendered as a virtual hand in a virtual environment that is displayed in the HMD.

2. The method of claim 1, further comprising emitting the energy using a plurality of energy emitters, the energy emitters integrated within the pad device and arranged in an array.

3. The method of claim 1, wherein communicating the stream of energy images to the game console to produce the model of the hand comprises communicating the stream of energy images to the game console for facilitating a determination of a position and an orientation of the hand, wherein the position of the hand is determined at least partially based on intensities of the disrupted portion of the energy within the energy images.

4. The method of claim 1, wherein communicating the stream of energy images to the game console to produce the model of the hand comprises communicating the stream of energy images to the game console for facilitating a determination of a position and an orientation of the hand, wherein another position and another orientation of the hand is determined using a camera connected to the game console or using a camera in the HMD or using a camera in the pad device or using a camera in an environment in which the pad device is located.

5. The method of claim 1, wherein communicating the stream of energy images to the game console to produce the model of the hand comprises communicating electrical signals to the game console for determining a position and an orientation of the hand, wherein the position and orientation of the hand is to be determined with respect to an xyz co-ordinate system, wherein a surface of the pad device defines a reference for the xyz co-ordinate system.

6. The method of claim 1, wherein the pad device includes markers that are located on a surface of the pad device, wherein the virtual hand is displayed on a display device of the HMD, wherein the HMD is to be worn on a head of the user and has a camera, wherein the camera is for capturing occlusion of one or more of the markers to facilitate a determination of a position and an orientation of the hand of the user.

7. The method of claim 1, wherein the energy includes light energy, said method further comprising emitting the light energy using a plurality of infrared light emitters, the infrared light emitters integrated within the pad device and arranged in an array on the pad device.

8. The method of claim 1, wherein sensing the disrupted portion of the energy is performed when the user is interacting with the virtual environment via the HMD.

9. The method of claim 1, wherein sensing the disrupted portion of the energy is performed when the user is interacting with the virtual environment via the HMD, wherein the energy includes light energy, or ultrasonic energy, or electromagnetic energy, or a combination of two or more thereof.

10. The method of claim 1, wherein the pad device has a surface on which the sensors are located, wherein the surface is for orientation towards the hand and for receiving a shadow of the hand when the hand interacts with an energy volume above the pad device, wherein the shadow defines intensities of the disrupted portion when disrupted by the user'"'"'s hand.

11. The method of claim 1, further comprising emitting the energy using a plurality of energy emitters, the energy emitters integrated within the pad device and arranged in an array, wherein the pad device is configured for receiving touch-less interaction from the hand, wherein the energy emitted extends to an energy boundary above the pad device, wherein the disrupted portion of the energy is generated when the hand is located within the energy boundary.

12. The method of claim 1, wherein the model of the virtual hand is defined by a position of the virtual hand, or an orientation of the virtual hand, or a shape of the virtual hand, or a color of the virtual hand, or a boundary of the virtual hand, or a texture of the virtual hand, or a combination of two or more thereof, wherein the boundary of the virtual hand is defined by pixels of an image that form an outline of the virtual hand.

13. The method of claim 1, wherein sensing the disrupted portion comprises sensing energy disrupted by a first finger joint of a finger of the hand and sensing energy disrupted by a second finger joint of the finger for facilitating a determination of a position of the finger of the user, the determination of the position performed using a line that couples a position of the first finger joint to a finger of the second finger joint.

14. The method of claim 1, wherein the virtual hand is for display in the HMD when worn on a head of the user, or is for display on a television that is located in a real-world environment in which the user is located.

15. The method of claim 1, wherein the stream of energy images is generated at a frame rate of the sensors, the frame rate being a rate of generation of frames of the energy images, wherein the frame rate is a frequency at which the disrupted portion of the energy is sensed or captured by the sensors.

16. The method of claim 1, further comprising emitting the energy using a plurality of energy emitters, the energy emitters integrated within the pad device and arranged in an array, wherein the energy emitted is for creating an energy volume above the pad device, wherein the hand is placed proximate to the pad device to interact with and disrupt the energy volume.

17. The method of claim 16, wherein the energy volume includes an electromagnetic energy field.

18. The method of claim 1, wherein the movement of the model of the hand defines movement of one or more images of fingers of the hand, wherein the virtual hand is enabled to interact with a virtual object in the virtual environment, wherein the interaction is defined by an object model for the virtual object, the object model is defined for pre-defined movement or actions within the virtual environment.

19. A pad device for enhancing detection of a user'"'"'s hand relative to a head-mounted display (HMD), comprising:

a plurality of sensors for sensing a disrupted portion of energy, the sensors arranged in an array on a surface of the pad device, wherein the disrupted portion of the energy is generated when the hand of the user is placed proximate to the pad device, wherein the sensors are configured to sense the disrupted portion of the energy for producing an energy image that mirrors a current position of the hand,

wherein the sensors are further configured to repeat the sensing continuously to produce a stream of energy images, the stream of energy images showing changes of the position of the hand while the sensing is being repeated; and

a communication device coupled to the sensors, the communication device for communicating the stream of energy images to a game console for processing each of the energy images to produce a model of the hand and movement of the model of the hand, wherein the model of the hand is at least partially rendered as a virtual hand in a virtual environment that is displayed in the HMD.

20. The pad device of claim 19, further comprising a plurality of emitters for emitting the energy, the energy emitters integrated within the pad device and arranged in an array on the surface of the pad device.

21. A system comprising:

a pad device including;

a plurality of sensors located on a surface of the pad device for receiving an energy portion upon interaction of a body part with energy emitted,

wherein the sensors are configured to generate an energy image from the energy portion by sensing the energy portion when the body part is at a current position, wherein the sensors are further configured to repeat the sensing continuously to produce a stream of energy images, the stream of energy images showing changes of the position of the hand while the sensing is being repeated;

a pad communication device coupled to the sensors for communicating the stream of energy images;

a game console coupled to the pad device, the game console including;

a game communication device coupled to the pad communication device for receiving the stream of the energy images from the pad communication device;

a game processor coupled to the game communication device for determining a movement of the body part based on the stream of energy images,

wherein the game processor is for determining a state of a virtual object in a virtual environment based on the movement of the body part,

wherein the game communication device is for sending data regarding the state of the virtual object; and

a head-mounted display (HMD) coupled to the game console, the HMD comprising;

an HMD communication device coupled to the game communication device for receiving the data regarding the state of the virtual image from the game communication device;

a processor coupled to the HMD communication device for displaying the virtual object having the state in an image.

22. The system of claim 21,

wherein the game processor is for determining a position of the virtual object with reference to a reference point defined in the image as being proportional to the position of the body part with respect a reference defined by the surface of the pad device,

wherein the game processor is for determining an orientation of the virtual object with reference to the reference point defined in the image as being proportional to an orientation of the body part with respect to the reference defined by the surface of the pad device.

23. The system of claim 21, wherein the pad device includes a plurality of light emitters for emitting light towards the HMD, wherein the HMD includes a plurality of sensors for sensing the light emitted by the light emitters to facilitate determination of a position of the hand of the user.