Gaming device with rotatably placed cameras

• US 9,649,558 B2
• Filed: 02/27/2015
• Issued: 05/16/2017
• Est. Priority Date: 03/14/2014
• Status: Active Grant

• Current Assignee: Sony Interactive Entertainment Inc. (Sony Group Corp.)
• Original Assignee: Sony Interactive Entertainment Inc. (Sony Group Corp.)
• Inventors: Stafford, Jeffrey Roger, Tokubo, Todd
• Primary Examiner(s): Yen, Jason
• Application Number:
  US14/633,415
• Publication Number:
  US 20150258431A1
• Time in Patent Office: 809 Days
• Field of Search: None
• CPC Class Codes: A63F 13/211 using inertial sensors, e.g. accelerometers or gyroscopes
• CPC Class Codes: A63F 13/212 using sensors worn by the player, e.g. for measuring heart beat or leg activity
• CPC Class Codes: A63F 13/213 comprising photodetecting means, e.g. cameras, photodiodes or infrared cells A63F13/219, A63F13/655 take precedence
• CPC Class Codes: A63F 13/5255 according to dedicated instructions from a player, e.g. using a secondary joystick to rotate the camera around a player's character
• CPC Class Codes: A63F 2300/1012 involving biosensors worn by the player, e.g. for measuring heart beat, limb activity
• CPC Class Codes: A63F 2300/1031 using a wireless connection, e.g. Bluetooth, infrared connections
• CPC Class Codes: A63F 2300/1087 comprising photodetecting means, e.g. a camera
• CPC Class Codes: A63F 2300/1093 using visible light
• CPC Class Codes: A63F 2300/301 using an additional display connected to the game console, e.g. on the controller
• CPC Class Codes: A63F 2300/303 for displaying additional data, e.g. simulating a Head Up Display
• CPC Class Codes: A63F 2300/308 Details of the user interface
• CPC Class Codes: A63F 2300/538 for performing operations on behalf of the game client, e.g. rendering
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/014 Hand-worn input/output arrangements, e.g. data gloves
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06V 40/11 Hand-related biometrics; Hand pose recognition

Abstract

A method to identify positions of fingers of a hand is described. The method includes capturing images of a first hand using a plurality of cameras that are part of a wearable device. The wearable device is attached to a wrist of a second hand and the plurality of cameras of the wearable device is disposed around the wearable device. The method includes repeating capturing of additional images of the first hand, the images and the additional images captured to produce a stream of captured image data during a session of presenting the virtual environment in a head mounted display (HMD). The method includes sending the stream of captured image data to a computing device that is interfaced with the HMD. The computing device is configured to process the captured image data to identify changes in positions of the fingers of the first hand.

Claims

1. A method to identify positions of fingers of a hand, the positions being used to render a virtual hand to be displayed in a head mounted display (HMD) when presenting a virtual environment in the HMD, comprising:

capturing images of a first hand using a plurality of cameras that are part of a wearable device, the wearable device being attached to a wrist of a second hand, the second hand being different from the first hand, the plurality of cameras of the wearable device being disposed around the wearable device so that the plurality of cameras are distributed around the wrist of the second hand;

repeating capturing of additional images of the first hand, the images and the additional images captured to produce a stream of captured image data during a session of presenting the virtual environment in the HMD; and

sending the stream of captured image data to a computing device that is interfaced with the HMD, the computing device configured to process the captured image data to identify changes in positions of the fingers of the first hand for rendering the virtual hand in the HMD corresponding to the changes in the positions of the fingers of the first hand.

2. The method of claim 1, wherein the plurality of cameras are disposed to be at angular positions with respect to each other, the angular positions formed with respect to lines that pass through a centroid of the wearable device.

3. The method of claim 1, further comprising capturing images of the first hand of a user using a plurality of cameras that are part of an additional wearable device, the additional wearable device being attached to a wrist of the first hand, the plurality of cameras of the additional wearable device being disposed at angular positions around the additional wearable device so that the plurality of cameras are distributed around the wrist of the first hand.

4. The method of claim 1, further comprising:

emitting light using a plurality of light emitters that are integrated within the wearable device, the light emitters interleaved with the cameras on a surface of the wearable device; and

detecting the light using a plurality of cameras integrated within an additional wearable device to generate images of positions of the wearable device that is attached to the wrist of the second hand, the additional wearable device attached to a wrist of the first hand.

5. The method of claim 1, further comprising:

determining based on one of the images of the first hand whether the first hand is visible in the one of the images; and

turning off the cameras upon determining that the first hand is not visible in the one of the images.

6. The method of claim 1, further comprising:

generating motion data using inertial sensors that are integrated within the wearable device;

sending a stream of the motion data to the computing device, the computing device configured to process the motion data to determine a position of the wearable device, the motion data used to determine the position of the wearable device, wherein the position of the wearable device provides the position of the second hand.

7. The method of claim 1, further comprising:

generating motion data using inertial sensors that are integrated within the wearable device;

sending a stream of the motion data to the computing device, the computing device configured to process the motion data and the captured image data to determine a position and an orientation of the wearable device.

8. The method of claim 1, further comprising:

emitting light in a round robin fashion using a plurality of light emitters that are integrated within the wearable device, wherein each light emitter is located at a pre-determined position around and on the second hand;

detecting the light using a plurality of cameras of an additional wearable device to generate electrical signal data, the additional wearable device for wearing on the first hand of a user;

sending the electrical signal data to the computing device, wherein the computing device is configured to identify an orientation of the second hand from the light and a correspondence between characteristics of light emitted by the light emitters and orientations of the second hand.

9. The method of claim 1, further comprising:

capturing images of the second hand of a user by using a plurality of cameras of an additional wearable device, the additional wearable device for wearing on the first hand of the user;

repeating capturing of additional images of the second hand, the images of the second hand and the additional images of the second hand captured to produce an additional stream of captured image data during the session of presenting the virtual environment in the HMD; and

sending the additional stream of captured image data to the computing device, the computing device configured to process the additional stream of captured image data to identify changes in positions of fingers of the second hand.

10. The method of claim 1, further comprising:

capturing images of the second hand of a user by using a plurality of cameras of an additional wearable device, the additional wearable device for wearing on the first hand of the user;

repeating capturing of additional images of the second hand, the images of the second hand and the additional images of the second hand captured to produce an additional stream of captured image data during the session of presenting the virtual environment in the HMD;

sending the additional stream of captured image data to the computing device, the computing device configured to process the captured image data from the additional stream to identify changes in positions of fingers of the second hand;

receiving a control signal to turn off one of the cameras of the wearable device, the control signal received when a determination is made that the one of the cameras of the wearable device is oriented with respect to the second hand so as to not capture images of the first hand of the user; and

receiving a control signal to turn off one of the cameras of the additional wearable device, the control signal to turn off the one of the cameras of the additional wearable device received when a determination is made that the one of the cameras of the additional wearable device is oriented with respect to the first hand so as to not capture images of the second hand of the user.

11. The method of claim 1, further comprising:

capturing images of the second hand of a user by using a plurality of cameras of an additional wearable device, the additional wearable device for wearing on the first hand of the user;

repeating capturing of additional images of the second hand, the images of the second hand and the additional images of the second hand captured to produce an additional stream of captured image data during the session of presenting the virtual environment in the HMD;

sending the additional stream of captured image data to the computing device, the computing device configured to process the captured image data from the additional stream to identify changes in positions of fingers of the second hand;

capturing additional image data of the first and second wearable devices when the first and second wearable devices are used with a pad device, the pad device placed below the first and second wearable devices, wherein capturing the additional image data of the first and second wearable devices is performed using a camera of the HMD; and

communicating the additional image data of the first and second wearable devices captured using the camera of the HMD to the computing device for identifying positions and orientations of the first and second hands of the user.

12. The method of claim 1,

wherein sending the stream of captured image data to the computing device that is interfaced with the HMD comprises communicating the image data captured using the cameras of the wearable device to a game console for facilitating an identification of the positions and orientations of the fingers of the first hand, wherein additional image data of the wearable device is captured with a camera that is located on the HMD, wherein the additional image data captured using the camera located on the HMD is for use by the game console to determine positions and orientations of the fingers of the first hand.

13. The method of claim 1,

wherein sending the stream of captured image data to the computing device is for facilitating an identification of an orientation of the fingers of the first hand of a user by a game console, the orientation of the fingers of the first hand identified for determining an orientation of display of virtual fingers of a virtual hand in the virtual environment that is displayed in the HMD.

14. The method of claim 1, further comprising:

capturing images of the second hand of a user by using a plurality of cameras of an additional wearable device, the additional wearable device for wearing on the first hand of the user;

repeating capturing of additional images of the second hand, the images of the second hand and the additional images of the second hand captured to produce an additional stream of captured image data during the session of presenting the virtual environment in the HMD;

sending the additional stream of captured image data to the computing device, the computing device configured to process the captured image data from the additional stream to identify changes in positions of fingers of the second hand,

capturing an image of a marker on the wearable device,

capturing an image of a marker on the additional wearable device,

communicating the image of the marker on the wearable device for facilitating an identification of a position of the marker of the wearable device,

communicating the image of the marker on the additional wearable device for facilitating an identification of a position of the marker of the additional wearable device,

wherein the position of the marker on the wearable device with respect to the position of the image of the additional marker is for providing an orientation of the first hand of the user with respect to the second hand of the user.

15. The method of claim 1, wherein the wearable device includes a fiber optic cable, wherein the wearable device includes a light emitter that is attached to the fiber optic cable for emitting light directed into the fiber optic cable.

16. The method of claim 1, wherein the wearable device includes a fiber optic cable having a plurality of openings, wherein the wearable device includes a light emitter that is attached to the fiber optic cable for emitting light directed into the fiber optic cable, wherein each opening emits a portion of the light that travels via the fiber optic cable, wherein each opening is a marker.

17. The method of claim 1, wherein the wearable device is configured to be worn around the wrist of a user in a manner similar to wearing a bracelet or wearing a watch or a wristband.

18. The method of claim 1, wherein additional image data of the wearable device is captured using a camera located within the game console, wherein the additional image data captured using the camera located within the game console is used by the game console to confirm or deny an accuracy of the positions of the fingers determined using the captured image data.

19. The method of claim 1, wherein additional image data of the wearable device is captured using a camera located within the HMD, wherein the additional image data captured using the camera located within the HMD is used by the game console to confirm or deny an accuracy of the positions of the fingers determined using the captured image data.

20. The method of claim 1, wherein the cameras of the wearable device are oriented with respect to the wearable device to capture image data of the second hand.

21. A method for identifying positions of hands of a user interacting with a virtual environment displayed in a head mounted display (HMD), comprising:

capturing images of a first hand of the user using a plurality of cameras that are part of a first wearable device, the first wearable device being attachable to a wrist of the first hand, the plurality of cameras of the first wearable device being disposed at angular positions around the first wearable device;

capturing images of a second hand of the user using a plurality of cameras that are part of a second wearable device, the second wearable device being attachable to a wrist of the second hand, the plurality of cameras of the second wearable device being disposed at angular positions around the second wearable device;

continuing the capturing of the images from the plurality of cameras of the first and second wearable devices during a session of interactivity with the virtual environment displayed in the HMD, the images captured by the first wearable device include images of the second wearable device and images captured by the second wearable device include images of the first wearable device;

capturing additional images of the first wearable device and the second wearable device using a reference camera;

sending the images from the first wearable device, the images from the second wearable device, and the additional images from the reference camera to a computing device that is interfaced with the HMD, the computing device configured to process the images from the first wearable device to identify positions of the second hand and process the images from the second wearable device to identify positions of the first hand, and the computing device uses the reference camera to provide a reference for the positions of the first and second hands.

22. The method of claim 21, further comprising illuminating at least one light on the first wearable device and at least one light on the second wearable device to provide for identification of the respective wearable devices in the images of the first and second hands captured by the cameras of the first and second wearable devices.

23. The method of claim 21, further comprising illuminating at least one light on the first wearable device and at least one light on the second wearable device to provide for identification of the respective wearable devices in the images of the first and second hands captured by the cameras of the first and second wearable devices, wherein the at least one light on the first wearable device has a state of constant illumination or is blinking or is strobing, wherein the at least one light on the second wearable device has a state of constant illumination or is blinking or is strobing.

24. The method of claim 21, further comprising illuminating at least one light on the first wearable device and at least one light on the second wearable device to provide for identification of the respective wearable devices in the images of the first and second hands captured by the cameras of the first and second wearable devices, wherein the at least one light on the first wearable device has a different wavelength than the at least one light on the second wearable device.

25. A system comprising:

a first wearable device for wearing on a wrist of a first hand of a user, wherein the first wearable device includes a camera for capturing image data of a second hand of the user, the second hand being different from the first hand,

wherein the first wearable device includes a communication device for communicating the image data captured using the first wearable device;

a game console coupled to the first wearable device, the game console having a console communication device coupled to the communication device of the wearable device for receiving the image data from the communication device of the wearable device,

wherein the game console includes a game processor coupled to the console communication device for identifying a position of the second hand of the user from the image data captured using the first wearable device,

wherein the game processor is configured to determine data regarding a state of a virtual object in a virtual environment based on the position of the second hand;

wherein the console communication device is for sending the data regarding the state of the virtual object, and

a head-mounted display (HMD) coupled to the game console, the HMD including;

an HMD communication device coupled to the console communication device for receiving the data regarding the state of the virtual object from the console communication device,

a processing unit coupled to the HMD communication device for displaying the virtual object having the state on a display screen of the HMD.

26. The system of claim 25, further comprising:

a second wearable device for wearing on the second hand of the user,

wherein the second wearable device includes a camera for capturing additional image data of the first hand of the user;

wherein the game processor is for identifying a position of the first hand of the user from the additional image data captured using the camera of the second wearable device,

wherein the game processor is configured to determine data regarding a state of an additional virtual object in the virtual environment based on the position of the first hand.

27. The system of claim 25, wherein the state includes a position and orientation of the virtual object, wherein the game processor is configured to determine the position and the orientation of the virtual object with respect to a reference point in the virtual environment as being proportional to the position of the second hand with respect to a reference point in a real-world environment.

28. A system comprising:

a first wearable device configured to be worn on a wrist of a left hand of a user;

a second wearable device configured to be worn on a wrist of a right hand of the user,

wherein the first wearable device includes a camera for capturing image data of the second wearable device,

wherein the second wearable device includes a camera for capturing image data of the first wearable device,

wherein the first wearable device includes a communication device coupled to the camera of the first wearable device for communicating the image data captured using the first wearable device,

wherein the second wearable device includes a communication device coupled to the camera of the second wearable device for communicating the image data captured using the second wearable device;

a game console coupled to the first and second wearable devices,

wherein the game console has a console communication device coupled to the communication device of the first wearable device for receiving the image data captured by the camera of the first wearable device,

wherein the console communication device is coupled to the communication device of the second wearable device for receiving the image data captured by the camera of the second wearable device,

wherein the game console includes a game processor coupled to the console communication device for identifying a position of the right hand of the user from the image data captured using the first wearable device,

wherein the game processor is for identifying a position of the left hand of the user from the image captured using the second wearable device,

wherein the game processor is for identifying a position of a virtual object in a virtual environment based on the position of the left hand and the position of the right hand;

wherein the console communication device is for communicating the position of the virtual object, and

a head-mounted display (HMD) including,

an HMD communication device coupled to the console communication device for receiving the position of the virtual object from the console communication device; and

a processing unit coupled to the HMD communication device for displaying the virtual object having the position of the virtual object on a display screen.

29. The system of claim 28,

wherein the game processor is configured to determine the position and an orientation of the virtual object with respect to a reference point in the virtual environment in which the virtual object is included, wherein the position and orientation of the virtual object with respect to the reference point in the virtual environment is determined as being proportional to the position and an orientation of the left hand with respect to a reference point in a real-world environment surrounding the game console, the HMD, and the first and second wearable devices,

wherein the position and orientation of the virtual object with respect to the reference point in the virtual environment is determined as being proportional to the position and an orientation of the right hand with respect to the reference point in the real-world environment.