Gaming device with rotatably placed cameras

• US 9,996,166 B2
• Filed: 09/01/2017
• Issued: 06/12/2018
• Est. Priority Date: 03/14/2014
• Status: Active Grant

• Current Assignee: Sony Interactive Entertainment Inc. (Sony Group Corp.)
• Original Assignee: Sony Interactive Entertainment Inc. (Sony Group Corp.)
• Inventors: Stafford, Jeffrey Roger, Tokubo, Todd
• Primary Examiner(s): YEN, JASON TAHAI
• Application Number:
  US15/694,511
• Publication Number:
  US 20180011545A1
• Time in Patent Office: 284 Days
• Field of Search: None
• CPC Class Codes: A63F 13/211 using inertial sensors, e.g. accelerometers or gyroscopes
• CPC Class Codes: A63F 13/212 using sensors worn by the player, e.g. for measuring heart beat or leg activity
• CPC Class Codes: A63F 13/213 comprising photodetecting means, e.g. cameras, photodiodes or infrared cells A63F13/219, A63F13/655 take precedence
• CPC Class Codes: A63F 13/5255 according to dedicated instructions from a player, e.g. using a secondary joystick to rotate the camera around a player's character
• CPC Class Codes: A63F 2300/1012 involving biosensors worn by the player, e.g. for measuring heart beat, limb activity
• CPC Class Codes: A63F 2300/1031 using a wireless connection, e.g. Bluetooth, infrared connections
• CPC Class Codes: A63F 2300/1087 comprising photodetecting means, e.g. a camera
• CPC Class Codes: A63F 2300/1093 using visible light
• CPC Class Codes: A63F 2300/301 using an additional display connected to the game console, e.g. on the controller
• CPC Class Codes: A63F 2300/303 for displaying additional data, e.g. simulating a Head Up Display
• CPC Class Codes: A63F 2300/308 Details of the user interface
• CPC Class Codes: A63F 2300/538 for performing operations on behalf of the game client, e.g. rendering
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/014 Hand-worn input/output arrangements, e.g. data gloves
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06V 40/11 Hand-related biometrics; Hand pose recognition

Abstract

A method to identify positions of fingers of a hand is described. The method includes capturing images of a first hand using a plurality of cameras that are part of a wearable device. The wearable device is attached to a wrist of a second hand and the plurality of cameras of the wearable device is disposed around the wearable device. The method includes repeating capturing of additional images of the first hand, the images and the additional images captured to produce a stream of captured image data during a session of presenting the virtual environment in a head mounted display (HMD). The method includes sending the stream of captured image data to a computing device that is interfaced with the HMD. The computing device is configured to process the captured image data to identify changes in positions of the fingers of the first hand.

Claims

1. A method to identify positions of a wrist of a user for displaying a virtual hand viewable via a head-mounted display, comprising:

capturing an image of a wearable device worn on the wrist of the user, wherein the wearable device includes a plurality of light emitters located on an outer surface of the wearable device, wherein the plurality of light emitters are configured to emit light in a different form compared to emission of light by a plurality of additional light emitters of another wearable device to facilitate distinguishing the wearable device from the other wearable device, wherein the other wearable device is worn on another wrist of the user, wherein said capturing of the image is performed by a camera having a field-of-view directed towards the user to capture the image, wherein said capturing is performed when the wearable device is moved to produce gestures that are captured by the camera;

repeating capturing of additional images of the wearable device by the camera, wherein said capturing of the image and said repeating are performed to generate a stream of image data;

sending the stream of image data to a computing device, wherein the computing device is configured to process the image data to identify changes in positions of the wrist for rendering the virtual hand in the head-mounted display.

2. The method of claim 1, wherein the light emitters are spaced apart in a periodic manner across the outer surface to facilitate capturing of the image and the additional images.

3. The method of claim 1, wherein the camera is located within a room in which the wearable device is located, wherein the camera is facing the user, wherein the field-of-view stays the same independent of movement of the head-mounted display.

4. The method of claim 1, wherein the camera is located on an outer surface of the head-mounted display, wherein the field-of-view of the camera moves with movement of the head-mounted display.

5. The method of claim 1, further comprising capturing an image of the other wearable device, wherein the other wearable device has a camera on an outer surface of the other wearable device, the method further comprising:

capturing an image of a portion of the wearable device by the camera on the other wearable device;

repeating capturing of additional images of the portion of the wearable device by the camera on the other wearable device to generate an additional stream of image data;

sending the additional stream of image data to the computing device, wherein the computing device is configured to process the additional stream of image data to identify changes in positions of the wrist.

6. The method of claim 5, wherein the computing device is configured to confirm an accuracy of the positions of the wrist determined from the additional stream of image data based on the positions of the wrist determined from the stream of image data.

7. The method of claim 1, wherein the wearable device has a camera having a field-of-view directed towards a forearm of the user, the method further comprising:

capturing an image of the forearm by the camera on the wearable device;

repeating capturing of additional images of the forearm by the camera on the wearable device to generate an additional stream of image data;

sending the additional stream of image data to the computing device, wherein the computing device is configured to process the additional stream of image data to identify changes in positions of the forearm.

8. The method of claim 7, wherein the computing device is configured to confirm an accuracy of the positions of the forearm determined from the additional stream of image data based on the positions of the wrist determined from the stream of image data.

9. The method of claim 1, wherein the wearable device has a camera having a field-of-view directed towards the other wrist of the user, the method further comprising:

capturing an image of the other wearable device worn on the other wrist, wherein said capturing of the image of the other wearable device is performed by the camera having the field-of-view directed towards the other wrist;

reiterating capturing of additional images of the other wearable device by the camera having the field-of-view directed towards the other wrist, wherein said capturing of the image of the other wearable device and said reiterating are performed to generate an additional stream of image data;

sending the additional stream of image data to the computing device, wherein the computing device is configured to process the additional stream of image data to identify changes in positions of the other wrist for rendering another virtual hand representative of the other wrist in the head-mounted display.

10. The method of claim 9, further comprising:

capturing an image of a portion of the other wearable device worn on the other wrist, wherein said capturing the image of portion of the other wearable device is performed by the camera on the wearable device;

repeating capturing of additional images of the portion of the other wearable device by the camera on the wearable device to generate another stream of image data; and

sending the other stream of image data to the computing device, wherein the computing device is configured to process the other stream of image data to identify changes in positions of the other wrist,

wherein the computing device is configured to confirm an accuracy of the positions of the other wrist determined from the other stream of image data based on the positions of the other wrist determined from the additional stream of image data.

11. A system to identify positions of a wrist of a user for displaying a virtual hand viewable via a head-mounted display, comprising:

a wearable device configured to be worn on the wrist of the user;

a head-mounted display configured to be worn by the user;

a camera facing the wearable device and having a field-of-view directed towards the user to capture an image of the wearable device, wherein the wearable device includes a plurality of light emitters located on an outer surface of the wearable device, wherein the plurality of light emitters are configured to emit light in a different form compared to emission of light by a plurality of additional light emitters of another wearable device to facilitate distinguishing the wearable device from the other wearable device, wherein the other wearable device is configured to be worn on another wrist of the user,

wherein the camera is configured to repeat capturing of additional images of the wearable device, wherein the image and the additional images are captured to generate a stream of image data,

a computing device coupled to the camera and the head-mounted display,

wherein the camera is configured to send the stream of image data to the computing device,

wherein the computing device is configured to process the image data to identify changes in positions of the wrist for rendering the virtual hand in the head-mounted display.

12. The system of claim 11, wherein the light emitters are spaced apart in a periodic manner across the outer surface to facilitate capturing of the image and the additional images.

13. The system of claim 11, wherein the camera is located within a room in which the wearable device is located, wherein the field-of-view stays the same independent of movement of the head-mounted display.

14. The system of claim 11, wherein the camera is located on an outer surface of the head-mounted display, wherein the field-of-view of the camera moves with movement of the head-mounted display.

15. The system of claim 11, the system further comprising:

a camera located on an outer surface of the other wearable device,

wherein the camera on the other wearable device is configured to capture an image of a portion of the wearable device,

wherein the camera on the other wearable device is configured to repeat capturing of additional images of the portion of the wearable device to generate an additional stream of image data,

wherein the camera on the other wearable device is configured to send the additional stream of image data to the computing device,

wherein the computing device is configured to process the additional stream of image data to identify changes in positions of the wrist.

16. The system of claim 15, wherein the computing device is configured to confirm an accuracy of the positions of the wrist determined from the additional stream of image data based on the positions of the wrist determined from the stream of image data.

17. The system of claim 11, wherein the wearable device has a camera having a field-of-view directed towards a forearm of the user,

wherein the camera of the wearable device is configured to capture an image of the forearm,

wherein the camera of the wearable device is configured to repeat capturing of additional images of the forearm to generate an additional stream of image data,

wherein the camera of the wearable device is configured to send the additional stream of image data to the computing device,

wherein the computing device is configured to process the additional stream of image data to identify changes in positions of the forearm.

18. The system of claim 17, wherein the computing device is configured to confirm an accuracy of the positions of the forearm determined from the additional stream of image data based on the positions of the wrist determined from the stream of image data.

19. The system of claim 11,

wherein the wearable device has a camera having a field-of-view directed towards the other wrist of the user,

wherein the camera that faces the other wearable device is configured to capture an image of the other wearable device,

wherein the camera that faces the other wearable device is configured to reiterate capturing of additional images of the other wearable device, wherein the image of the other wearable device and the additional images of the other wearable device are captured to generate an additional stream of image data,

wherein the camera that faces the other wearable device is configured to send the additional stream of image data to the computing device,

wherein the computing device is configured to process the additional stream of image data to identify changes in positions of the other wrist for rendering another virtual hand representative of the other wrist in the head-mounted display.

20. The system of claim 19,

wherein the camera on the wearable device is configured to capture an image of a portion of the other wearable device worn on the other wrist,

wherein the camera on the wearable device is configured to repeat capturing of additional images of the portion of the other wearable device to generate another stream of image data,

wherein the camera on the wearable device is configured to send the other stream of image data to the computing device,

wherein the computing device is configured to process the other stream of image data to identify changes in positions of the other wrist,

wherein the computing device is configured to confirm an accuracy of the positions of the other wrist determined from the other stream of image data based on the positions of the other wrist determined from the additional stream of image data.

21. A non-transitory computer-readable medium containing program instructions for identifying positions of a wrist of a user for displaying a virtual hand viewable via a head-mounted display, wherein execution of the program instructions by one or more processors of a computer system causes the one or more processors to carry out operations of:

capturing an image of a wearable device worn on the wrist of the user, wherein said capturing of the image is performed by a camera having a field-of-view directed towards the user, wherein the wearable device includes a plurality of light emitters located on an outer surface of the wearable device, wherein the plurality of light emitters are configured to emit light in a different form compared to emission of light by a plurality of additional light emitters of another wearable device to facilitate distinguishing the wearable device from the other wearable device, wherein the other wearable device is worn on another wrist of the user, wherein said capturing is performed when the wearable device is moved to produce gestures that are captured by the camera;

repeating capturing of additional images of the wearable device by the camera, wherein said capturing of the image and said repeating are performed to generate a stream of image data;

sending the stream of image data to a computing device, wherein the computing device is configured to process the image data to identify changes in positions of the wrist for rendering the virtual hand representative of the wrist in the head-mounted display.

22. The non-transitory computer-readable medium of claim 21, wherein the light emitters are spaced apart in a periodic manner across the outer surface to facilitate capturing of the image and the additional images.