Control device for communicating visual information

• US 9,046,927 B2
• Filed: 06/29/2012
• Issued: 06/02/2015
• Est. Priority Date: 10/26/2005
• Status: Active Grant

• Current Assignee: Sony Interactive Entertainment Inc. (Sony Group Corp.)
• Original Assignee: Sony Computer Entertainment Incorporated (Sony Group Corp.)
• Inventors: Marks, Richard, Mikhailov, Anton, Huang, Ennin, Zalewski, Gary M.
• Primary Examiner(s): CARPENTER, ROBERT K
• Application Number:
  US13/539,292
• Publication Number:
  US 20120302349A1
• Time in Patent Office: 1,068 Days
• Field of Search: 463/33, 463/47.1, 463/47.2, 463 49- 56
• US Class Current: 1/1
• CPC Class Codes: A63F 13/211 using inertial sensors, e.g. accelerometers or gyroscopes
• CPC Class Codes: A63F 13/213 comprising photodetecting means, e.g. cameras, photodiodes or infrared cells A63F13/219, A63F13/655 take precedence
• CPC Class Codes: A63F 13/216 using geographical information, e.g. location of the game device or player using GPS
• CPC Class Codes: A63F 13/219 for aiming at specific areas on the display, e.g. light-guns
• CPC Class Codes: A63F 13/22 Setup operations, e.g. calibration, key configuration or button assignment
• CPC Class Codes: A63F 13/235 using a wireless connection, e.g. infrared or piconet
• CPC Class Codes: A63F 13/24 Constructional details thereof, e.g. game controllers with detachable joystick handles
• CPC Class Codes: A63F 13/25 Output arrangements for video game devices
• CPC Class Codes: A63F 13/28 responding to control signals received from the game device for affecting ambient conditions, e.g. for vibrating players' seats, activating scent dispensers or affecting temperature or light controlling the output signals based on the game progress A63F13/50
• CPC Class Codes: A63F 13/825 Fostering virtual characters
• CPC Class Codes: G06F 2203/013 Force feedback applied to a game
• CPC Class Codes: G06F 2203/0333 Ergonomic shaped mouse for one hand
• CPC Class Codes: G06F 2203/0381 Multimodal input, i.e. interface arrangements enabling the user to issue commands by simultaneous use of input devices of different nature, e.g. voice plus gesture on digitizer
• CPC Class Codes: G06F 2203/0383 Remote input, i.e. interface arrangements in which the signals generated by a pointing device are transmitted to a PC at a remote location, e.g. to a PC in a LAN
• CPC Class Codes: G06F 3/016 Input arrangements with force or tactile feedback as computer generated output to the user
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06F 3/0383 Signal control means within the pointing device
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06T 2207/10016 : Video; Image sequence
• CPC Class Codes: G06T 2207/10024 : Color image
• CPC Class Codes: G06T 2207/30241 : Trajectory
• CPC Class Codes: G06T 2207/30242 : Counting objects in image
• CPC Class Codes: G06T 7/20 : Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51

Abstract

Methods and systems for processing input by a computing device are presented. One method includes operations for receiving images of a control device that includes an object section, and for determining a location of the control device utilizing image analysis for each captured image. Additionally, the movement of the control device is tracked based on the determined locations, where the tracking of the movement includes receiving inertial sensor information obtained by sensors in the control device, and determining an orientation of the control device based on the sensor information. Additionally, the method includes an operation for translating the movement and orientation of the control device into input for a game executing in the computing device, where the input is translated into a motion and orientation of an object in the game based on the movement of the control device.

Claims

1. A method for processing input by a computing device, the method comprising:

receiving images of a control device that includes an object section having a spherical shape;

determining locations of the control device in space during a period of time utilizing image analysis to identify a circular area corresponding to the spherical shape;

receiving inertial sensor information obtained by sensors in the control device;

determining a plurality of orientations of the control device based within the period of time based on the sensor information;

tracking movement of the control device based on the determined locations and the determined orientations; and

translating the movement of the control device into input for a game executing in the computing device, the input for the game being translated into a motion of an object in the game, wherein operations of the method are executed through a processor.

2. The method as recited in claim 1, wherein the control device includes wireless capability for communication with the computing device, wherein the inertial sensor information is received utilizing the wireless capability.

3. The method as recited in claim 1, wherein the movement of the control device is tracked in a three-dimensional space.

4. The method as recited in claim 1, further including:

receiving sound cues originated at the control device; and

combining the sound cues with the image analysis to track the control device movement.

5. The method as recited in claim 1, wherein the movement of the control device enables the control device to act as a pointing device on a display.

6. The method as recited in claim 1, wherein the movement of the control device created by a user holding the control device creates a corresponding motion of a flashlight held by an avatar in the game, wherein the avatar illuminates areas of a display with the flashlight based on the motion of the flashlight held by the avatar.

7. The method as recited in claim 1, wherein the movement of the control device is utilized to move a cursor on a display based on the orientation of the control device, and wherein pressing a button on the control device selects an item associated with the cursor.

8. The method as recited in claim 1, wherein the control device is defined to be used as a weapon to fire virtual bullets on a display.

9. The method as recited in claim 1, wherein the control device is used as a pointer to draw a line on a display.

10. The method as recited in claim 1, wherein the control device is used as a virtual sword and the movement of the control device translates into movement of the virtual sword in the game.

11. A non-transitory computer-readable storage medium storing a computer program for processing input by a computing device, the computer-readable storage medium comprising:

program instructions for receiving images of a control device that includes an object section having a spherical shape;

program instructions for determining locations of the control device in space during a period of time utilizing image analysis to identify a circular area corresponding to the spherical shape;

program instructions for receiving inertial sensor information obtained by sensors in the control device;

program instructions for determining a plurality of orientations of the control device within the period of time based on the sensor information;

program instructions for tracking movement of the control device based on the determined locations and the determined orientations; and

program instructions for translating the movement of the control device into input for a game executing in the computing device, the input for the game being translated into a motion of an object in the game.

12. The storage medium as recited in claim 11, wherein the control device includes wireless capability for communication with the computing device, wherein the inertial sensor information is received utilizing the wireless capability.

13. The storage medium as recited in claim 11, wherein the movement of the control device is tracked in a three-dimensional space.

14. The storage medium as recited in claim 11, further including:

program instructions for receiving sound cues originated at the control device; and

program instructions for combining the sound cues with the image analysis to track the control device movement.

15. The storage medium as recited in claim 11, wherein the movement of the control device enables the control device to act as a pointing device on a display.

16. The storage medium as recited in claim 11, wherein the movement of the control device created by a user holding the control device creates a corresponding motion of a flashlight held by an avatar in the game, wherein the avatar illuminates areas of a display with the flashlight based on the motion of the flashlight held by the avatar.

17. The storage medium as recited in claim 11, wherein the movement of the control device is utilized to move a cursor on a display based on the orientation of the control device, and wherein pressing a button on the control device selects an item associated with the cursor.

18. The storage medium as recited in claim 11, wherein the control device is defined to be used as a weapon to fire virtual bullets on a display.

19. The storage medium as recited in claim 11, wherein the control device is used as a pointer to draw a line on a display.

20. The storage medium as recited in claim 11, wherein the control device is used as a virtual sword and the movement of the control device translates into movement of the virtual sword in the game.