Computer image and audio processing of intensity and input devices for interfacing with a computer program

• US 7,627,139 B2
• Filed: 05/04/2006
• Issued: 12/01/2009
• Est. Priority Date: 07/27/2002
• Status: Expired due to Term

• Current Assignee: Sony Interactive Entertainment Inc. (Sony Group Corp.)
• Original Assignee: Sony Computer Entertainment Incorporated (Sony Group Corp.)
• Inventors: Marks, Richard L., Zalewski, Gary M., Mao, Xiadong
• Primary Examiner(s): Lu; Tom Y
• Application Number:
  US11/429,414
• Publication Number:
  US 20060277571A1
• Time in Patent Office: 1,307 Days
• Field of Search: 382/103, 345/156, 463/36
• US Class Current: 382/103
• CPC Class Codes: A63F 13/00 Video games, i.e. games using an electronically generated display having two or more dimensions gaming systems which provide a financial reward G07F17/32
• CPC Class Codes: A63F 13/211 using inertial sensors, e.g. accelerometers or gyroscopes
• CPC Class Codes: A63F 13/213 comprising photodetecting means, e.g. cameras, photodiodes or infrared cells A63F13/219, A63F13/655 take precedence
• CPC Class Codes: A63F 13/215 comprising means for detecting acoustic signals, e.g. using a microphone
• CPC Class Codes: A63F 13/219 for aiming at specific areas on the display, e.g. light-guns
• CPC Class Codes: A63F 13/42 by mapping the input signals into game commands, e.g. mapping the displacement of a stylus on a touch screen to the steering angle of a virtual vehicle
• CPC Class Codes: A63F 13/54 involving acoustic signals, e.g. for simulating revolutions per minute [RPM] dependent engine sounds in a driving game or reverberation against a virtual wall
• CPC Class Codes: A63F 13/655 by importing photos, e.g. of the player
• CPC Class Codes: A63F 13/837 Shooting of targets
• CPC Class Codes: A63F 2300/105 using inertial sensors, e.g. accelerometers, gyroscopes
• CPC Class Codes: A63F 2300/1081 Input via voice recognition
• CPC Class Codes: A63F 2300/1087 comprising photodetecting means, e.g. a camera
• CPC Class Codes: A63F 2300/6045 for mapping control signals received from the input arrangement into game commands
• CPC Class Codes: A63F 2300/6081 generating an output signal, e.g. under timing constraints, for spatialization
• CPC Class Codes: A63F 2300/695 Imported photos, e.g. of the player
• CPC Class Codes: A63F 2300/8076 Shooting
• CPC Class Codes: F41A 33/02 Light- or radiation-emitting guns ; Light- or radiation-sensitive guns; Cartridges carrying light emitting sources, e.g. laser
• CPC Class Codes: F41J 5/02 Photo-electric hit-detector systems comprising a light-emitting device for simulating the firing of a gun F41G3/2622; detecting by optical means G01V8/10
• CPC Class Codes: G06F 3/0325 using a plurality of light emitters or reflectors or a plurality of detectors forming a reference frame from which to derive the orientation of the object, e.g. by triangulation or on the basis of reference deformation in the picked up image

Abstract

A method for determining an intensity value of an interaction with a computer program is described. The method and device includes capturing an image of a capture zone, identifying an input object in the image, identifying an initial value of a parameter of the input object, capturing a second image of the capture zone, and identifying a second value of the parameter of the input object. The parameter identifies one or more of a shape, color, or brightness of the input object and is affected by human manipulation of the input object. The extent of change in the parameter is calculated, which is the difference between the second value and the first value. An activity input is provided to the computer program, the activity input including an intensity value representing the extent of change of the parameter. A method for detecting an intensity value from sound generating input objects, and a computer video game are also described. A game controller having LEDs, sound capture and generation, or an accelerometer is also described.

Claims

1. A computer executing a program that renders a method for identifying a position and orientation of a controller, the method comprising:

receiving video images of a focus zone containing a controller, the controller having a plurality of light sources and all of the video images being captured via a single camera;

identifying an orientation of the controller by locating the light sources in the video images, and measuring relative distances between the light sources in the video images, the light sources in the video images further being examined to identify a distance of the controller to the single camera;

identifying a position of the controller in the focus zone by identifying a position of the light sources in the video images;

generating data representing the orientation and position of the controller; and

triggering an input at the computer program, the input being selected based on the generated data representing the orientation and position of the controller.

2. The method of claim 1, wherein the identifying the orientation including determining a pitch angle, a yaw angle, and a roll angle of the controller.

3. The method of claim 1, further comprising determining at least one of a player number or controller actuation, the player number being determined by correlating a unique light pattern to each of a plurality of player numbers, and identifying the one of the player numbers that corresponds to a pattern formed from the light, the player number being selectable by a user of the controller, the controller actuation being determined by identifying a modulation of one or more of the lights which are modulated in response to the controller actuation.

4. A game controller comprising:

an interface having a plurality of buttons for actuation by a user; and

a plurality of lights facing in a general direction, the lights being viewable by an image capture device when the image capture device is in the general direction from the controller, the game controller communicating at least one of a player number identification, a position of the controller, or indication of the actuation to the image capture device using the lights;

wherein the plurality of lights are arranged along a first row and a second row, wherein the second row is arranged below the first row.

5. The game controller of claim 4 further comprising a speaker, the speaker generating at least one of an audible or ultrasonic sound in response to user interaction with the interface.

6. The game controller of claim 4 further comprising a microphone for receiving verbal speech, the verbal speech being encoded into a data stream, the data stream being incorporated into a communication signal transmitted to a computer system.

7. A method for processing captured light data that is used for interfacing with a computer game system during a session, comprising:

(a) generating, at a controller device, a first set of light signals that are to be captured by a video capture device that is coupled to the computer gaming system, the first set of light signals defining a first state for tracking the controller device;

(b) generating, at the controller device, a second set of light signals that are to be captured by the video capture device, the second set of light signals defining a second state for data transmission with the computer game system; and

(c) interleaving between the first state and the second state automatically during the session, such that tracking of the controller device can occur periodically during data transmission with the computer game system.

8. A method for processing captured light data that is used for interfacing with a computer game system during a session as recited in claim 7, wherein the interleaving is facilitated by time division multiplexing between the first state and the second state during the session.

9. A method for processing captured light data that is used for interfacing with a computer game system during a session as recited in claim 7, wherein the first set of light signals are defined by two or more light emitting diodes, and an ON/OFF state of each of the first set of light signals defines a tracking pattern.

10. A method for processing captured light data that is used for interfacing with a computer game system during a session as recited in claim 7, wherein the second set of light signals are defined by two or more light emitting diodes, and

setting an ON/OFF state of selected ones of the second set of light signals to define one of a plurality of data transmission patterns.

11. A method for processing captured light data that is used for interfacing with a computer game system during a session as recited in claim 10, wherein each of the plurality of data transmission patters defines a particular command or interaction provided to the computer game system.

12. A method for processing captured light data that is used for interfacing with a computer game system during a session as recited in claim 7, wherein the computer game system interfaces with one or more computer game programs.

13. A method for processing captured light data that is used for interfacing with a computer game system during a session as recited in claim 9, wherein the tracking pattern identifies a controller identifier.

14. A method for processing captured light data that is used for interfacing with a computer game system during a session as recited in claim 7, further comprising:

modulating the second set of light signals when in the second state to communicate a particular command or an interaction to the computer game system.

15. A method for processing captured light data that is used for interfacing with a computer game system during a session as recited in claim 7, wherein the controller device is a second controller device, and the second controller device being configured to process (a)-(c), so as to enable tracking of the second controller device independently of other controller devices.

16. A controller for interfacing with a computer game being displayed on a screen, comprising:

the controller being hand-held, the controller having at least two lights on a surface of the controller, the lights being operable to convey position and orientation of the controller using light data captured by an image capture device, the controller having an interface defined by a plurality of buttons, the plurality of buttons being selectable to cause input to the computer game; and

the controller including an accelerometer for determining directional data relative to a pull of gravity, the directional data configured to augment the position and orientation of the controller.

17. The controller of claim 16, wherein the lights further provide input data selected from one or more of a player number identification or indication of actuation of the buttons.

18. The controller of claim 16, wherein the orientation defining one or more of a pitch angle, a yaw angle, or a roll angle of the controller.

19. The controller of claim 17, wherein the lights communicate data by one of an ON state, an OFF state, a modulation of the ON/OFF states, or a plurality of ON/OFF state patterns.

20. The controller of claim 16, wherein some of the lights are oriented along a line on the surface of the controller, orientation along the line enables identification of the orientation.