Selective sound source listening in conjunction with computer interactive processing

• US 7,760,248 B2
• Filed: 05/04/2006
• Issued: 07/20/2010
• Est. Priority Date: 07/27/2002
• Status: Expired due to Term

• Current Assignee: Sony Interactive Entertainment Inc. (Sony Group Corp.)
• Original Assignee: Sony Computer Entertainment Incorporated (Sony Group Corp.)
• Inventors: Marks, Richard L., Mao, Xiadong
• Primary Examiner(s): Ye; Lin
• Assistant Examiner(s): DAGNEW, MEKONNEN D
• Application Number:
  US11/429,133
• Publication Number:
  US 20060204012A1
• Time in Patent Office: 1,538 Days
• Field of Search: 381/26, 381/92, 381/356, 381/357, 381111-115, 381/95, 381/122, 381/91, 348/231.4, 348/515, 348/632, 348/736, 348/738, 348/14.09
• US Class Current: 348/231.4
• CPC Class Codes: A63F 13/00 Video games, i.e. games using an electronically generated display having two or more dimensions gaming systems which provide a financial reward G07F17/32
• CPC Class Codes: A63F 13/213 comprising photodetecting means, e.g. cameras, photodiodes or infrared cells A63F13/219, A63F13/655 take precedence
• CPC Class Codes: A63F 13/215 comprising means for detecting acoustic signals, e.g. using a microphone
• CPC Class Codes: A63F 13/424 involving acoustic input signals, e.g. by using the results of pitch or rhythm extraction or voice recognition
• CPC Class Codes: A63F 2300/1012 involving biosensors worn by the player, e.g. for measuring heart beat, limb activity
• CPC Class Codes: A63F 2300/1093 using visible light
• CPC Class Codes: A63F 2300/6045 for mapping control signals received from the input arrangement into game commands
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: H04S 7/30 Control circuits for electronic adaptation of the sound field

Abstract

A method and apparatus for capturing image and sound during interactivity with a computer program is provided. The apparatus includes an image capture unit that is configured to capture one or more image frames. Also provided is a sound capture unit. The sound capture unit is configured to identify one or more sound sources. The sound capture unit generates data capable of being analyzed to determine a zone of focus, at which to process sound to the substantial exclusion of sounds outside of the zone of focus. In this manner, sound that is captured and processed for the zone of focus is used for interactivity with the computer program.

Claims

1. An apparatus for capturing image and sound during interactivity with a computer game in a game environment, comprising:

an image capture unit for capturing one or more image frames of the game environment, the captured one or more image frames being processed to identify an active image area, the active image area being defined by a location of a user during interactivity with the computer game, the active image area being continuously updated so as to track movement of the user; and

a sound capture unit for capturing sound from the game environment, the sound capture unit identifying one or more sound sources, the sound capture unit generating data capable of being analyzed to determine a zone of focus at which to process sound to the substantial exclusion of sounds outside of the zone of focus, the excluded sounds being sounds which are not central to gameplay of the computer game, the zone of focus being substantially determined by the active image area, the zone of focus being updated as the active image area is updated so as to track interactive sounds produced by the user, wherein sound that is captured and processed for the zone of focus is used for interactivity with the computer game.

2. An apparatus for capturing image and sound during interactivity with a computer game as recited in claim 1, wherein the sound capture unit includes an array of microphones, the array of microphones for receiving sound from the one or more sound sources, the sounds of the one or more sound sources defining sound paths to each of the microphones.

3. An apparatus for capturing image and sound during interactivity with a computer game as recited in claim 2, wherein the sound paths include particular delays that enable calculation of direction of each of the one or more sound sources relative to the apparatus for capturing image and sound.

4. An apparatus for capturing image and sound during interactivity with a computer game as recited in claim 1, further comprising:

a computing system for interfacing with the apparatus for capturing image and sound, the computing system including,

a processor, and

memory, the memory for storing at least part of the computer game and selective sound source listening code, the selective sound source listening code enabling the identification of which of the one or more sound sources identify as the zone of focus.

5. An apparatus for capturing image and sound during interactivity with a computer game as recited in claim 1, wherein the sound capture unit includes at least four microphones, and one of the four microphones is not in a same plane as the others.

6. An apparatus for capturing image and sound during interactivity with a computer game as recited in claim 5, wherein the four microphones define a spatial volume.

7. An apparatus for capturing image and sound during interactivity with a computer program as recited in claim 6, wherein the spatial volume is defined as a volume of focus for listening during interactivity with the computer game.

8. An apparatus for capturing image and sound during interactivity with a computer game as recited in claim 1, wherein the image capture unit is a camera and the sound capture unit is defined by an array of two or more microphones.

9. The apparatus of claim 1, wherein the image capture unit has an orientation defined by the active image area.

10. The apparatus of claim 9, wherein the orientation includes a focus point of the image capture unit defined by the active image area.

11. A method for selective sound source listening during interactivity with a computer game in a game environment, comprising:

receiving input from one or more sound sources in the game environment at two or more sound source capture microphones;

determining delay paths from each of the sound sources;

identifying a direction for each of the received inputs of each of the one or more sound sources; and

filtering out sound sources that are not in an identified direction of a zone of focus, the filtered out sound sources being sound sources which are not central to gameplay of the computer game, the zone of focus supplying the sound source for the interactivity with the computer game;

wherein filtering receives processed input data after analysis by an image capture unit, the image capture unit producing image data to at least partially identify the zone of focus, the zone of focus being continuously updated so as to track the sound source for the interactivity with the computer game, and the image capture device being directionally positioned to receive image input for the computer game.

12. A method for selective sound source listening during interactivity with a computer game as recited in claim 11, wherein the computer game receives interactive input from both image data and sound data, the sound data being from the sound source of the zone of focus.

13. A method for selective sound source listening during interactivity with a computer game as recited in claim 11, wherein the two or more sound capture microphones include at least four microphones, and at least one of the four microphones is on a different plane than the others.

14. A method for selective sound source listening during interactivity with a computer game as recited in claim 13, wherein identifying the direction for each of the received inputs of each of the one or more sound sources includes processing a triangulation algorithm, the triangulation algorithm defining the direction that is relative to a location at which input is received from the one or more sound sources at the two or more sound source capture microphones.

15. A method for selective sound source listening during interactivity with a computer game as recited in claim 14, further comprising:

buffering the received input from the one or more sound sources associated with the two or more sound source capture microphones; and

delay processing the received buffered inputs;

the filtering further including,

selecting the one of the sound sources, the selected sound source output being a summation of sound from each of the sound source capture microphones.

16. The method of claim 11, wherein the image capture unit has an orientation defined by the zone of focus.

17. The method of claim 16, wherein the orientation includes a focus point of the image capture unit defined by the zone of focus.

18. A game system, comprising:

an image-sound capture device, the image-sound capture device being configured to interface with a computing system that enables execution of an interactive computer game, the interactive computer game facilitating interactivity in a game environment, the image-capture device including,

video capture hardware positioned to capture video from a zone of focus in the game environment, and

an array of microphones for capturing sound from one or more sound sources in the game environment, each sound source being identified and associated with a direction relative to the image-sound capture device, the zone of focus associated with the video capture hardware is used to at least partially identify one of the sound sources at the direction that is in the proximity of the zone of focus, the zone of focus being updated so as to track the identified one of the sound sources.

19. A game system as recited in claim 18, wherein the video capture hardware receives video data to enable interactivity with features of the computer game.

20. A game system as recited in claim 18, wherein the sound source in the proximity of the zone of focus enables interactivity with the computer game or voice communication with other game users.

21. A game system as recited in claim 20, wherein sound sources outside of the zone of focus are filtered out of interactivity with the computer game, the filtered out sound sources being sound sources which are not central to gameplay of the computer game.

22. The game system of claim 18, wherein the video capture hardware has an orientation defined by the zone of focus.

23. The game system of claim 22, wherein the orientation includes a focus point of the video capture hardware defined by the zone of focus.