System and method of audio processing

• US 8,932,134 B2
• Filed: 01/23/2009
• Issued: 01/13/2015
• Est. Priority Date: 02/18/2008
• Status: Active Grant

• Current Assignee: Sony Interactive Entertainment Europe Limited (Sony Group Corp.)
• Original Assignee: Sony Computer Entertainment Europe Limited (Sony Group Corp.)
• Inventors: Andall, Hogarth
• Primary Examiner(s): Rowland, Steve
• Application Number:
  US12/866,274
• Publication Number:
  US 20100323793A1
• Time in Patent Office: 2,181 Days
• Field of Search: 463/35, 381/58, 381/300, 381/303
• US Class Current: 463/35
• CPC Class Codes: A63F 13/10 Control of the course of the game, e.g. start, progess, end
• CPC Class Codes: A63F 13/40 Processing input control signals of video game devices, e.g. signals generated by the player or derived from the environment
• CPC Class Codes: A63F 13/45 Controlling the progress of the video game
• CPC Class Codes: A63F 13/54 involving acoustic signals, e.g. for simulating revolutions per minute [RPM] dependent engine sounds in a driving game or reverberation against a virtual wall
• CPC Class Codes: A63F 2300/6081 generating an output signal, e.g. under timing constraints, for spatialization
• CPC Class Codes: A63F 2300/8088 involving concurrently several players in a non-networked game, e.g. on the same game console
• CPC Class Codes: H04S 2400/11 Positioning of individual sound objects, e.g. moving airplane, within a sound field H04S2420/13 takes precedence
• CPC Class Codes: H04S 7/301 Automatic calibration of stereophonic sound system, e.g. with test microphone
• CPC Class Codes: H04S 7/303 Tracking of listener position or orientation

Abstract

A method of audio processing for an entertainment device operable to communicate with a game controller, the method comprising the steps of generating a source sound for reproduction by a plurality of loudspeakers at a current position where acoustic signals from the plurality of loudspeakers are coincident (hereafter referred to as a ‘sweet spot’), requesting that the user steers the sound using the game controller until the user considers the sound to be centered upon them, and interpreting user inputs from the game controller to adjust the output timing of the source sound data signals for reproduction by respective ones of the plurality of loudspeakers in accordance with the user directional inputs so as to move the sweet spot.

Claims

1. A method of audio processing in which a video camera is used to detect a position of a user relative to a position of the video camera, the video camera having an associated microphone located substantially at the position of the camera and the camera being placed in a position other than the user'"'"'s listening position, the method comprising the steps of:

calibrating, by one or more processors, the position of the video camera relative to a plurality of loudspeakers by (i) generating test signals for the plurality of loudspeakers;

(ii) measuring acoustic signals from the plurality of loudspeakers with the microphone located substantially at the position of the camera; and

(iii) adjusting output timing of sound data signals for reproduction by respective ones of the plurality of loudspeakers so that simultaneous source ones of the sound data signals are then timed to cause corresponding ones of the acoustic signals from the respective loudspeakers to reach the microphone substantially simultaneously so as to set a position where the acoustic signals from selected ones of the plurality of loudspeakers are coincident at a sweet spot to the position of the camera;

detecting a position of the user relative to the camera from images captured by the camera, thereby estimating the position of the user relative to an estimated position of the sweet spot; and

adjusting, by the one or more processors, the output timing of the sound data signals for reproduction by the respective loudspeakers in accordance with the estimated position of the user relative to the position of the camera so as to move the estimated position of the sweet spot to the estimated position of the user.

2. A method of audio processing according to claim 1, further comprising the steps of:

estimating the position of the sweet spot; and

estimating the position of the user.

3. A method according to claim 2, in which the step of estimating the position of the user comprises capturing and analysing data from a non-acoustic sensor.

4. A method according to claim 2, in which the step of estimating the position of the user comprises capturing an image encompassing some or all of the user, and analysing the image to identify the position of the user.

5. A method according to claim 4, comprising the step of augmenting analysis of one of the captured images by the use of distance data corresponding to captured image data and obtained from an image capture device further equipped with a distance measuring arrangement.

6. A method according to claim 2, in which the step of estimating the position of the user comprises receiving a radio signal from a radio transmitting device carried by the user by two or more phased array antennas, and analysing respective angles of reception at the phased arrays to detect a position of transmission of the radio signal.

7. A method according to claim 2, in which the step of estimating the position of the user relative to the estimated position of the sweet spot comprises comparing the estimated position of the user with the estimated position of the sweet spot to determine the relative difference in position.

8. A method according to claim 2, in which the step of estimating the position of the sweet spot comprises:

capturing an image of a room housing the loudspeakers;

estimating a centre point of the room by analysis of the captured image; and

estimating the position of the sweet spot as the centre point of the room.

9. A method according to claim 2, in which the step of estimating the position of the sweet spot comprises:

capturing an image of a room housing the loudspeakers;

analysing the captured image to identify a seat, or if there are two or more seats, the seat closest to a lateral centre of the captured image; and

estimating the position of the identified seat by analysis of the captured image.

10. A method according to claim 8, comprising the step of augmenting analysis of one of the captured images by the use of distance data corresponding to captured image data and obtained from an image capture device further equipped with a distance measuring arrangement.

11. A method according to claim 2, the method being carried out by an entertainment device operable to interact with two or more users.

12. A method according to claim 11, where the two or more users interact with the entertainment device in turn, and in which the step of adjusting the output timing comprises moving an effective position of the sweet spot to the position of the currently interacting user.

13. A method according to claim 11, where the two or more users interact with the entertainment device simultaneously, and in which the step of adjusting the output timing comprises:

adjusting the output timing of some or all of respective sound data signals associated with each respective user'"'"'s interaction in accordance with their respective sweet spots.

14. A method according to claim 2, where two or more users are identified, in which the step of estimating the position of the sweet spot relative to the estimated position of the user comprises:

identifying a plurality of users within a captured video image;

defining a notional boundary dependent upon a lateral and axial extent of a group of some or all of these users; and

adjusting the output timing of the sound data signals for reproduction by the respective ones of the plurality of loudspeakers in accordance with a difference between the estimated position of the sweet spot and the notional boundary, so that resultant acoustic signals reach the notional boundary substantially simultaneously.

15. An entertainment device, comprising:

a video camera configured to detect a position of a user relative to a position of the video camera;

a microphone associated with the video camera, the microphone being located substantially at the position of the video camera and the video camera being placed in a position other than the user'"'"'s listening position; and

a processor operatively coupled to the camera and the microphone, the processor being configured to;

calibrate the position of the video camera relative to a plurality of loudspeakers by (i) generating test signals for the plurality of loudspeakers;

(ii) measuring acoustic signals from the plurality of loudspeakers with the microphone located substantially at the position of the camera; and

(iii) adjusting output timing of sound data signals for reproduction by respective ones of the plurality of loudspeakers so that simultaneous source ones of the sound data signals are then timed to cause corresponding ones of the acoustic signals from the respective loudspeakers to reach the microphone substantially simultaneously so as to set a position where the acoustic signals from selected ones of the plurality of loudspeakers are coincident at a sweet spot to the position of the camera;

detect a position of the user relative to the camera from images captured by the camera, thereby estimating the position of the user relative to an estimated position of the sweet spot; and

adjust the output timing of the sound data signals for reproduction by the respective loudspeakers in accordance with the estimated position of the user relative to the position of the camera so as to move the estimated position of the sweet spot to the estimated position of the user.

16. An entertainment device according to claim 15, comprising:

a non-acoustic sensor;

a coincident signal estimator operable to estimate a physical location at which acoustic signals from the plurality of loudspeakers are coincident at the sweet spot;

a sensor analyser operable to estimate a physical location of the user from non-acoustic sensor data generated by the non-acoustic sensor; and

wherein

the processor is operable to adjust the output timing of the sound data signals for reproduction by respective ones of the plurality of loudspeakers in accordance with a difference between the estimated position of the user and the estimated position of the sweet spot so as to move an effective position of the sweet spot towards the estimated position of the user.

17. An entertainment device according to claim 16, in which the non-acoustic sensor is the video camera.

18. An entertainment device according to claim 17, in which the video camera comprises a distance measuring arrangement.

19. An entertainment device according to claim 17, in which the video camera includes the microphone.

20. An entertainment device according to claim 16, in which the non-acoustic sensor is a phased radio array.

21. A non-transitory computer-readable storage medium having instructions stored thereon, the instructions, when executed by a processor, cause the processor to perform method of audio processing in which a video camera is used to detect a position of a user relative to a position of the video camera, the video camera having an associated microphone located substantially at the position of the camera and the camera being placed in a position other than the user'"'"'s listening position, the method comprising the steps of:

calibrating the position of the video camera relative to a plurality of loudspeakers by (i) generating test signals for the plurality of loudspeakers;

(ii) measuring acoustic signals from the plurality of loudspeakers with the microphone located substantially at the position of the camera; and

(iii) adjusting output timing of sound data signals for reproduction by respective ones of the plurality of loudspeakers so that simultaneous source ones of the sound data signals are then timed to cause corresponding ones of the acoustic signals from the respective loudspeakers to reach the microphone substantially simultaneously so as to set a position where the acoustic signals from selected ones of the plurality of loudspeakers are coincident at a sweet spot to the position of the camera;

detecting a position of the user relative to the camera from images captured by the camera, thereby estimating the position of the user relative to an estimated position of the sweet spot; and

adjusting the output timing of the sound data signals for reproduction by the respective loudspeakers in accordance with the estimated position of the user relative to the position of the camera so as to move the estimated position of the sweet spot to the estimated position of the user.

22. A non-transitory computer-readable storage medium according to claim 21, the method further comprising:

estimating the position of the sweet spot; and

estimating the position of the user.