Sound generator synchronized with image display

• US 5,862,229 A
• Filed: 10/09/1997
• Issued: 01/19/1999
• Est. Priority Date: 06/12/1996
• Status: Expired due to Term

• Current Assignee: Nintendo Company Limited
• Original Assignee: Nintendo Company Limited
• Inventors: Shimizu, Hideaki
• Primary Examiner(s): Kuntz, Curtis A.
• Assistant Examiner(s): Mei, Xu
• Application Number:
  US08/948,250
• Time in Patent Office: 467 Days
• Field of Search: 381/17-22, 381/104, 381/105, 381/107, 381/109, 381/119, 463/32-35, 352/12, 352/19, 345/425, 345/121
• US Class Current: 381/17
• CPC Class Codes: A63F 13/44 involving timing of operations, e.g. performing an action within a time slot
• CPC Class Codes: A63F 13/52 involving aspects of the displayed game scene
• CPC Class Codes: A63F 13/5255 according to dedicated instructions from a player, e.g. using a secondary joystick to rotate the camera around a player's character
• CPC Class Codes: A63F 13/54 involving acoustic signals, e.g. for simulating revolutions per minute [RPM] dependent engine sounds in a driving game or reverberation against a virtual wall
• CPC Class Codes: A63F 13/95 Storage media specially adapted for storing game information, e.g. video game cartridges
• CPC Class Codes: A63F 2300/206 Game information storage, e.g. cartridges, CD ROM's, DVD's, smart cards
• CPC Class Codes: A63F 2300/6063 for sound processing
• CPC Class Codes: A63F 2300/6081 generating an output signal, e.g. under timing constraints, for spatialization
• CPC Class Codes: A63F 2300/63 for controlling the execution of the game in time
• CPC Class Codes: A63F 2300/64 for computing dynamical parameters of game objects, e.g. motion determination or computation of frictional forces for a virtual car
• CPC Class Codes: A63F 2300/66 for rendering three dimensional images
• CPC Class Codes: A63F 2300/6676 by dedicated player input
• CPC Class Codes: H04S 1/00 Two-channel systems H04S5/00, H04S7/00 take precedence

Abstract

Digital sound data are stored in a data memory. When a first display object (such as an enemy character, a waterfall, or the like) is displayed in a three-dimensional manner on a display screen, an audio processor reads out corresponding sound source data from the data memory to produce first and second sound source data. The first and second sound source data are converted into analog audio signals by digital-to-analog converters, and then used to drive left and right speakers. The audio processor calculates a delay between generation of the first and second sound source data based on a direction of the first display object as viewed from a "virtual" camera which might correspond for example to one of the graphical characters in the three-dimensional display. The audio processor also controls sound volume levels for the first and second sound source data depending on the distance between the first display object and the virtual camera. Using the present invention, realistic sound effects consistent with and synchronized to visual changes of a three-dimensional image are generated by selectively controlling the timing and volume of sound source data provided to the left and right speakers.

Claims

1. A method for generating realistic three-dimensional sound signals in a three-dimensional (3D) image displayed by an image display device, comprising the steps of:

determining a display position of a first object displayed on the displayed 3D image;

determining a display position of a second object displayed in the displayed 3D image, the second object being associated with a sound signal;

selectively delaying when the sound signal is applied to one of the left and right speakers relative to when the sound is applied to the other of the left and right speakers based on relative movement in the displayed 3D image between the display position of the first object and the display position of the second object.

2. The method of claim 1, wherein there is no delay under a predetermined condition when the first and second objects are located within a range of display positions in the 3D image.

3. The method in claim 1, further comprising:

determining a direction of the second display object relative to the first display object,

wherein the amount of delay is based on the determined direction of the second display object relative to the first display object.

4. The method in claim 3, wherein as the second display object moves from left of the first display object to right of the first display object, the delay of the sound signal applied to the right speaker decreases to zero, and thereafter, the delay of the sound signal applied to the left speaker increases.

5. The method in claim 4, wherein the increase and decrease are linear, and wherein there is no delay for a range of positions of the second display object substantially directly in front of the first display object.

6. The method in claim 3, wherein a maximum delay is less than or equal to one-fifth of a unit time used to process one increment of sound signal.

7. The method in claim 3, wherein when the second display object is at 0°

relative to the first display object, the delay of the sound signal provided to the right speaker is a maximum.

8. The method in claim 3, wherein when the second displayed object is at 90°

relative to the first display object, there is no delay of the sound signal provided to either the left and right speakers.

9. The method in claim 3, wherein when the second display object is at 180°

relative to the first display object, the delay of the sound signal provided to the left speaker is a maximum.

10. The method in claim 3, wherein the amount of delay is based on an angle defined by a line of sight between the first display object and the second display object.

11. The method in claim 10, wherein a lookup table is provided with a list of delays for various angles, the method further comprising:

determining a current angle corresponding to the line of sight between the first display object and the second display object;

accessing the lookup table with the determined angle;

retrieving the delay corresponding to the determined angle; and

applying the retrieved delay to the sound signal provided to one of the left and right speakers.

12. The method in claim 1, wherein the first display object is a character in the 3D image.

13. A method for generating realistic three-dimensional sound signals in a three-dimensional (3D) image displayed by an image display device, comprising the steps of:

determining a display position of a first object displayed on the displayed 3D image;

determining a display position of a second object displayed in the displayed 3D image, the second object being associated with a sound signal;

determining a distance between the first display object and the second display object,

controlling a volume of a sound signal associated with the second display object based on the determined distance.

14. The method in claim 13, wherein the sound signal is generated on a first channel applied to a left speaker and a second channel applied to a right speaker, and wherein different volumes are applied to sound signals provided to the left and right speakers in accordance with relative movement between the second display object and the first display object.

15. The method in claim 14, wherein when as the second display object moves from left of the first display object to right of the first display object, the volume of the sound signal applied to the left speaker decreases and the volume of the sound signal applied to the right speaker increases.

16. The method in claim 13, wherein for distances up to a first distance, volume changes based on a linear function, and wherein for distances greater than the first distance, volume changes based on an exponential function.

17. A method for generating realistic three-dimensional sound signals in a three-dimensional (3D) image displayed by an image display device, comprising the steps of:

determining a display position of a first object displayed on the displayed 3D image;

determining a display position of a second object displayed in the displayed 3D image the second object being associated with a sound signal;

determining a current distance between the first display object and the second display object;

determining a current relative direction between the first display object and the second display object;

changing how a sound signal associated with the second display object is generated based on the relative direction and the current distance between the first display object and the second display object.

18. The method in claim 17, wherein the sound signal is generated on a first channel applied to a left speaker and a second channel applied to a right speaker, and wherein different volumes are applied to sound signals provided to the left and right speakers in accordance with relative movement between the first and second display objects.

19. The method in claim 17, wherein the sound signal is generated on a first channel applied to a left speaker and on a second channel applied to a right speaker, the changing step further comprising:

selectively delaying when sound signal is generated in one of the left and right speakers relative to when the sound signal is generated in the other of the left and right speakers.

20. The method in claim 17, wherein when the object is at 0°

relative to the first display object, the delay of the sound signal generated in the left speaker is zero, the delay of the sound signal generated in the right speaker is a maximum, the volume applied to the left speaker is a maximum, and the volume applied to the right speaker is a minimum value.

21. The method in claim 17, wherein when the object is at 90°

relative to the first display object, the delay of the sound signal generated in the left speaker is equal to the delay of the sound signal generated in the right speaker and the volume of the sound signals applied to the left and right speakers is the same.

22. The method in claim 17, wherein when the object is at 180°

relative to the point of reference, the delay of the sound signal generated in the left speaker is a maximum, the delay of the sound signal generated in the right speaker is zero, the volume applied to the left speaker is a minimum and the volume applied to the right speaker is a maximum.

23. The method in claim 17, wherein when as the second display object moves from left of the first display object to right of the first display object, the delay of the sound signal applied to the left speaker increases and the delay the sound signal applied to the right speaker decreases.

24. The method in claim 23, wherein as the second display object moves from left of the first display object to right of the first display object, the volume of the sound signal applied to the left speaker decreases and the volume of the sound signal applied to the right speaker increases.

25. The method in claim 24, wherein the applied volume varies from zero to a maximum volume.

26. The method in claim 25, wherein the increase and decrease occur gradually between zero and the maximum volume.

27. The method in claim 24, wherein the applied volume varies from one half of a maximum volume to the maximum volume.

28. The method in claim 27, wherein the increase and decrease occur gradually between one half the maximum volume and the maximum volume.

29. The method in claim 24, wherein when the second display object is in front of the first display object, the volume of the sound signals applied to the left and right speakers is the same.

30. An imaging system generating a sound signal synchronized with movement of a first moving object displayed in a three-dimensional (3D) image, comprising:

a display displaying the 3D image and the first object, the first object being viewed from a point of reference of a second object displayed in the displayed 3D image, and

a data processor determining a position of the first display object in the displayed 3D image relative to the point of reference of the second display object and generating a sound signal associated with the first display object that changes based on relative movement between the point of reference of the second display object and the position of the first display object.

31. The system in claim 30, wherein the relative movement includes a distance between the second display object and the point of reference of the first display object, the data processor selectively modifying the volume of the sound signals applied to the left and right speakers based on the distance.

32. The system in claim 30, further comprising:

a left speaker and a right speaker to which the data processor applies the sound signal, and

a memory for storing the sound signal, the data processor retrieving the sound signal from the memory and selectively applying the retrieved sound signal to the right and left speakers.

33. The system in claim 32, wherein the memory includes:

a first memory area storing the sound signal, and

a second memory area storing the sound signal,

wherein the data processor reads out the sound signal from the first memory for application to one of the left and right speakers and reads out the sound signal from the second memory for application to the other of the left and right speakers after a delay.

34. The system in claim 33, wherein the delay varies between zero to a maximum value depending on a current position of the second display object relative to the point of reference of the first display object, and wherein the data processor selectively delays when the sound signal is generated in one of the left and right speakers relative to when the sound signal is generated in the other of the left and right speakers.

35. The system in claim 33, wherein the first and second memory areas are first-in-first-out memories, and the second memory area has a fraction of the capacity of the first memory area.

36. The system in claim 33, wherein the first and second memory areas are first-in-first-out memories, and the second memory area has more capacity than the first memory area.

37. The system in claim 33, wherein the first and second memory areas are first-in-first-out memories, and an initial write location in the second memory area varies depending on the delay.