Systems and methods for simulating sounds of a virtual object using procedural audio

• US 10,359,858 B2
• Filed: 09/07/2016
• Issued: 07/23/2019
• Est. Priority Date: 09/07/2016
• Status: Active Grant

• Current Assignee: Disney Enterprises Incorporated (The Walt Disney Company)
• Original Assignee: Disney Enterprises Incorporated (The Walt Disney Company)
• Inventors: Barton, Mark R., Kaye, Cameron J.
• Primary Examiner(s): Zhang, Leshui
• Application Number:
  US15/259,019
• Publication Number:
  US 20180068487A1
• Time in Patent Office: 1,049 Days
• Field of Search: 381 1, 381310, 381306, 381303, 381304, 381305, 381 89, 381332, 381333, 381309, 381 17, 381 18, 381 19, 381 26, 381 58, 381 91, 381 92, 381122, 381182, 381326, 381370, 381374, 381 74, 381 63, 381 61, 381 77, 381 79, 700 94, 345 8, 345633, 345173, 704233, 704235, 704270, 704272, 704273, 704275, 715716, 715727, 715848, 463 49, 463 56, 463 476, 463 38, 3405731, 3404072, 3405391, 340 758, 434 4, 434 10, 434 81, 434 90, 434 91, 434 92, 434102, 434230, 472 64, 472 65, 446175, 446188, 446192, 446203, 446207, 446213, 446215, 446216, 446242, 446251, 446254, 446258, 446266, 446270, 446297, 446397, 446398, 446401, 446402, 446404, 446405, 446406
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06F 3/03545 Pens or stylus
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: H04R 1/028 associated with devices performing functions other than acoustics, e.g. electric candles
• CPC Class Codes: H04R 3/04 for correcting frequency response
• CPC Class Codes: H04R 5/02 Spatial or constructional arrangements of loudspeakers
• CPC Class Codes: H04S 2400/11 Positioning of individual sound objects, e.g. moving airplane, within a sound field H04S2420/13 takes precedence
• CPC Class Codes: H04S 7/307 Frequency adjustment, e.g. tone control H04S7/301 takes precedence

Abstract

This disclosure relates to simulating sounds of a virtual object using procedural audio. A position and/or an orientation for the virtual object may be determined. Changes in the position and/or the orientation for the virtual object may be determined and used to determine the procedural audio. The procedural audio may include a first modulation of a first sound, a second modulation of a second sound, and/or other sounds. A speaker may be operated to provide the procedural audio to simulate the sounds of the virtual object.

Claims

1. A system for simulating sounds of a virtual object, the system comprising:

a storage medium storing a repository of sounds, the repository of sounds including a first sound and a second sound;

a handle;

a sensor carried by the handle, the sensor configured to generate motion and/or orientation output signals conveying motion and/or orientation information of the handle, the motion and/or orientation information of the handle specifying one or more of a yaw of the handle, a pitch of the handle, and/or a roll of the handle;

a speaker;

a display configured to display an overlay image, the overlay image depicting the virtual object, the overlay image being displayed such that the virtual object has a visual appearance of being attached to the handle during changes in the yaw of the handle, the pitch of the handle, and/or the roll of the handle; and

one or more physical processors configured by machine readable instructions to;

determine a position and/or an orientation for the virtual object based on the motion and/or orientation output signals, wherein the position and/or the orientation for the virtual object includes a yaw of the virtual object, a pitch of the virtual object, and/or a roll of the virtual object;

determine changes in the position and/or the orientation for the virtual object based on the motion and/or orientation output signals, wherein the changes in the position and/or the orientation for the virtual object include one or more of a change in the yaw of the virtual object, a change in the pitch of the virtual object, and/or a change in the roll of the virtual object;

determine audio output to simulate the sounds of the virtual object based on the changes in the position and/or the orientation for the virtual object, wherein the audio output to simulate the sounds of the virtual object includes a first modulation of the first sound and/or a second modulation of the second sound; and

effectuate operation of the speaker to provide the audio output to simulate the sounds of the virtual object.

2. The system of claim 1, wherein the first sound includes a base tone and the second sound includes an overtone.

3. The system of claim 1, wherein the first modulation and/or the second modulation include changes in one or more of a pitch, an amplitude, and/or a timbre.

4. The system of claim 1, wherein the first modulation and the second modulation include different changes in one or more of a pitch, an amplitude, and/or a timbre.

5. The system of claim 1, wherein the first modulation and/or the second modulation includes a low-frequency oscillation multiplier.

6. The system of claim 1, wherein the changes in the position and/or the orientation for the virtual object include changes in the position and/or the orientation of a tip of the virtual object.

7. The system of claim 1, wherein determining the changes in the position and/or the orientation for the virtual object includes:

determining a speed of the virtual object; and

using a two-stage low-pass filter to determine a smoothed speed of the virtual object.

8. The system of claim 1, wherein the one or more physical processors are further configured to select the virtual object, wherein the audio output is determined further based on the selected virtual object.

9. The system of claim 1, further comprising:

an image sensor configured to generate visual output signals conveying visual information within a field of view of the image sensor;

wherein the one or more physical processors are further configured by machine readable instructions to;

determine the position and/or the orientation for the virtual object further based on the visual output signals;

determine the overlay image depicting the virtual object, wherein the virtual object is depicted within the overlay image according to the position and/or the orientation for the virtual object; and

effectuate displaying of the overlay image on the display so that the virtual object appears to augment an appearance of the handle.

10. A method for simulating sounds of a virtual object, the method comprising:

generating motion and/or orientation output signals conveying motion and/or orientation information of a handle, the motion and/or orientation information of the handle specifying one or more of a yaw of the handle, a pitch of the handle, and/or a roll of the handle;

displaying an overlay image, the overlay image depicting the virtual object, the overlay image being displayed such that the virtual object has a visual appearance of being attached to the handle during changes in the yaw of the handle, the pitch of the handle, and/or the roll of the handle;

determining a position and/or an orientation for the virtual object based on the motion and/or orientation output signals, wherein the position and/or the orientation for the virtual object includes a yaw of the virtual object, a pitch of the virtual object, and/or a roll of the virtual object;

determining changes in the position and/or the orientation for the virtual object based on the motion and/or orientation output signals, wherein the changes in the position and/or the orientation for the virtual object include one or more of a change in the yaw of the virtual object, the pitch of the virtual object, and/or a change in the roll of the virtual object;

determining audio output to simulate the sounds of the virtual object based on the changes in the position and/or the orientation for the virtual object, wherein the audio output includes a first modulation of a first sound and/or a second modulation of a second sound; and

effectuating operation of a speaker to provide the audio output to simulate the sounds of the virtual object.

11. The method of claim 10, wherein the first sound includes a base tone and the second sound includes an overtone.

12. The method of claim 10, wherein the first modulation and/or the second modulation include changes in one or more of a pitch, an amplitude, and/or a timbre.

13. The method of claim 10, wherein the first modulation and the second modulation include different changes in one or more of a pitch, an amplitude, and/or a timbre.

14. The method of claim 10, wherein the first modulation and/or the second modulation includes a low-frequency oscillation multiplier.

15. The method of claim 10, wherein the changes in the position and/or the orientation for the virtual object include changes in the position and/or the orientation of a tip of the virtual object.

16. The method of claim 10, wherein determining the changes in the position and/or the orientation for the virtual object includes:

determining a speed of the virtual object; and

using a two-stage low-pass filter to determine a smoothed speed of the virtual object.

17. The method of claim 10, further comprising selecting the virtual object, wherein the audio output is determined further based on the selected virtual object.

18. The method of claim 10, further comprising:

generating visual output signals conveying visual information within a field of view of an image sensor;

determining the position and/or the orientation for the virtual object further based on the visual output signals;

determining the overlay image depicting the virtual object, wherein the virtual object is within the overlay image according to the position and/or the orientation for the virtual object; and

effectuating displaying of the overlay image on the display so that the virtual object appears to augment an appearance of the handle.

19. A system for simulating sounds of a virtual object, the system comprising:

a storage medium storing a repository of sounds, the repository of sounds including a first sound and a second sound;

an image sensor configured to generate visual output signals conveying visual information within a field of view of the image sensor;

a handle;

a display configured to display an overlay image, the overlay image depicting the virtual object, the overlay image being displayed such that the virtual object has a visual appearance of being attached to the handle during changes in the yaw of the handle, the pitch of the handle, and/or the roll of the handle;

a sensor carried by the handle, the sensor configured to generate motion and/or orientation output signals conveying motion and/or orientation information of the handle, the motion and/or orientation information of the handle specifying one or more of a yaw of the handle, a pitch of the handle, and/or a roll of the handle;

a speaker; and

one or more physical processors configured by machine readable instructions to;

select the virtual object;

change the virtual object;

determine a position and/or an orientation for the virtual object based on the visual output signals and/or the motion and/or orientation output signals, wherein the position and/or the orientation for the virtual object includes a yaw of the virtual object, a pitch of the virtual object, and/or a roll of the virtual object;

determine the overlay image comprising the virtual object, wherein the virtual object is placed within the overlay image according to the position and/or the orientation for the virtual object;

effectuate displaying of the overlay image on the display so that the virtual object appears to augment an appearance of the handle;

determine changes in the position and/or the orientation for the virtual object based on the motion and/or orientation output signals, wherein the changes in the position and/or the orientation for the virtual object include one or more of a change in the yaw of the virtual object, a change in the pitch of the virtual object, and/or a change in the roll of the virtual object;

determine audio output to simulate the sounds of the virtual object based on the selected virtual object or the changed virtual object, and further based on the changes in the position and/or the orientation for the virtual object, wherein the audio output includes a first modulation of the first sound and/or a second modulation of the second sound; and

effectuate operation of the speaker to provide the audio output to simulate the sounds of the virtual object.

20. The system of claim 19, wherein:

the first sound includes a base tone and the second sound includes an overtone; and

the first modulation and/or the second modulation include changes in one or more of a pitch, an amplitude, and/or a timbre.