Augmented reality enhanced audio

• US 8,170,222 B2
• Filed: 04/18/2008
• Issued: 05/01/2012
• Est. Priority Date: 04/18/2008
• Status: Active Grant

• Current Assignee: Sony Corporation (Sony Group Corp.)
• Original Assignee: Sony Mobile Communications AB (Sony Group Corp.)
• Inventors: Dunko, Gregory A.
• Primary Examiner(s): Williams, A O
• Application Number:
  US12/105,367
• Publication Number:
  US 20090262946A1
• Time in Patent Office: 1,474 Days
• Field of Search: 381/58, 381/74, 381/19, 381/91, 381/17, 381/310
• US Class Current: 381/58
• CPC Class Codes: G06F 3/165 Management of the audio stream, e.g. setting of volume, audio stream path
• CPC Class Codes: H04S 1/002 Non-adaptive circuits, e.g. manually adjustable or static, for enhancing the sound image or the spatial distribution control circuits for electronic adaptation of the sound field H04S7/30
• CPC Class Codes: H04S 7/302 Electronic adaptation of stereophonic sound system to listener position or orientation H04S7/301 takes precedence

Abstract

A device and method for providing augmented reality enhanced audio. The device is portable and may include a location processing module capable of receiving location information and determining a location of the device in an environment, an orientation processing module capable of determining an orientation of the device at the location, a processor capable of modifying audio content based on the location of the device in the environment and the orientation of the device at the location, and an audio output device capable of outputting the modified audio content. The orientation of the device includes an azimuth and an elevation of the device.

Claims

1. A method for augmented reality enhanced audio comprising:

obtaining audio content by a device from at least one of a source in the device or an external source located remotely from the device in an environment;

determining, by the device, a location of the device with respect to the external source;

moving the device through the environment to a different location;

modifying, by the device, a loudness of the audio content based on a location of the device with respect to the external source, wherein the loudness of the audio content increases as the device approaches the external source;

modifying, by the device, a left/right balance of the audio content based on an orientation of the device with respect to the external source, wherein the orientation of the device with respect to the external source includes at least one of an azimuth, an elevation, or a tilt of the device with respect to the external source; and

outputting the modified audio content by the device.

2. The method according to claim 1, further comprising

modifying the obtained audio content responsive to the moving the device to the different location and an orientation of the device at the different location;

and

outputting the modified audio content by the device.

3. The method according to claim 1, further comprising providing a central source that stores audio content associated with one or more external sources in the environment.

4. The method according to claim 1, further comprising obtaining audio content from at least one external source located one of within the environment or outside the environment.

5. The method according to claim 1, wherein the external source is a virtual sound source associated with a real object in the environment, and wherein a user of the device perceives that the modified audio content is being emitted from the real object.

6. The method according to claim 1, wherein the outputting further comprises:

simultaneously outputting modified audio content associated with at least two external sources.

7. The method according to claim 6, further comprising the loudness of the sound varying based on the location of the device in the environment and the orientation of the device at the location.

8. The method according to claim 1, further comprising receiving at the device the audio content, information on an external source of at least one sound in the audio content, and a location of the external source of the at least one sound within the environment.

9. The method according to claim 1, wherein a user of the device determines at least one of a sound in the audio content, an external source of a sound in the audio content or a location of the external source of a sound within the environment.

10. The method according to claim 1, further comprising scaling audio content associated with a first environment to a second environment when the device is located in the second environment.

11. The method according to claim 1, further comprising receiving the audio content at the device upon the device entering the environment.

12. The method according to claim 1, further comprising the device assigning a location in a new environment to a source of a sound in the audio content obtained by the device.

20. The method according to claim 10, wherein first dimensions associated with the first environment are different from second dimensions associated with the second environment.

13. A device for augmented reality enhanced audio comprising:

a receiving module for obtaining audio content, wherein the device obtains audio content from at least one of a source in the device or an external source located remotely from the device in an environment;

a location processing module, the location processing module being capable of receiving location information and determining a location of the device with respect to the external source in the environment;

an orientation processing module, the orientation processing module being capable of determining an orientation of the device at the location with respect to the external source;

a processor, the processor being capable of modifying a loudness of the audio content based on the location of the device with respect to the external source in the environment wherein the loudness of the audio content increases as the device approaches the external source, the processor additionally being capable of modifying a left/right balance of the audio content based on an orientation of the device at the location with respect to the external source, wherein the orientation of the device with respect to the external source includes at least one of an azimuth, an elevation, or a tilt of the device with respect to the external source; and

an audio output device, the audio output device being capable of outputting the modified audio content,

wherein the device moves through the environment comprising at least one external source.

14. The device according to claim 13, further comprising an input interface, the input interface being capable of receiving the audio content.

15. The device according to claim 13, further comprising a storage device, the storage device being capable of storing the audio content.

16. The device according to claim 13, wherein the device comprises one of a mobile phone, a computer, a personal digital assistant (PDA), an electronic game device, or an audio device.

17. The device according to claim 13, wherein the location information comprises at least one of GPS coordinates of the location of the device, a distance of the device from the external source, position information of the device, or triangulation information of the device.

18. The device according to claim 13, wherein the audio output device is capable of simultaneously outputting modified audio content associated with at least two external sources.

19. The device according to claim 13, wherein the audio output device comprises at least one of a headset, earplugs, or at least one speaker.