Augmented reality system

• US 8,660,679 B2
• Filed: 12/02/2010
• Issued: 02/25/2014
• Est. Priority Date: 12/02/2010
• Status: Expired due to Fees

• Current Assignee: Empire Technology Development LLC (Allied Inventors Management, LLC)
• Original Assignee: Empire Technology Development LLC (Allied Inventors Management, LLC)
• Inventors: Kurabayashi, Shuichi, Yoshida, Naofumi, Takano, Kosuke
• Primary Examiner(s): Flanders, Andrew C
• Assistant Examiner(s): SIEGEL, DAVID F
• Application Number:
  US13/139,819
• Publication Number:
  US 20120143361A1
• Time in Patent Office: 1,181 Days
• Field of Search: 715/201, 348/25, 348/116, 348/142, 348/462, 348/586, 348/600
• US Class Current: 700/94
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G10H 2220/161 with 2D or x/y surface coordinates sensing graphical user interface or touchscreen input G10H2220/091; microtonal keyboard G10H2210/401
• CPC Class Codes: H04R 5/04 Circuit arrangements, e.g. for selective connection of amplifier inputs/outputs to loudspeakers, for loudspeaker detection, or for adaptation of settings to personal preferences or hearing impairments combinations of amplifiers H03F3/68; stereophonic systems H04S

Abstract

Technologies are generally described for a system for processing an augmented reality image. In some examples, an augmented reality system may include an augmented reality engine configured to provide an augmented reality image by integrating a real-world image and a virtual object, and a message processing unit configured to receive a message related to the virtual object and to translate spatial attributes of the virtual object into audio attributes of a sound file.

Claims

1. An augmented reality system comprising:

a memory comprising instructions; and

an augmented reality engine coupled to the memory, the augmented reality engine executable by a processor and configured to provide an augmented reality image by integrating a real-world image and a virtual object; and

a message processing unit coupled to the memory, the message processing unit executable by the processor and configured to;

receive an x-axis spatial component, a y-axis spatial component, and a z-axis spatial component for the virtual object;

determine a tone audio component based on the x-axis spatial component;

determine a pitch audio component based on the y-axis spatial component;

determine a volume audio component based on the z-axis spatial component; and

compose a sound file associated with the virtual object, the sound file comprising the tone audio component, the pitch audio component, and the volume audio component.

2. The augmented reality system of claim 1, further comprising a sound player executable by the processor, the sound player configured to play the sound file.

3. The augmented reality system of claim 1, further comprising a user recognition unit executable by the processor and configured to detect a user input and, in response, transmit a message comprising the x-axis spatial component, the y-axis spatial component, and the z-axis spatial component for the virtual object.

4. The augmented reality system of claim 3, wherein the user input is a selection of the virtual object.

5. The augmented reality system of claim 1, wherein the x-axis spatial component, the y-axis spatial component, and the z-axis spatial component are received in a message associated with the virtual object.

6. The augmented reality system of claim 5, wherein the message comprises an active message automatically generated by the virtual object.

7. The augmented reality system of claim 5, wherein the message comprises a passive message generated in response to a user selection of the virtual object.

8. The augmented reality system of claim 1, further comprising:

a first sound file repository configured to store a first sound mapping table; and

a second sound file repository configured to store a second sound mapping table,

wherein the x-axis spatial component, the y-axis spatial component, and the z-axis spatial component are received in a message associated with the virtual object,

wherein the message comprises an active message automatically generated by the virtual object and a passive message generated in response to a user selection of the virtual object,

wherein the sound file is associated with the active message,

wherein the message processing unit is further configured to use the first sound mapping table to determine the tone audio component based on the x-axis spatial component, the pitch audio component based on the y-axis spatial component, and the volume audio component based on the z-axis spatial component, and

wherein the message processing unit is further configured to;

use the second sound mapping table to determine a second tone audio component based on the x-axis spatial component, a second pitch audio component based on the y-axis spatial component, and a second volume audio component based on the z-axis spatial component, and

compose a second sound file associated with the passive message, the second sound file comprising the second tone audio component, the second pitch audio component, and the second volume audio component.

9. The augmented reality system of claim 1, further comprising:

a sound file repository configured to store a sound mapping table,

wherein the message processing unit is further configured to use the sound mapping table to determine the tone audio component based on the x-axis spatial component, the pitch audio component based on the y-axis spatial component, and the volume audio component based on the z-axis spatial component.

10. The augmented reality system of claim 9, wherein the sound mapping table comprises a functional formula showing a relationship between at least one of the x-axis spatial component, the y-axis spatial component, and the z-axis spatial component and at least one of the tone audio component, the pitch audio component, and the volume audio component.

11. The augmented reality system of claim 1, further comprising:

a depth recognition unit executable by the processor and configured to recognize spatial attributes of the real-world image; and

a distance calculation unit executable by the processor and configured to calculate a distance between a user and the virtual object based on the spatial attributes of the real-world image.

12. The augmented reality system of claim 1, further comprising:

a display unit configured to display the augmented reality image.

13. The augmented reality system of claim 1, further comprising:

a user recognition unit executable by the processor and configured to recognize a user'"'"'s behavior captured by an image capture unit.

14. A method performed under the control of an augmented reality system, the method comprising:

generating, by a computer processor, an augmented reality image using an augmented reality engine by integrating a real-world image and a virtual object;

receiving, by the computer processor, an x-axis spatial component, a y-axis spatial component, and a z-axis spatial component for the virtual object;

determining, by the computer processor, a tone audio component based on the x-axis spatial component;

determining, by the computer processor, a pitch audio component based on the y-axis spatial component;

determining, by the computer processor, a volume audio component based on the z-axis spatial component; and

composing, by the computer processor, a sound file associated with the virtual object, the sound file comprising the tone audio component, the pitch audio component, and the volume audio component.

15. The method of claim 14, further comprising playing the sound file.

16. The method of claim 14, wherein the x-axis spatial component, the y-axis spatial component, and the z-axis spatial component are received in a message associated with the virtual object.

17. The method of claim 16, wherein the message comprises at least one of an active message automatically generated by the virtual object or a passive message generated in response to a user selection of the virtual object.

18. The method of claim 14, wherein a sound mapping table is used to determine the tone audio component based on the x-axis spatial component, the pitch audio component based on the y-axis spatial component, and the volume audio component based on the z-axis spatial component.

19. A non-transitory computer-readable storage medium whose contents cause a processor to:

generate, using an augmented reality engine, an augmented reality image by integrating a real-world image and a virtual object;

receive an x-axis spatial component, a y-axis spatial component, and a z-axis spatial component for the virtual object;

determine a tone audio component based on the x-axis spatial component;

determine a pitch audio component based on the y-axis spatial component;

determine a volume audio component based on the z-axis spatial component; and

compose a sound file associated with the virtual object, the sound file comprising the tone audio component, the pitch audio component, and the volume audio component.

20. The non-transitory computer-readable storage medium of claim 19, wherein the contents further cause the processor to play the sound file.

21. The non-transitory computer-readable storage medium of claim 19, wherein the x-axis spatial component, the y-axis spatial component, and the z-axis spatial component are received in a message associated with the virtual object.

22. The non-transitory computer-readable storage medium of claim 21, wherein the message comprises at least one of an active message automatically generated by the virtual object or a passive message generated in response to a user selection of the virtual object.

23. The non-transitory computer-readable storage medium of claim 19, wherein the contents further cause the processor to use a sound mapping table to determine the tone audio component based on the x-axis spatial component, the pitch audio component based on the y-axis spatial component, and the volume audio component based on the z-axis spatial component.