Augmented reality with sound and geometric analysis

• US 9,563,265 B2
• Filed: 08/15/2012
• Issued: 02/07/2017
• Est. Priority Date: 01/12/2012
• Status: Active Grant

• Current Assignee: Qualcomm, Inc.
• Original Assignee: Qualcomm, Inc.
• Inventors: You, Kisun, Kim, Taesu, Hwang, Kyuwoong, Jin, Minho, Cho, Hyun-Mook, Lee, Te-Won
• Primary Examiner(s): Nguyen, Duc
• Assistant Examiner(s): PATEL, YOGESHKUMAR G
• Application Number:
  US13/585,927
• Publication Number:
  US 20130182858A1
• Time in Patent Office: 1,637 Days
• Field of Search: 345/633, 345/419, 345/158, 381/58, 348/207.1
• US Class Current: 1/1
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00

Abstract

A method for responding in an augmented reality (AR) application of a mobile device to an external sound is disclosed. The mobile device detects a target. A virtual object is initiated in the AR application. Further, the external sound is received, by at least one sound sensor of the mobile device, from a sound source. Geometric information between the sound source and the target is determined, and at least one response for the virtual object to perform in the AR application is generated based on the geometric information.

Claims

1. A method for responding in an augmented reality (AR) application of a mobile device to an external sound, the method comprising:

detecting a target via an image sensor of the mobile device;

initiating a virtual object in the AR application and positioning the virtual object at a position corresponding to a location of the target based on detecting that the target satisfies an AR application initialization criterion;

receiving, by at least one sound sensor of the mobile device, the external sound from a sound source;

determining, at the mobile device, first geometric information that includes a first direction between the sound source and the target, the first direction different than a second direction between the target and the mobile device; and

causing the virtual object to perform, in the AR application, at least one response to the external sound based on the first geometric information.

2. The method of claim 1, further comprising:

determining the second direction;

determining a third direction between the mobile device and the sound source; and

calculating the first direction based on the second direction and the third direction.

3. The method of claim 2, wherein the first direction is calculated based further on:

a distance between the mobile device and the target; and

a distance between the mobile device and the sound source.

4. The method of claim 2, wherein the third direction is determined based on a camera pose estimation.

5. The method of claim 1, wherein causing the virtual object to perform the at least one response includes:

extracting at least one sound feature from the received external sound;

identifying the extracted sound feature as corresponding to at least one sound class; and

identifying the at least one response based on the at least one sound class.

6. The method of claim 1, further comprising, after receiving the external sound:

determining whether the external sound is received from two or more sound sources;

if the external sound is received from the two or more sound sources, separating the received external sound into a plurality of sounds;

identifying additional sound sources associated with the plurality of separated sounds; and

determining fourth geometric information between the identified sound sources and the target.

7. The method of claim 6, further comprising determining an order of the at least one response for the virtual object, wherein the virtual object is caused to perform the at least one response based on the determined order.

8. The method of claim 1, further comprising:

displaying the virtual object but not the target at the mobile device based on detecting the target.

9. The method of claim 1, wherein determining the first geometric information includes determining one or more of:

a distance between the sound source and the target, a location of the mobile device, a location of the target, or a location of the sound source.

10. The method of claim 1, further comprising, after receiving the external sound:

determining whether the external sound is received from two or more sound sources;

if the external sound is received from the two or more sound sources, separating the received external sound into a plurality of sounds;

determining second geometric information between each of two or more sound sources and the target;

identifying each of the separated sounds as corresponding to at least one sound class; and

identifying the at least one response based on the at least one sound class.

33. The method of claim 1, further comprising determining the first direction based on the external sound using a beam forming technique.

34. The method of claim 1, wherein causing the virtual object to perform the at least one response includes causing the virtual object to turn toward the sound source, smile, or ignore the external sound.

35. The method of claim 7, wherein the order is determined based on a distance between the target and the sound source.

37. The method of claim 1, wherein initiating the virtual object is further based on detecting that the target satisfies an AR application initialization criterion.

11. A device for responding to an external sound in an augmented reality (AR) application, the device comprising:

an image sensor configured to receive at least one image including a target;

at least one sound sensor configured to receive the external sound from a sound source; and

a control unit including;

a target object detector configured to detect the target in the at least one image;

an application initiating unit configured to initiate a virtual object in the AR application and to position the virtual object at a position corresponding to a location of the target based on detecting the target;

a geometric information calculation unit configured to determine first geometric information that includes a first direction between the sound source and the target, the first direction different than a second direction between the target and the device; and

an AR response generator configured to cause the virtual object to perform, in the AR application, an action that comprises at least one response to the external sound based on the first geometric information.

12. The device of claim 11, wherein the geometric information calculation unit is further configured to:

determine the second direction;

determine a third direction between the device and the sound source; and

calculate the first direction based on the second direction and the third direction.

13. The device of claim 12, wherein the first direction is calculated based further on:

a distance between the device and the target; and

a distance between the device and the sound source.

14. The device of claim 12, wherein the third direction is determined based on a camera pose estimation.

15. The device of claim 11, wherein the AR response generator is further configured to:

extract at least one sound feature from the received external sound;

identify the extracted sound feature as corresponding to at least one sound class; and

identify the at least one response based on the at least one sound class.

16. The device of claim 11, further comprising:

a sound analyzer configured to determine whether the external sound is received from two or more sound sources, separate the received external sound into a plurality of sounds if the external sound is received from the two or more sound sources, identify additional sound sources associated with the plurality of separated sounds, and determine second geometric information between the identified sound sources and the target.

17. The device of claim 16, wherein the AR response generator is further configured to determine an order of actions, wherein the virtual object is caused to perform the at least one action based on the determined order.

18. The device of claim 11, wherein:

the AR application is configured to cause a display of the virtual object but not the target based on the target object detector detecting the target.

19. The device of claim 11, wherein the geometric information calculation unit is further configured to determine one or more of:

a distance between the sound source and the target, a location of the device, a location of the target, or a location of the sound source.

20. The device of claim 11, further comprising:

a sound analyzer configured to determine whether the external sound is received from two or more sound sources, separate the received external sound into a plurality of sounds if the external sound is received from the two or more sound sources, determine second geometric information between each of the two or more sound sources and the target, identify each of the separated sounds as corresponding to at least one sound class, and identify the at least one response for the virtual object based on the at least one sound class and the second geometric information.

36. The device of claim 16, wherein an order of the at least one response is determined based on distances between the target and the two or more sound sources, determined based on user input, or randomly selected.

21. A non-transitory computer-readable storage medium storing instructions that are executable by a processor to perform operations comprising:

detecting a target;

initiating a virtual object in an augmented reality (AR) application and positioning the virtual object at a position corresponding to a location of the target based on detecting the target;

receiving, by at least one sound sensor of a mobile device, an external sound from a sound source;

determining first geometric information that includes a first direction between the sound source and the target, the first direction different than a second direction between the target and the mobile device; and

causing the virtual object to perform, in the AR application, at least one action comprising a response to the external sound based on the first geometric information.

22. The non-transitory computer-readable storage medium of claim 21, wherein the operations further include:

determining the second direction;

determining a third direction between the mobile device and the sound source; and

calculating the first direction based on the second direction and the third direction.

23. The non-transitory computer-readable storage medium of claim 22, wherein the operations further comprise, after receiving the external sound:

determining whether the external sound is received from two or more sound sources;

if the external sound is received from the two or more sound sources, separating the received external sound into a plurality of sounds;

identifying additional sound sources associated with the plurality of separated sounds; and

determining fourth geometric information between the identified additional sound sources and the target.

24. The non-transitory computer-readable storage medium of claim 23, wherein the operations further comprise determining an order of actions;

wherein the virtual object is caused to perform the at least one action based on the determined order.

25. A device for responding to an external sound in an augmented reality (AR) application, the device comprising:

means for detecting a target;

means for initiating a virtual object in the AR application and positioning the virtual object at a position corresponding to a location of the target based on detecting the target;

means for receiving the external sound from a sound source;

means for determining first geometric information that includes a first direction between the sound source and the target, the first direction different than a second direction between the target and the device; and

means for causing the virtual object to perform, in the AR application, at least one response to the external sound based on the first geometric information.

26. The device of claim 25, wherein the means for determining the first geometric information is configured to:

determine the second direction;

determine a third direction between the device and the sound source; and

calculate the first direction based on the second direction and the third direction.

27. The device of claim 25, wherein the means for determining the first geometric information is configured to:

determine a distance between the device and the target; and

determine a distance between the device and the sound source.

28. The device of claim 25, wherein the means for determining the first geometric information is configured to determine the second direction based on a camera pose estimation.

29. The device of claim 25, wherein the means for causing the virtual object to perform, in the AR application, the at least one response is configured to:

extract at least one sound feature from the received external sound;

identify the extracted sound feature as corresponding to at least one sound class; and

identify the at least one response based on the at least one sound class.

30. The device of claim 25, wherein the means for determining the first geometric information is configured to:

determine whether the external sound is received from two or more sound sources;

if the external sound is received from the two or more sound sources, separate the received external sound into a plurality of sounds;

identify sound sources associated with the plurality of separated sounds; and

determine second geometric information between the identified sound sources and the target.

31. The device of claim 30, wherein the means for causing the virtual object to perform, in the AR application, the at least one response is configured to determine an order of the at least one response for the virtual object, wherein the virtual object is caused to perform the at least one response based on the determined order.

32. The device of claim 25, wherein:

the means for determining the first geometric information is configured to determine whether the external sound is received from two or more sound sources and, if the external sound is received from the two or more sound sources, separate the received external sound into a plurality of sounds, and determine second geometric information between each of two or more sound sources and the target; and

the means for causing the virtual object to perform is configured to identify each of the separated sounds as corresponding to at least one sound class, and to identify the at least one response based on the at least one sound class and the second geometric information.