Apparatus and method for audio reactive UI information and display

• US 10,048,933 B2
• Filed: 11/21/2012
• Issued: 08/14/2018
• Est. Priority Date: 11/30/2011
• Status: Active Grant

• Current Assignee: Nokia Technologies Oy (Nokia Corporation)
• Original Assignee: Nokia Technologies Oy (Nokia Corporation)
• Inventors: Reponen, Erika, Shenoy, Ravi, Tammi, Mikko, Vesa, Sampo
• Primary Examiner(s): Salomon, Phenuel
• Application Number:
  US14/361,255
• Publication Number:
  US 20140337741A1
• Time in Patent Office: 2,092 Days
• Field of Search: None
• CPC Class Codes: G01S 3/801 Details G01S3/82, G01S3/84, G01S3/86 take precedence
• CPC Class Codes: G06F 1/1694 the I/O peripheral being a single or a set of motion sensors for pointer control or gesture input obtained by sensing movements of the portable computer
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06F 3/04817 using icons graphical or visual programming using iconic symbols G06F8/34
• CPC Class Codes: G06F 3/0484 for the control of specific functions or operations, e.g. selecting or manipulating an object, an image or a displayed text element, setting a parameter value or selecting a range
• CPC Class Codes: G06F 3/0487 using specific features provided by the input device, e.g. functions controlled by the rotation of a mouse with dual sensing arrangements, or of the nature of the input device, e.g. tap gestures based on pressure sensed by a digitiser
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: H04W 4/33 for indoor environments, e.g. buildings

Abstract

A method includes determining, using signals captured from two or more microphones (1A) configured to detect an acoustic signal from one or more sound sources, one or more prominent sound sources based on the one or more sound sources (1C-1D). The method further includes determining one or more directions relative to a position of one or more of the two or more microphones for the one or more prominent sound sources (1B-1D). The method includes modifying one or more user interface elements displayed on a user interface of a display to provide an indication at least in part of the one or more directions, relative to position of at least one microphone, of the one or more prominent sound sources (1G).

Claims

1. An apparatus, comprising:

one or more processors; and

one or more memories including computer program code,

the one or more memories and the computer program code configured to, with the one or more processors, cause the apparatus to;

determine, using signals captured from two or more microphones configured to detect an acoustic signal from one or more prominent sound sources, one or more directions of the one or more prominent sound sources relative to a position of at least one of the two or more microphones, wherein the apparatus comprises the at least one of the two or more microphones, where locations of the two or more microphones are predetermined inside the apparatus;

present at least one displayed user interface element that is to receive a modification to provide directional information of the one or more prominent sound sources, wherein the at least one displayed user interface element is presented for at least one purpose independent of the one or more prominent sound sources; and

modify the at least one displayed user interface element dependent on the one or more determined directions while the two or more microphones are capturing the acoustic signal, wherein the at least one displayed user interface element is modified on a display of the apparatus to provide an indication at least in part of the one or more determined directions of the one or more prominent sound sources so that a relative position of the one or more prominent sound sources to the apparatus is indicated from the modified at least one displayed user interface element while the two or more microphones are capturing the acoustic signal.

2. The apparatus of claim 1, wherein the one or more memories and the computer program code are further configured to, with the one or more processors, cause the apparatus to modify the at least one displayed user interface element by rotating at least one of the at least one displayed user interface element on a user interface of the apparatus, wherein the rotated one of the at least one displayed user interface element indicates the one or more determined directions of the one or more prominent sound sources.

3. The apparatus of claim 2, wherein rotating further comprises rotating the at least one of the at least one displayed user interface element by rotating an axis of the at least one displayed user interface element approximately parallel to the one or more determined directions of the one or more prominent sound sources.

4. The apparatus of claim 1, wherein the one or more memories and the computer program code are further configured to, with the one or more processors, cause the apparatus to modify a selected one of the at least one displayed user interface element by moving the selected one of the at least one displayed user interface element on a user interface of the apparatus in a second direction away from the one or more determined directions of the one or more prominent sound sources.

5. The apparatus of claim 4, wherein the moving further comprises moving the selected at least one displayed user interface element on the user interface on a trajectory that is at least in part along the second direction.

6. The apparatus of claim 1, wherein the one or more memories and the computer program code are further configured to, with the one or more processors, cause the apparatus to modify a selected one of the at least one displayed user interface element, wherein the selected one of the at least one displayed interface element is configured to react to a sound of the one or more prominent sound sources.

7. The apparatus of claim 1, wherein the one or more memories and the computer program code are further configured to, with the one or more processors, cause the apparatus to modify a selected one of the at least one displayed user interface element by modifying one or more colors of the selected one of the at least one displayed interface element based on the one or more determined directions of the one or more prominent sound sources.

8. The apparatus of claim 1, wherein the at least one displayed user interface element comprises at least one of an icon on a user interface of the apparatus, text on the user interface, a background of the] user interface, a photograph on the user interface, content on the user interface, or a page of the user interface.

9. The apparatus of claim 1, wherein the at least one displayed user interface element comprises a page of a user interface on the display of the apparatus, and wherein the one or more memories and the computer program code are further configured to, with the one or more processors, cause the apparatus to modify a selected one of the at least one displayed user interface element by moving, responsive to the one or more determined directions of the one or more prominent sound sources, a page of the user interface off the display and moving a different page of the user interface onto the display.

10. The apparatus of claim 1, wherein the at least one displayed user interface element comprises an unlock icon of a user interface on the display of the apparatus, and wherein the one or more memories and the computer program code are further configured to, with the one or more processors, cause the apparatus to modify a selected one of the at least one displayed user interface element by moving, responsive to the one or more determined directions of the one or more prominent sound sources, the unlock icon from an initial position to an end position.

11. A method, comprising:

determining, using signals captured from two or more microphones configured to detect an acoustic signal from one or more prominent sound sources, one or more directions of the one or more prominent sound sources relative to a position of at least one of the two or more microphones, wherein an apparatus comprises the at least one of the two or more microphones, where locations of the two or more microphones are predetermined inside the apparatus;

presenting at least one displayed user interface element that is to receive a modification to provide directional information of one or more prominent sound sources, wherein the at least one displayed user interface element is presented for at least one purpose independent of the one or more prominent sound sources; and

modifying the at least one displayed user interface element dependent on the one or more determined directions while the two or more microphones are capturing the acoustic signal, wherein the at least one displayed user interface element is displayed on a display of the apparatus to provide an indication at least in part of the one or more determined directions of the one or more prominent sound sources so that a relative position of the one or more prominent sound sources to the apparatus is indicated from the modified at least one displayed user interface element while the two or more microphones are capturing the acoustic signal.

12. The method of claim 11, wherein modifying further comprises rotating one of the at least one displayed user interface element on a user interface of the apparatus, wherein the rotated one of the at least one displayed user interface element indicates one or more determined directions.

13. The method of claim 12, wherein the rotating further comprises rotating the one of the at least one displayed user interface element by rotating an axis of the at least one displayed user interface element approximately parallel to the one or more determined directions.

14. The method of claim 11, wherein the modifying further comprises modifying the at least one displayed user interface element by moving the at least one displayed user interface element on a user interface of the apparatus in a second direction away from the one or more determined directions.

15. The method of claim 14, wherein the moving further comprises moving the at least one displayed user interface element on a user interface of the apparatus on a trajectory that is at least in part along the second direction.

16. The method of claim 11, wherein the modifying further comprises modifying one or more colors of the at least one displayed user interface element based on the one or more determined directions.

17. The method of claim 11, wherein the at least one user interface element comprises a page of a user interface on the display of the apparatus, and wherein the modifying further comprises moving, responsive to the one or more determined directions, the page of the user interface off the display and moving a different page of the user interface onto the display.

18. The method of claim 11, wherein the at least one displayed user interface element comprises an unlock icon of a user interface on the display of the apparatus, and wherein modifying further comprises moving, responsive to the one or more determined directions, the unlock icon from an initial position to an end position.