Visual audio processing apparatus

• US 10,635,383 B2
• Filed: 04/04/2013
• Issued: 04/28/2020
• Est. Priority Date: 04/04/2013
• Status: Active Grant

• Current Assignee: Nokia Technologies Oy (Nokia Corporation)
• Original Assignee: Nokia Technologies Oy (Nokia Corporation)
• Inventors: Makinen, Toni Henrik, Tammi, Mikko, Vilermo, Miikka Tapani
• Primary Examiner(s): Chin, Vivian C
• Assistant Examiner(s): Hamid, Ammar T
• Application Number:
  US14/781,182
• Publication Number:
  US 20160299738A1
• Time in Patent Office: 2,581 Days
• Field of Search: 381 98, 381119, 381 97
• CPC Class Codes: G06F 1/1684 Constructional details or arrangements related to integrated I/O peripherals not covered by groups G06F1/1635 - G06F1/1675
• CPC Class Codes: G06F 1/1686 the I/O peripheral being an integrated camera
• CPC Class Codes: G06F 3/0416 Control or interface arrangements specially adapted for digitisers
• CPC Class Codes: G06F 3/04847 Interaction techniques to control parameter settings, e.g. interaction with sliders or dials
• CPC Class Codes: G06F 3/0488 using a touch-screen or digitiser, e.g. input of commands through traced gestures
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text
• CPC Class Codes: G06F 3/162 Interface to dedicated audio devices, e.g. audio drivers, interface to CODECs
• CPC Class Codes: G06F 3/165 Management of the audio stream, e.g. setting of volume, audio stream path
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: H04N 23/62 Control of parameters via user interfaces
• CPC Class Codes: H04R 2227/003 Digital PA systems using, e.g. LAN or internet
• CPC Class Codes: H04R 2420/09 Applications of special connectors, e.g. USB, XLR, in loudspeakers, microphones or headphones
• CPC Class Codes: H04R 2499/11 Transducers incorporated or for use in hand-held devices, e.g. mobile phones, PDA's, camera's
• CPC Class Codes: H04S 2400/01 Multi-channel, i.e. more than two input channels, sound reproduction with two speakers wherein the multi-channel information is substantially preserved
• CPC Class Codes: H04S 2400/15 Aspects of sound capture and related signal processing for recording or reproduction
• CPC Class Codes: H04S 2420/01 Enhancing the perception of the sound image or of the spatial distribution using head related transfer functions [HRTF's] or equivalents thereof, e.g. interaural time difference [ITD] or interaural level difference [ILD]
• CPC Class Codes: H04S 2420/07 Synergistic effects of band splitting and sub-band processing
• CPC Class Codes: H04S 3/002 Non-adaptive circuits, e.g. manually adjustable or static, for enhancing the sound image or the spatial distribution control circuits for electronic adaptation of the sound field H04S7/30
• CPC Class Codes: H04S 5/00 Pseudo-stereo systems, e.g. in which additional channel signals are derived from monophonic signals by means of phase shifting, time delay or reverberation
• CPC Class Codes: H04S 7/30 Control circuits for electronic adaptation of the sound field

Abstract

An apparatus comprising: an image element determiner configured to determine at least one visual image element; a spatial audio analyser configured to determine, using at least two audio signals, at least one audio source with a location associated with the at least one visual image element; a user interface controller configured to provide at least one control input associated with the at least one visual element, and an audio processor configured to process the at least one audio source to change at least one audio characteristic of the at least one audio source based on the at least one control input.

Claims

1. An apparatus comprising at least one processor and at least one memory including computer code for one or more programs, the at least one memory and the computer code configured to with the at least one processor cause the apparatus to:

determine at least one visual image element;

receive at least two audio signals from at least two microphones, wherein each microphone of the at least two microphones is separated from each other;

determine, using the at least two audio signals, a location of at least one audio source relative to the at least two microphones, wherein the at least one audio source is determined at least in part as environmental sound;

associate the at least one visual image element with the location of the at least one audio source relative to the at least two microphones;

provide at least one control input associated with the at least one visual element, the at least one control input comprising at least one control for at least one focus region direction and width;

process the at least one audio source to change at least one audio characteristic of the at least one audio source according to the determined location of the at least one audio source based on the provided and width of the at least one focus region; and

amplify or attenuate the at least one audio source for playback dependent on the location of the at least one audio source based on the at least one control for the at least one focus region direction and width.

2. The apparatus as claimed in claim 1, further caused to:

generate the at least one visual image element comprising at least one user interface overlay element and at least one captured image frame; and

display the at least one visual image element on a touch interface display configured to provide the at least one control input.

3. The apparatus as claimed in claim 2, wherein providing the at least one control input causes the apparatus to determine at least one user interface parameter associated with the at least one control input.

4. The apparatus as claimed in claim 3, wherein generating the at least one visual image element causes the apparatus to generate the at least one user interface overlay element based on the at least one user interface parameter.

5. The apparatus as claimed in claim 2, wherein the at least one user interface overlay comprises at least one of:

an overlay displaying the at least one audio source signal with the location;

a focus region overlay displaying a processing focus range and direction;

a zoom location overlay displaying a processing zoom or motion location;

a rotation overlay displaying a processing rotation;

a toggle activate spatial processing overlay configured to display a toggle state for generating at least one further at least one user interface overlay element;

a focus amplification overlay configured to display at least one amplification gain for audio sources inside the at least one focus region;

a focus attenuation overlay configured to display at least one attenuation dampening for audio sources outside of the at least one focus region;

a focus inverter overlay configured to display a toggle state for inverting the focus operation;

or

a focus amplification profile overlay configured to display an amplification profile for directions surrounding the apparatus.

6. The apparatus as claimed in claim 1, wherein the provided at least one control input associated with the at least one visual image element causes the apparatus to provide at least one of:

at least one control input generated by a motion of the apparatus displaying the at least one visual image element; and

at least one control input generated by an operation of camera inputs associated with the at least one visual image element.

7. The apparatus as claimed in claim 1, wherein the at least one audio source to change at least one audio characteristic of the at least one audio source based on the at least one control input causes the apparatus to:

generate at least one processing parameter associated with the at least one control input; and

process the at least one audio source based on the at least one processing parameter.

8. The apparatus as claimed in claim 7, wherein the generated at least one processing parameter associated with the at least one control input causes the apparatus to generate at least one of:

at least one focus region range;

at least one focus region direction;

at least one smoothing region direction;

at least one smoothing region range;

at least one focus region amplification gain;

at least one focus region out of region attenuation factor;

at least one focus region attenuation factor;

at least one focus region out of region amplification gain;

at least one virtual motion or zoom direction;

at least one virtual motion or zoom displacement;

or

at least one virtual motion or rotation angle of rotation.

9. The apparatus as claimed in claim 8, wherein the generated at least one processing parameter causes the apparatus to at least one of:

determine a zoom location based on the at least one virtual motion or zoom direction and at least one virtual motion or zoom displacement;

determine for the at least one audio source a new distance between the zoom location and the at least one audio source location;

determine for the at least one audio source a new direction between the zoom location and the at least one audio source location;

or

replace the at least one audio source location with a new audio source location based on the new distance and new direction values.

10. The apparatus as claimed in claim 8, wherein the generated at least one processing parameter causes the apparatus to:

determine a rotation orientation based on the at least one virtual motion or rotation angle;

determine for the at least one audio source a new direction between the rotation orientation and the at least one audio source location; and

replace the at least one audio source location with a new audio source location based on the new direction value.

11. The apparatus as claimed in claim 8, wherein the generated at least one processing parameter causes the apparatus to at least one of:

amplify the at least one audio source by the at least one focus region amplification gain where the at least one audio source location is within the at least one focus region direction and range;

attenuate the at least one audio source by the at least one focus region out of region attenuation factor where the at least one audio source location is outside the at least one focus region direction and range and outside the at least one smoothing region direction and range; and

apply an interpolation of the at least one focus region amplification gain and the at least one focus region out of region attenuation factor where the at least one audio source location is within the at least one smoothing region direction and range.

12. The apparatus as claimed in claim 8, wherein the generated at least one processing parameter causes the apparatus to at least one of:

attenuate the at least one audio source by the at least one focus region attenuation factor where the at least one audio source location is within the at least one focus region direction and range;

amplify the at least one audio source by the at least one focus region out of region amplification gain where the at least one audio source location is outside the at least one focus region direction and range and outside the at least one smoothing region direction and range; and

apply an interpolation of the at least one focus region attenuation factor and the at least one focus region out of region amplification gain where the at least one audio source location is within the at least one smoothing region direction and range.

13. The apparatus as claimed in claim 8, wherein the generated at least one processing parameter causes the apparatus to apply a time smoothing factor to at least one of:

the at least one focus region amplification gain;

at least one focus region out of region attenuation factor;

at least one focus region attenuation factor;

or at least one focus region out of region amplification gain.

14. The apparatus as claimed in claim 1, further caused to receive the at least two audio signals from at least one of:

the at least two microphones;

an apparatus memory;

or

at least one further apparatus.

15. The apparatus as claimed in claim 1, further caused to:

receive at least one captured image frame from which the at least one visual image element is determined, wherein the at least one captured image frame is received from at least one of;

at least one camera;

a memory; and

at least one further apparatus.

16. A method comprising:

determining at least one visual image element;

receiving at least two audio signals from at least two microphones, wherein each microphone of the at least two microphones is separated from each other;

determining, using the at least two audio signals, a location of at least one audio source relative the at least two microphones, wherein the at least one audio source is determined at least in part as environmental sound;

associating the at least one visual image element with the location of the at least one audio source relative to the at least two microphones;

providing at least one control input associated with the at least one visual element, the at least one control input comprising at least one control for at least one focus region direction and width;

processing the at least one audio source to change at least one audio characteristic of the at least one audio source according to the determined location of the at least one audio source based on the provided direction and width of the at least one focus region during playback; and

amplifying or attenuating the at least one audio source for playback dependent on the location of the at least one audio source based on the at least one control for the at least one focus region direction and width.

17. The method as claimed in claim 16, further comprising:

generating the at least one visual image element comprising at least one user interface overlay element and at least one captured image frame; and

displaying the at least one visual image element on a touch interface display configured to provide the at least one control input.

18. The method as claimed in claim 17, wherein the at least one user interface overlay comprises at least one of:

an overlay displaying the at least one audio source signal with a location;

a focus region overlay displaying a processing focus range and direction;

a zoom location overlay displaying a processing zoom or motion location;

a rotation overlay displaying a processing rotation;

a toggle activate spatial processing overlay configured to display the toggle state for generating at least one further at least one user interface overlay element;

a focus amplification overlay configured to display at least one amplification gain for audio sources inside a focus region;

a focus attenuation overlay configured to display at least one attenuation dampening for audio sources outside of a focus region;

a focus inverter overlay configured to display a toggle state for inverting the focus operation;

or

a focus amplification profile overlay configured to display an amplification profile for directions surrounding the apparatus.

19. The method as claimed in claim 16, wherein providing at least one control input associated with the at least one visual image element causes the apparatus to provide at least one of:

at least one control input generated by a motion of the apparatus displaying the at least one visual image element;

or

at least one control input generated by an operation of camera inputs associated with the at least one visual image element.

20. An apparatus comprising:

an image element determiner configured to determine at least one visual image element;

a spatial audio analyzer configured to receive at least two audio signals from at least two microphones, wherein each microphone of the at least two microphones is separated from each other;

the spatial audio analyzer configured to determine, using the at least two audio signals, a location of at least one audio source relative to the at least two microphones, wherein the at least one audio source is determined at least in part as environmental sound;

the spatial audio analyzer further configured to associate the at least one visual image element with the location of the at least one audio source relative to the at least two microphones;

a user interface controller configured to provide at least one control input associated with the at least one visual element, the at least one control input comprising at least one control for at least one focus region direction and width;

an audio processor configured to process the at least one audio source to change at least one audio characteristic of the at least one audio source according to the determined location of the at least one audio source based on the provided direction and width of the at least one focus region; and

the audio processor further configured to amplify or attenuate the at least one audio source for playback dependent on the location of the at least one audio source based on the at least one control for the at least one focus region direction and width.