Image point of interest analyser with animation generator

• US 10,068,363 B2
• Filed: 03/27/2013
• Issued: 09/04/2018
• Est. Priority Date: 03/27/2013
• Status: Active Grant

• Current Assignee: Nokia Technologies Oy (Nokia Corporation)
• Original Assignee: Nokia Technologies Oy (Nokia Corporation)
• Inventors: Laaksonen, Lasse Juhani, Vilermo, Miikka Tapani, Tammi, Mikko Tapio
• Primary Examiner(s): Broome, Said
• Application Number:
  US14/777,852
• Publication Number:
  US 20160086368A1
• Time in Patent Office: 1,987 Days
• Field of Search: None
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06T 13/80 2D [Two Dimensional] animation, e.g. using sprites
• CPC Class Codes: G06T 7/73 using feature-based methods
• CPC Class Codes: H04N 23/61 based on recognised objects
• CPC Class Codes: H04N 23/62 Control of parameters via user interfaces
• CPC Class Codes: H04N 23/635 Region indicators; Field of view indicators
• CPC Class Codes: H04N 7/142 Constructional details of the terminal equipment, e.g. arrangements of the camera and the display
• CPC Class Codes: H04N 9/8063 using time division multiplex of the PCM audio and PCM video signals

Abstract

An apparatus comprising a point of interest analyser configured to define at least one region within an image as an interest region, determine a position associated with the at least one region; an audio track generator configured to determine at least one audio signal based on the position; and an animated presentation generator configured to generate an animated image comprising the at least one region and the at least one audio signal.

Claims

1. A method comprising:

defining at least one region within an image as an interest region;

from among sound sources in the image, determining at least one audio signal associated with the at least one region based upon a direction of arrival of the at least one audio signal as recorded by a microphone or microphone array, wherein determining at least one audio signal based on the direction of arrival comprises processing the at least one audio signal based on the at least one region to generate at least one positional audio signal part and synthesizing the at least one audio signal from the at least one positional audio signal part; and

generating an animated image comprising the at least one region and the at least one audio signal, wherein generating the animated image comprises separately controlling the at least one audio signal associated with the at least one region relative to an audio signal associated with another region or representative of ambient audio.

2. The method as claimed in claim 1, further comprising capturing the image using at least one camera associated with an apparatus and displaying the image on a display associated with the apparatus.

3. The method as claimed in claim 1, wherein defining at least one region comprises at least one of:

analyzing the image to determine the at least one region;

receiving a user input defining the at least one region;

receiving a touch input over a displayed version of the image defining the at least one region;

analyzing eye/gaze positioning of a displayed version of the image to determine the at least one region; and

analyzing the at least one audio signal associated with the image to determine the at least one region.

4. The method as claimed in claim 1, wherein processing at least one audio signal based on the at least one region to generate at least one positional audio signal part comprises:

analyzing the at least one audio signal to determine at least one positional component;

selecting the at least one positional component based on the at least one region; and

synthesizing the at least one audio signal from the at least one positional component.

5. The method as claimed in claim 4, wherein synthesizing the at least one audio signal from the at least one of the positional components further comprises synthesizing the at least one audio signal from the at least one positional component and a non-positional/ambient audio signal component.

6. The method as claimed in claim 1, wherein processing at least one audio signal based on the at least one region to generate at least one positional audio signal part comprises separating the at least one audio signal into at least two parts based on the at least one region.

9. The method as claimed in claim 1, further comprising:

displaying the at least one positional audio signal part on the display;

displaying at least one control element associated with the at least one positional audio signal part on the display; and

controlling application of the at least one positional audio signal part based on an input associated with the at least one control element.

10. The method as claimed in claim 9, wherein displaying at least one control element associated with the at least one positional audio signal part on the display comprises at least one of:

displaying a muting control element associated with the at least one positional audio signal part on the display, wherein the at least one positional audio signal part is muted/unmuted based on the input associated with the muting control element;

displaying a volume control element associated with the at least one positional audio signal part on the display, wherein a volume of the at least one positional audio signal part is based on the input associated with the volume control element; and

displaying a fading control element associated with the at least one positional audio signal part on the display, wherein a fading up/fading down of the at least one positional audio signal part is based on the input associated with the fading control element.

11. The method as claimed in claim 1, wherein defining at least one region within an image as an interest region comprises:

defining at least two regions within the image; and

selecting at least one region from the at least two regions within the image as the interest region.

12. The method as claimed in claim 2, wherein determining a position associated with the at least one region comprises determining a position relative to an apparatus.

13. The method as claimed in claim 1, further comprising determining a touch input towards defining the at least one region within the image and wherein the touch input is provided from a user interface to select the interest region.

14. The method as claimed in claim 1, wherein determining the at least one audio signal further comprises generating the at least one audio signal from the microphone or microphone array in such a way that the generated at least one audio signal is passed to an analyzer so as to determine the at least one audio signal.

15. The method as claimed in claim 14, wherein the analyzer further comprises a mid/side signal generator in such a way that a mid signal is generated comprising a dominant sound source as a direction and a side signal is generated comprising an ambient sound source.

16. The method as claimed in claim 1, further comprising mixing and synchronizing the at least one audio signal and a video data of the at least one region for the animated image.

19. The method as claimed in claim 1 further comprising determining a position associated with the at least one region.

7. A method comprising:

defining at least one region within an image as an interest region;

from among sound sources in the image, determining at least one audio signal associated with the at least one region based upon a direction of arrival of the at least one audio signal as recorded by a microphone or microphone array;

determining a time period associated with the at least one region; and

generating an animated image comprising the at least one region and the at least one audio signal, wherein generating the animated image comprises separately controlling the at least one audio signal associated with the at least one region relative to an audio signal associated with another region or representative of ambient audio, and wherein generating an animated image further comprises generating the at least one audio signal based on the at least one region and the time period associated with the at least one region.

8. The method as claimed in claim 7, wherein determining the at least one audio signal based on the at least one region and the time period associated with the at least one region comprises:

spatially and temporally processing the at least one audio signal based on the at least one region and time period associated with the at least one region to generate at least one spatially and temporally processed audio signal part; and

synthesizing the at least one audio signal from the at least one spatially and temporally processed audio signal part.

17. An apparatus comprising:

an analyzer configured to define at least one region within an image as a point of interest region;

an audio generator configured to determine, from among sound sources in the image, at least one audio signal associated with the at least one region based upon a direction of arrival of the at least one audio signal as recorded by a microphone or microphone array, wherein the audio generator is configured to determine at least one audio signal based on the direction of arrival by processing the at least one audio signal based on the at least one region to generate at least one positional audio signal part and synthesizing the at least one audio signal from the at least one positional audio signal part; and

a generator configured to generate an animated image comprising the at least one region and the at least one audio signal, wherein the generator is configured to permit separate control of the at least one audio signal associated with the at least one region relative to an audio signal associated with another region or representative of ambient audio.

20. The apparatus as claimed in claim 17 wherein the analyser is also configured to determine a position associated with the at least one region.

18. An apparatus comprising at least one processor and at least one memory including computer code for one or more programs, the at least one memory and the computer code configured to with the at least one processor cause the apparatus to at least:

define at least one region within an image as a point of interest region;

from among sound sources in the image, determine at least one audio signal associated with the at least one region based upon a direction of arrival of the at least one audio signal as recorded by a microphone or microphone array, wherein the apparatus is caused to determine at least one audio signal based on the direction of arrival by processing the at least one audio signal based on the at least one region to generate at least one positional audio signal part and synthesizing the at least one audio signal from the at least one positional audio signal part; and

generate an animated image comprising the at least one region and the at least one audio signal, wherein the at least one memory and the computer code are configured to with the at least one processor cause the apparatus to generate the animated image by separately controlling the at least one audio signal associated with the at least one region relative to an audio signal associated with another region or representative of ambient audio.

21. The apparatus as claimed in claim 18 wherein the at least one memory and the computer code are configured to with the at least one processor cause the apparatus to determine a position associated with the at least one region.

22. An apparatus comprising at least one processor and at least one memory including computer code for one or more programs, the at least one memory and the computer code configured to with the at least one processor cause the apparatus to at least:

define at least one region within an image as a point of interest region;

from among sound sources in the image, determine at least one audio signal associated with the at least one region based upon a direction of arrival of the at least one audio signal as recorded by a microphone or microphone array;

determine a time period associated with the at least one region; and

generate an animated image comprising the at least one region and the at least one audio signal, wherein the apparatus is caused to generate the animated image by separately controlling the at least one audio signal associated with the at least one region relative to an audio signal associated with another region or representative of ambient audio, and wherein the apparatus is further caused to generate an animated image by generating the at least one audio signal based on the at least one region and the time period associated with the at least one region.