VISUAL SALIENCY ESTIMATION FOR IMAGES AND VIDEO

• US 20140063275A1
• Filed: 08/29/2012
• Published: 03/06/2014
• Est. Priority Date: 08/29/2012
• Status: Active Grant

• Current Assignee: Disney Enterprises Incorporated (The Walt Disney Company)
• Original Assignee: Federico Perazzi, Philipp Krahenbuhl
• Inventors: Krhenbhl, Philipp, Pritch, Yael, Perazzi, Federico, Hornung, Alexander
• Granted Patent: US 9,025,880 B2
• Time in Patent Office: Days
• US Class Current: 348/208.4
• CPC Class Codes: G06F 18/22 Matching criteria, e.g. proximity measures
• CPC Class Codes: G06T 2207/10004 Still image; Photographic image
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06T 7/194 involving foreground-background segmentation
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G06T 7/44 using image operators, e.g. filters, edge density metrics or local histograms
• CPC Class Codes: G06T 7/90 Determination of colour characteristics
• CPC Class Codes: G06V 10/462 Salient features, e.g. scale invariant feature transforms [SIFT]
• CPC Class Codes: G06V 10/56 relating to colour
• CPC Class Codes: G06V 10/758 Involving statistics of pixels or of feature values, e.g. histogram matching
• CPC Class Codes: H04N 19/117 Filters, e.g. for pre-processing or post-processing sub-band filter banks H04N19/635

Abstract

Methods and apparatus for visual saliency estimation for images and video are described. In an embodiment, a process includes decomposing, by a processor, an image into elements, wherein each element includes at least one pixel. The processor then calculates a first image measure indicative of each element'"'"'s uniqueness in the image on a per element basis, and a second image measure indicative of each element'"'"'s spatial distribution in the image on a per element basis. A per element saliency measure is provided by combining the first image measure and the second image measure, or by utilizing the first image measure, or by utilizing the second image measure.

Claims

1. A method, comprising:

decomposing, by a processor, an image into elements, wherein each element comprises at least one pixel;

calculating, by the processor, at least one of a first image measure indicative of each element'"'"'s uniqueness in the image on a per element basis and a second image measure indicative of each element'"'"'s spatial distribution in the image on a per element basis; and

providing, by the processor, a per element saliency measure by at least one of combining the first image measure and the second image measure, utilizing the first image measure, or utilizing the second image measure.

2. The method of claim 1, further comprising generating, by the processor based on the per element saliency measure, a pixel-accurate image saliency map.

3. The method of claim 1, wherein at least one of the first image measure and the second image measure is calculated based on at least one discriminating feature extracted from the pixels associated with each element.

4. The method of claim 3, wherein the discriminating feature comprises at least one of a color feature, a texture feature, a disparity feature, or a motion feature.

5. The method of claim 1, wherein decomposing comprises utilizing, by the processor, a modified simple linear iterative clustering (SLIC) superpixels protocol that utilizes K-means clustering in geodesic image distance in CEILab space.

6. The method of claim 1, wherein calculating the at least one of the first image measure and the second image measure comprises utilizing Gaussian filtering.

7. The method of claim 1, wherein combining the first image measure and the second image measure comprises:

normalizing uniqueness measures and distribution measures to a range; and

combining each normalized uniqueness measure with a corresponding distribution measure to obtain the per element saliency measure.

8. The method of claim 2, wherein generating the pixel accurate saliency map comprises utilizing an up-sampling process that is both local and feature sensitive.

9. A computer-readable medium storing non-transitory instructions configured to cause a processor to:

decompose an image into elements, wherein each element comprises at least one pixel;

calculate at least one of a first image measure indicative of each element'"'"'s uniqueness in the image on a per element basis and a second image measure indicative of each element'"'"'s spatial distribution in the image on a per element basis; and

provide a per element saliency measure by at least one of combining the first image measure and the second image measure, utilizing the first image measure, or utilizing the second image measure.

10. The computer-readable medium of claim 9, further comprising instructions configured to cause the processor to generate, based on the per element saliency measure, a pixel-accurate image saliency map.

11. The computer-readable medium of claim 9, wherein the instructions for calculating the first image measure or the second image measure further comprise instructions configured to cause the processor to calculate the first image measure or the second image measure based on at least one discriminating feature extracted from the pixels associated with each element.

12. The computer-readable medium of claim 9, wherein the instructions for decomposing further comprise instructions configured to cause the processor to utilize a modified simple linear iterative clustering (SLIC) superpixels protocol that utilizes K-means clustering in geodesic image distance in CEILab space.

13. The computer-readable medium of claim 9, wherein the instructions for calculating the first and second image measures further comprise instructions configured to cause the processor to utilize Gaussian filtering.

14. The computer-readable medium of claim 9, wherein the instructions for combining the first and second image measures further comprise instructions configured to cause the processor to:

normalize uniqueness measures and distribution measures to a range; and

combine each normalized uniqueness measure with a corresponding distribution measure to obtain the per element saliency measure.

15. The computer-readable medium of claim 10, wherein the instructions for generating the pixel accurate saliency map further comprise instructions configured to cause the processor to utilize an up-sampling process that is both local and feature sensitive.

16. An apparatus, comprising:

a processor;

a communication device operably connected to the processor; and

a storage device operably connected to the storage device, the storage device storing non-transitory instructions configured to cause the processor to;

decompose an image into elements, wherein each element comprises at least one pixel;

calculate at least one of a first image measure indicative of each element'"'"'s uniqueness in the image on a per element basis and a second image measure indicative of each element'"'"'s spatial distribution in the image on a per element basis; and

provide a per element saliency measure by at least one of combining the first image measure and the second image measure, utilizing the first image measure, or utilizing the second image measure.

17. A method comprising:

identifying, by a processor based on a per element saliency measure, a background portion of a video frame of a video;

aligning, by the processor based on the identified background portion, the video frame with an adjacent video frame using a global transformation to remove camera motion;

computing an optical flow between the two aligned video frames;

clustering the optical flow into flow elements; and

computing a uniqueness measure and a spatial distribution measure of the optical flow elements to result in a video saliency map.

18. The method of claim 17, wherein aligning based on the background portion of the video frame comprises calculating, by the processor, the global transform using a RANSAC-based method.

19. A computer-readable medium storing non-transitory instructions configured to cause a processor to:

identify, based on a per element saliency measure, a background portion of a video frame of a video;

align, based on the identified background portion, the video frame with an adjacent video frame using a global transformation to remove camera motion;

compute an optical flow between the two aligned video frames;

cluster the optical flow into flow elements; and

compute a uniqueness measure and a spatial distribution measure of the optical flow elements to result in a video saliency map.

20. The computer-readable medium of claim 19, wherein the instructions for aligning further comprises instructions configured to cause the processor to calculate the global transform using a RANSAC-based method.

21. An apparatus, comprising:

a processor;

a communication device operably connected to the processor; and

a storage device operably connected to the storage device, the storage device storing non-transitory instructions configured to cause the processor to;

identify, based on a per element saliency measure, a background portion of a video frame of a video;

align, based on the identified background portion, the video frame with an adjacent video frame using a global transformation to remove camera motion;

compute an optical flow between the two aligned video frames;

cluster the optical flow into flow elements; and

compute a uniqueness measure and a spatial distribution measure of the optical flow elements to result in a video saliency map.