Methods and apparatuses for generating co-salient thumbnails for digital images

• US 8,619,098 B2
• Filed: 08/25/2010
• Issued: 12/31/2013
• Est. Priority Date: 09/18/2009
• Status: Active Grant

• Current Assignee: Adobe Inc.
• Original Assignee: Adobe Systems Incorporated (Adobe Inc.)
• Inventors: Shechtman, Elya, Goldman, Daniel R., Jacobs, David E.
• Primary Examiner(s): Amini, Javid A
• Application Number:
  US12/868,519
• Publication Number:
  US 20130120438A1
• Time in Patent Office: 1,224 Days
• Field of Search: 345/634, 345/629, 345/204, 345/2.1, 345/156, 345/168, 345/173, 345/589, 345/619, 345/632, 345/633
• US Class Current: 345/635
• CPC Class Codes: G06T 11/60 Editing figures and text; Combining figures or text
• CPC Class Codes: G06T 2210/22 Cropping
• CPC Class Codes: G06T 3/04 Context-preserving transformations, e.g. by using an importance map panospheric to cylindrical image transformations G06T3/12
• CPC Class Codes: G06V 10/751 Comparing pixel values or logical combinations thereof, or feature values having positional relevance, e.g. template matching
• CPC Class Codes: G06V 10/759 Region-based matching
• CPC Class Codes: G09G 2320/0626 for control of overall brightness
• CPC Class Codes: G09G 2320/0686 with two or more screen areas displaying information with different brightness or colours
• CPC Class Codes: G09G 2320/08 Arrangements within a display terminal for setting, manually or automatically, display parameters of the display terminal
• CPC Class Codes: G09G 2320/10 Special adaptations of display systems for operation with variable images
• CPC Class Codes: G09G 2340/12 Overlay of images, i.e. displayed pixel being the result of switching between the corresponding input pixels
• CPC Class Codes: G09G 5/14 Display of multiple viewports
• CPC Class Codes: H04N 1/3875 combined with enlarging or reducing enlarging or reducing per se H04N1/393

Abstract

Methods and apparatus for generating context-sensitive, co-salient thumbnails are described. Co-salient thumbnails may effectively convey more useful information than conventional thumbnails while using the same or similar display space. As such, co-salient thumbnails may make it easier for a user to locate a particular image in a large collection of similar images. In some embodiments, for each of a plurality of patches in a first image, an illustrative method may identify a corresponding similar patch in a second image based on one or more similarity measures. The method may then determine a distance from each of the plurality of patches to its corresponding similar patch and select a crop in the first image relative to the second image based on the determined distance. Optionally, the method may repeat these operations in the opposite direction to select a crop in the second image relative to the first image.

Claims

1. A method comprising:

locating, for each of a plurality of patches in a first image, a corresponding similar patch in a second image based on one or more similarity measures;

determining, by one or more computing devices, a score for each of the plurality of patches with respect to its corresponding similar patch, determining the score comprising determining one or more of a divergence metric or a distance metric;

selecting at least one saliently different crop in the first image based at least in part on the determined scores for the plurality of patches, the saliently different crop highlighting a salient difference in the first image relative to the second image; and

generating at least one thumbnail image for the first image based at least in part on the at least one saliently different crop.

2. The method of claim 1, wherein the divergence metric comprises a local similarity of offsets that captures a divergence of neighboring correspondence vectors or offsets.

3. The method of claim 1, wherein the method further comprises creating a distance map based at least in part on one or more of the divergence metric or the distance metric, and wherein selecting the at least one saliently different crop is based at least in part on the distance map.

4. The method of claim 3, wherein the performing by one or more computer systems further includes removing an artifact from the distance map prior to selecting the at least one saliently different crop, wherein the artifact includes one or more of a panning artifact, a rotation artifact, or a planar motion.

5. The method of claim 4, wherein the method further comprises normalizing the distance map according to a local contrast measure prior to selecting the at least one saliently different crop.

6. The method of claim 5, wherein the performing by one or more computer systems further includes emphasizing independently moving regions of at least one of the first and second images on the distance map prior to selecting the at least one saliently different crop.

7. The method of claim 1, wherein the method further comprises:

repeating said selecting the at least one saliently different crop in the first image relative to the second image at each of a plurality of scales; and

comparing results generated at the plurality of scales to select at least one crop from the first image and at least one crop from the second image as co-salient thumbnails for the first and second images, respectively.

8. A system, comprising:

a processor;

a memory comprising instructions stored thereon that, responsive to execution by the processor, perform operations comprising;

identifying a first image and a second image;

calculating a first distance map for each of a plurality of areas in the first image with respect to corresponding areas in the second image;

selecting a first crop window within the first image that includes one or more areas in the first image that are farthest, in the first distance map, from their corresponding areas in the second image; and

creating a first thumbnail image for the first image based, at least in part, on the first crop window, the first thumbnail image highlighting a salient difference in the first image relative to the second image.

9. The system of claim 8, the operations further comprising filtering the first distance map.

10. The system of claim 9, wherein filtering the first distance map comprises removing a panning artifact, said removing including:

identifying an offset between at least a portion of the first image and at least a portion of the second image; and

masking a region from the first distance map that is outside of an intersection between the first image and the second image based at least in part on the offset.

11. The system of claim 9, wherein filtering the first distance map comprises normalizing the distance map with respect to a local contrast, said normalizing including:

multiplying the first distance map by a measure inversely proportional to a local contrast of at least one of the first and second images.

12. The system of claim 11, wherein the measure comprises a blurred gradient magnitude of a luminance channel of at least one of the first and second images.

13. The system of claim 9, the operations further comprising:

determining an offset value for each of a plurality of pixels in the first image with respect to a corresponding pixel in the second image; and

generating an offset field based at least in part on the determined offset values.

14. The system of claim 13, wherein filtering the first distance map comprises:

multiplying the first distance map by a measure of a local incoherency of the offset field.

15. The system of claim 8, wherein the selecting the first crop window within the first image comprises:

applying a filter to the first distance map; and

identifying a crop at a position within the first image that yields a high distance value in the filtered distance map.

16. The system of claim 15, wherein the high distance value is a maximum distance value.

17. The system of claim 8, the operations further comprising:

repeating at least one of said selecting operations at each of a plurality of scales.

18. The system of claim 8, the operations further comprising:

calculating a second distance map for each of a plurality of areas in the second image with respect to corresponding areas in the first image;

selecting a second crop window within the second image that includes one or more areas in the second image that are farthest, in the second distance map, from their corresponding areas in the first image; and

creating a second thumbnail image for the second image based, at least in part, on the second crop window.

19. A non-transitory computer-readable storage medium having instructions stored thereon that, upon execution by a computer system, cause the computer system to perform operations comprising:

identifying a first image and a second image;

determining, for a first area within the first image, a first corresponding area within the second image;

calculating a first distance between the first area and the first corresponding area;

determining, for a second area within the first image, a second corresponding area within the second image;

calculating a second distance between the second area and the second corresponding area;

selecting the first area in response to the first distance being greater than the second distance; and

creating a thumbnail image for the first image based, at least in part, on the selected area, the thumbnail image for the first image highlighting a salient difference in the first image relative to the second image.

20. The non-transitory computer-readable storage medium of claim 19, wherein the operations further comprise:

selecting the second area in response to the second distance being greater than the first distance.

21. The non-transitory computer-readable storage medium of claim 19, wherein the operations further comprise:

determining, for a third area within the second image, a third corresponding area within the first image;

calculating a third distance between the third area and the third corresponding area;

determining, for a fourth area within the second image, a fourth corresponding area within the first image;

calculating a fourth distance between the fourth area and the fourth corresponding area;

selecting the third area in response to the third distance being greater than the fourth distance; and

creating a thumbnail image for the second image based, at least in part, on the selected area.

22. The non-transitory computer-readable storage medium of claim 21, wherein the operations further comprise:

selecting the fourth area in response to the fourth distance being greater than the third distance.