Multi-dimensional realization of visual content of an image collection

• US 10,691,743 B2
• Filed: 08/05/2014
• Issued: 06/23/2020
• Est. Priority Date: 08/05/2014
• Status: Active Grant

• Current Assignee: SRI International, Inc.
• Original Assignee: SRI International, Inc.
• Inventors: Sawhney, Harpreet Singh, Eledath, Jayakrishnan, Bansal, Mayank, Matei, Bogdan C., Lv, Xutao, Desai, Chaitanya, Shields, Timothy
• Primary Examiner(s): Park, Chan S
• Assistant Examiner(s): Rice, Elisa M
• Application Number:
  US14/452,237
• Publication Number:
  US 20160042252A1
• Time in Patent Office: 2,149 Days
• CPC Class Codes: G06F 16/50 of still image data
• CPC Class Codes: G06F 16/54 Browsing; Visualisation therefor
• CPC Class Codes: G06F 16/55 Clustering; Classification
• CPC Class Codes: G06F 16/583 using metadata automatically derived from the content
• CPC Class Codes: G06F 16/5838 using colour
• CPC Class Codes: G06F 18/2323 based on graph theory, e.g. minimum spanning trees [MST] or graph cuts
• CPC Class Codes: G06V 10/7635 based on graphs, e.g. graph cuts or spectral clustering
• CPC Class Codes: G06V 20/35 Categorising the entire scene, e.g. birthday party or wedding scene
• CPC Class Codes: G06V 2201/09 Recognition of logos

Abstract

A computing system for realizing visual content of an image collection executes feature detection algorithms and semantic reasoning techniques on the images in the collection to elicit a number of different types of visual features of the images. The computing system indexes the visual features and provides technologies for multi-dimensional content-based clustering, searching, and iterative exploration of the image collection using the visual features and/or the visual feature indices.

Claims

1. A computing system for understanding the content of a collection of visual media files comprising one or more of digital images and digital videos, the computing system comprising a plurality of instructions embodied in one or more non-transitory machine accessible storage media and executable by one or more computing devices to cause the computing system to:

execute a plurality of different feature detection algorithms on the collection of visual media files;

determine, based on the execution of the feature detection algorithms, a plurality of different visual features depicted in the visual media files; and

cluster the visual media files by, for each of the visual media files in the collection;

computing a plurality of similarity measures, each similarity measure representing a measurement of similarity of visual content of at least one of the visual media files in the collection to only one of the visual features determined as a result of the execution of the feature detection algorithms; and

associating the at least one of the visual media files in the collection with a visual feature based on the similarity measure computed for the at least one of the visual media files with respect to the visual feature.

2. The computing system of claim 1, comprising instructions executable by the computing system to interactively display a cluster, the cluster graphically indicating the associations of the visual media files with the visual feature determined as a result of the execution of the feature detection algorithms.

3. The computing system of claim 1, comprising instructions executable by the computing system to display a graphical representation of the similarity measure computed for each of the visual media files in the cluster with respect to the visual feature determined as a result of the execution of the feature detection algorithms.

4. The computing system of claim 1, comprising instructions executable by the computing system to interactively display a plurality of different clusters, wherein each of the clusters graphically indicates the associations of the visual media files with a different visual feature determined as a result of the execution of the feature detection algorithms.

5. The computing system of claim 1, comprising instructions executable by the computing system to interactively display a graphical representation of the similarity measure computed for each of the visual media files with respect to each of the plurality of different clusters.

6. The computing system of claim 5, comprising instructions executable by the computing system to associate each of the visual media files with one or more clusters.

7. The computing system of claim 6, comprising instructions executable by the computing system to compute a super-cluster comprising a plurality of visual media files that are all associated with the same combination of multiple clusters.

8. The computing system of claim 1, comprising instructions executable by the computing system to (i) determine, based on the execution of the feature detection algorithms, a distinctive visual feature depicted in one or more of the visual media files, (ii) for each of the visual media files in the collection:

compute a similarity measure representing a measurement of similarity of visual content of the visual media file to the distinctive visual feature, and associate the visual media file with the distinctive visual feature based on the computed similarity measure; and

(iii) interactively display the cluster graphically indicating the associations of the visual media files with the distinctive visual feature.

9. The computing system of claim 8, wherein the distinctive visual feature is representative of one or more of:

a logo, a trademark, a slogan, a distinctive object, a distinctive scene, and a distinctive pattern of imagery.

10. The computing system of claim 1, comprising instructions executable by the computing system to (i) determine, based on the execution of the feature detection algorithms, an attribute of one of the visual features, (ii) for each of the visual media files in the collection:

compute a similarity measure representing a measurement of similarity of visual content of the visual media file to the attribute of the visual feature, and associate the visual media file with the attribute of the visual feature based on the computed similarity measure; and

(iii) interactively display the cluster graphically indicating the associations of the visual media files with the attribute of the visual feature.

11. The computing system of claim 10, wherein the attribute of the visual feature comprises one or more of:

a shape, a size, a location, a color, and a texture of the visual feature.

12. The computing system of claim 1, wherein the visual content used to compute the similarity measure is a localized portion of the visual content of the entire visual media file, and the computing system comprises instructions executable by the computing system to associate the visual media file with a visual feature determined as a result of the execution of the feature detection algorithms based on the similarity measure computed for the localized portion of the visual content of at least one of the visual media files in the collection with respect to the visual feature.

13. The computing system of claim 1, comprising instructions executable by the computing system to determine one of the visual features depicted in the visual media files based on the execution of a combination of different feature detection algorithms.

14. The computing system of claim 1, comprising instructions executable by the computing system to select the plurality of different feature detection algorithms to execute on the collection of visual media files based on an algorithm selection criterion.

15. The computing system of claim 1, comprising instructions executable by the computing system to select the plurality of different feature detection algorithms from a set of feature detection algorithms comprising algorithms configured to generate visual feature descriptors at a plurality of different levels of abstraction.

16. The computing system of claim 15, comprising instructions executable by the computing system to select the plurality of different feature detection algorithms from a set of feature detection algorithms comprising algorithms configured to detect low-level features and algorithms configured to detect semantic-level features.

17. The computing system of claim 1, comprising instructions executable by the computing system to select the plurality of different feature detection algorithms to execute on the collection of visual media files based on an algorithm selection criterion.

18. The computing system of claim 1, comprising instructions executable by the computing system to modify the cluster in response a clustering cue comprising one or more of:

a geometrically-based image similarity criterion, user input specifying an image similarity criterion, and user feedback implicitly indicating a similarity criterion.

19. The computing system of claim 1, comprising instructions executable by the computing system to interactively display an unspecified number of different clusters, wherein each of the clusters graphically indicates the associations of the visual media files with a different visual feature determined as a result of the execution of the feature detection algorithms.

20. The computing system of claim 1, comprising instructions executable by the computing system to compute a purity metric indicative of the degree to which images in a given cluster have a visual feature or set of visual features in common, and modify one or more of the clusters based on the purity metric.