Semantic image classification and search

• US 8,391,618 B1
• Filed: 11/26/2008
• Issued: 03/05/2013
• Est. Priority Date: 09/19/2008
• Status: Active Grant

• Current Assignee: Adobe Inc.
• Original Assignee: Adobe Systems Incorporated (Adobe Inc.)
• Inventors: Chuang, Jason C., Winnemoeller, Holger
• Primary Examiner(s): Chang, Jon
• Application Number:
  US12/324,328
• Time in Patent Office: 1,560 Days
• Field of Search: 382/224, 382/305, 382/306, 707/739
• US Class Current: 382/224
• CPC Class Codes: G06F 16/58 Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
• CPC Class Codes: G06F 16/583 using metadata automatically derived from the content
• CPC Class Codes: G06V 10/56 relating to colour
• CPC Class Codes: G06V 20/10 Terrestrial scenes scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59
• CPC Class Codes: G06V 30/274 Syntactic or semantic context, e.g. balancing

Abstract

A method, system, and computer-readable storage medium for categorizing digital images. A plurality of semantic category scores for a digital image are determined via application of a corresponding plurality of classifiers. A semantic category profile for the image is automatically determined based on the plurality of semantic category scores, where the semantic category profile characterizes semantic content of the image, and is useable to perform semantic based operations with respect to the image.

Claims

1. A non-transitory computer-readable storage medium that stores program instructions computer-executable to implement:

determining a plurality of semantic category scores for a digital image via application of a corresponding plurality of classifiers;

automatically determining a semantic category profile for the image based on the plurality of semantic category scores, wherein the semantic category profile characterizes semantic content of the image, and is useable to perform semantic based operations with respect to the image; and

performing a semantic based operation wherein the semantic based operation comprises;

a search operation based on a semantic similarity measure for a plurality of semantic category profiles for a plurality of digital images, wherein the plurality of semantic category profiles includes the semantic category profile for the image;

or

a keyword operation based on a semantic similarity measure for a plurality of semantic category profiles for a plurality of digital images, wherein the plurality of semantic category profiles includes the semantic category profile for the image.

2. The non-transitory computer-readable storage medium of claim 1, wherein the digital image is comprised in the plurality of digital images, wherein the program instructions are further computer-executable to implement:

performing said determining a plurality of semantic category scores and said automatically determining a semantic category profile for each other digital image of the plurality of digital images.

3. The non-transitory computer-readable storage medium of claim 1, wherein the plurality of classifiers composes at least one generic classifier comprising a plurality of sub-programs respectively implementing at least a subset of the plurality of classifiers.

4. The non-transitory computer-readable storage medium of claim 1, wherein the application of the plurality of classifiers is performed serially and/or concurrently.

5. The non-transitory computer-readable storage medium of claim 1, wherein the application of the plurality of classifiers is performed via multiple devices operating in conjunction over a network.

6. The non-transitory computer-readable storage medium of claim 1, wherein the semantic category profile for the image comprises the plurality of semantic category scores.

7. The non-transitory computer-readable storage medium of claim 1, wherein the digital image is comprised in the plurality of digital images, wherein the program instructions are further computer-executable to implement:

automatically determining a respective value of the semantic similarity measure for each digital image of at least a subset of the plurality of digital images with respect to the first digital image based on the semantic category profile of the first digital image and semantic category profiles of each digital image of the at least a subset of the plurality of digital images, wherein each digital image of the at least a subset of the plurality of digital images has one or more respective keywords; and

automatically determining one or more keywords from the respective keywords for association with the first digital image based on the values of the semantic similarity measure.

8. The non-transitory computer-readable storage medium of claim 7,

wherein the first digital image is associated with at least one keyword;

wherein said automatically determining the value of a respective semantic similarity measure for each digital image of at least a subset of a plurality of digital images comprises;

automatically determining co-occurrence of keywords associated respectively with each digital image of the plurality of digital images with respect to the at least one keyword of the first digital image, wherein the at least a subset of the plurality of digital images comprises the digital images with co-occurring keywords; and

wherein said automatically determining the one or more keywords from the respective keywords for association with the first digital image based on the values of the semantic similarity measure comprises;

automatically determining one or more keywords from the co-occurring keywords based on the values of the semantic similarity measure for association with the first digital image.

9. The non-transitory computer-readable storage medium of claim 1, wherein the digital image is comprised in a digital image database, wherein the program instructions are further computer-executable to implement:

receiving input specifying one or more keywords for a search query to a digital image database, wherein the digital image database comprises the plurality of digital images, each digital image having a respective semantic category profile;

determining a first semantic category profile that corresponds to the one or more keywords by accessing a database of keywords associated with the plurality of semantic category profiles;

determining a respective value of the semantic similarity measure for each digital image in the digital image database based on the first semantic category profile and the respective semantic category profiles of the digital images in the digital image database; and

determining one or more digital images in the digital image database based on the values of the semantic similarity measure.

10. The non-transitory computer-readable storage medium of claim 1, wherein the digital image is comprised in the plurality of digital images, wherein the program instructions are further computer-executable to implement:

analyzing a plurality of semantic categories with respect to the plurality of digital images based on one or more criteria, wherein the one or more criteria comprise at least one of;

relevance of each of the semantic categories;

consistency with human based classification;

or

accuracy of each of the semantic categories; and

automatically determining a subset of the semantic categories that efficiently spans the semantic content of the plurality of digital images based on said analyzing.

11. The non-transitory computer-readable storage medium of claim 1, wherein the digital image is comprised in the plurality of digital images, wherein the program instructions are further computer-executable to implement:

analyzing a plurality of sets of semantic categories with respect to the plurality of digital images based on one or more criteria, wherein the one or more criteria comprises at least one of;

relevance of each of the semantic categories;

consistency with human based classification;

or

accuracy of each of the semantic categories; and

automatically ranking the plurality of sets of semantic categories based on said analyzing.

12. The non-transitory computer-readable storage medium of claim 1, wherein the digital image is comprised in the plurality of digital images, wherein the program instructions are further computer-executable to implement:

determining information content of a plurality of semantic categories with respect to the plurality of digital images; and

automatically determining a subset of the plurality of semantic categories based on the determined information content, wherein each of the subset of the plurality of semantic categories is operable to partition the plurality of digital images.

13. A computer-implemented method, comprising:

determining a plurality of semantic category scores for a digital image via application of a corresponding plurality of classifiers; and

automatically determining a semantic category profile for the image based on the plurality of semantic category scores, wherein the semantic category profile characterizes semantic content of the image, and is useable to perform semantic based operations with respect to the image;

analyzing a plurality of semantic categories with respect to a plurality of digital images based on one or more criteria, wherein the one or more criteria comprise at least one of;

relevance of each of the semantic categories;

consistency with human based classification;

or

accuracy of each of the semantic categories; and

automatically performing;

determining a subset of the semantic categories that efficiently spans the semantic content of the plurality of digital images based on said analyzing;

or

ranking the plurality of sets of semantic categories based on said analyzing.

14. The method of claim 13, wherein the digital image is comprised in the plurality of digital images, the method further comprising:

performing said determining a plurality of semantic category scores and said automatically determining a semantic category profile for each other digital image of the plurality of digital images.

15. The method of claim 13, wherein the plurality of classifiers composes at least one generic classifier comprising a plurality of sub-programs respectively implementing at least a subset of the plurality of classifiers.

16. The method of claim 13, wherein the application of the plurality of classifiers is performed serially and/or concurrently.

17. The method of claim 13, wherein the application of the plurality of classifiers is performed via multiple devices operating in conjunction over a network.

18. The method of claim 13, wherein the semantic category profile for the image comprises the plurality of semantic category scores.

19. The method of claim 13, wherein the digital image is comprised in the plurality of digital images, the method further comprising:

automatically determining a respective value of a semantic similarity measure for each digital image of at least a subset of the plurality of digital images with respect to the first digital image based on the semantic category profile of the first digital image and semantic category profiles of each digital image of the at least a subset of the plurality of digital images, wherein each digital image of the at least a subset of the plurality of digital images has one or more respective keywords; and

automatically determining one or more keywords from the respective keywords for association with the first digital image based on the values of the semantic similarity measure.

20. The method of claim 19,

wherein the first digital image is associated with at least one keyword;

wherein said automatically determining a respective semantic similarity measure for each digital image of at least a subset of a plurality of digital images comprises;

automatically determining co-occurrence of keywords associated respectively with each digital image of the plurality of digital images with respect to the at least one keyword of the first digital image, wherein the at least a subset of the plurality of digital images comprises the digital images with co-occurring keywords; and

wherein said automatically determining the one or more keywords from the respective keywords for association with the first digital image based on the values of the semantic similarity measure comprises;

automatically determining one or more keywords from the co-occurring keywords based on the values of the semantic similarity measure for association with the first digital image.

21. The method of claim 13, wherein the digital image is comprised in a digital image database, the method further comprising:

receiving input specifying one or more keywords for a search query to a digital image database, wherein the digital image database comprises the plurality of digital images, each digital image having a respective semantic category profile;

determining a first semantic category profile that corresponds to the one or more keywords by accessing a database of keywords associated with semantic category profiles;

determining a respective semantic similarity measure for each digital image in the digital image database based on the first semantic category profile and the respective semantic category profiles of the digital images in the digital image database; and

determining one or more digital images in the digital image database based on the values of the semantic similarity measure.

22. The method of claim 13, wherein the digital image is comprised in the plurality of digital images, the method further comprising:

determining information content of the plurality of semantic categories with respect to the plurality of digital images; and

automatically determining a subset of the plurality of semantic categories based on the determined information content, wherein each of the subset of the plurality of semantic categories is operable to partition the plurality of digital images.

23. A system, comprising:

at least one processor; and

a memory coupled to the at least one processor, wherein the memory stores program instructions, wherein the program instructions are executable by the at least one processor to;

determine a plurality of semantic category scores for a digital image via application of a corresponding plurality of classifiers;

automatically determine a semantic category profile for the image based on the plurality of semantic category scores, wherein the semantic category profile characterizes semantic content of the image, and is useable to perform semantic based operations with respect to the image; and

perform a semantic based operation, wherein the semantic based operation comprises;

a search operation based on a semantic similarity measure for a plurality of semantic category profiles for a plurality of digital images, wherein the plurality of semantic category profiles includes the semantic category profile for the image;

or

a keyword operation based on a semantic similarity measure for a plurality of semantic category profiles for a plurality of digital images, wherein the plurality of semantic category profiles includes the semantic category profile for the image.

24. The system of claim 23, wherein the digital image is comprised in the plurality of digital images, wherein the program instructions are further executable to:

perform said determining a plurality of semantic category scores and said automatically determining a semantic category profile for each other digital image of the plurality of digital images.

25. The system of claim 23, wherein the plurality of classifiers composes at least one generic classifier comprising a plurality of sub-programs respectively implementing at least a subset of the plurality of classifiers.

26. The system of claim 23, wherein the application of the plurality of classifiers is performed serially and/or concurrently.

27. The system of claim 23, wherein the application of the plurality of classifiers is performed via multiple devices operating in conjunction over a network.

28. The system of claim 23, wherein the semantic category profile for the image comprises the plurality of semantic category scores.

29. The system of claim 23, wherein the digital image is comprised in the plurality of digital images, wherein the program instructions are further executable to:

automatically determine the respective semantic similarity measure for each digital image of at least a subset of the plurality of digital images with respect to the first digital image based on the semantic category profile of the first digital image and semantic category profiles of each digital image of the at least a subset of the plurality of digital images, wherein each digital image of the at least a subset of the plurality of digital images has one or more respective keywords; and

automatically determine one or more keywords from the respective keywords for association with the first digital image based on the values of the semantic similarity measure.

30. The system of claim 29,

wherein the first digital image is associated with at least one keyword;

wherein to automatically determine a respective semantic similarity measure for each digital image of at least a subset of the plurality of digital images, the program instructions are executable to;

automatically determining co-occurrence of keywords associated respectively with each digital image of the plurality of digital images with respect to the at least one keyword of the first digital image, wherein the at least a subset of the plurality of digital images comprises the digital images with co-occurring keywords; and

wherein to automatically determine the one or more keywords from the respective keywords for association with the first digital image based on the values of the semantic similarity measure, the program instructions are executable to;

automatically determine one or more keywords from the co-occurring keywords based on the values of the semantic similarity measure for association with the first digital image.

31. The system of claim 23, wherein the digital image is comprised in a digital image database, wherein the program instructions are further executable to:

receive input specifying one or more keywords for a search query to a digital image database, wherein the digital image database comprises the plurality of digital images, each digital image having a respective semantic category profile;

determine a first semantic category profile that corresponds to the one or more keywords by accessing a database of keywords associated with the plurality of semantic category profiles;

determine a respective semantic similarity measure for each digital image in the digital image database based on the first semantic category profile and the respective semantic category profiles of the digital images in the digital image database; and

determine one or more digital images in the digital image database based on the values of the semantic similarity measure.

32. The system of claim 23, wherein the digital image is comprised in the plurality of digital images, wherein the program instructions are further executable to:

analyze a plurality of semantic categories with respect to the plurality of digital images based on one or more criteria, wherein the one or more criteria comprise at least one of;

relevance of each of the semantic categories;

consistency with human based classification;

or

accuracy of each of the semantic categories; and

automatically determine a subset of the semantic categories that efficiently spans the semantic content of the plurality of digital images based on said analyzing.

33. The system of claim 23, wherein the digital image is comprised in the plurality of digital images, wherein the program instructions are further executable to:

analyze a plurality of sets of semantic categories with respect to the plurality of digital images based on one or more criteria, wherein the one or more criteria comprises at least one of;

relevance of each of the semantic categories;

consistency with human based classification;

or

accuracy of each of the semantic categories; and

automatically rank the plurality of sets of semantic categories based on said analyzing.

34. The system of claim 23, wherein the digital image is comprised in the plurality of digital images, wherein the program instructions are further executable to:

determine information content of a plurality of semantic categories with respect to the plurality of digital images; and

automatically determine a subset of the plurality of semantic categories based on the determined information content, wherein each of the subset of the plurality of semantic categories is operable to partition the plurality of digital images.

35. A computer-implemented method, comprising:

executing instructions on a specific apparatus to perform determining a plurality of semantic category scores for a digital image via application of a corresponding plurality of classifiers, wherein the application of the plurality of classifiers is performed via multiple devices operating in conjunction over a network;

executing instructions on said specific apparatus to perform automatically determining a semantic category profile for the image based on the plurality of semantic category scores, wherein the semantic category profile characterizes semantic content of the image, and is useable to perform semantic based operations with respect to the image.