Identifying particular images from a collection

• US 9,037,569 B2
• Filed: 11/14/2013
• Issued: 05/19/2015
• Est. Priority Date: 02/04/2011
• Status: Active Grant

• Current Assignee: Kodak Alaris Inc (KPP (no. 2) Trustees Ltd.)
• Original Assignee: Kodak Alaris Inc (KPP (no. 2) Trustees Ltd.)
• Inventors: Wood, Mark D., Loui, Alexander C.
• Primary Examiner(s): Ruiz, Angelica
• Application Number:
  US14/079,787
• Publication Number:
  US 20140074825A1
• Time in Patent Office: 551 Days
• US Class Current: 707/713
• CPC Class Codes: G06F 16/24573 using data annotations, e.g. user-defined metadata
• CPC Class Codes: G06F 16/50 of still image data
• CPC Class Codes: G06F 16/53 Querying
• CPC Class Codes: G06F 16/583 using metadata automatically derived from the content
• CPC Class Codes: G06F 16/5866 using information manually generated, e.g. tags, keywords, comments, manually generated location and time information
• CPC Class Codes: G06F 16/60 of audio data
• CPC Class Codes: G06F 16/70 of video data
• CPC Class Codes: G06F 16/907 Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
• CPC Class Codes: G06F 16/9535 Search customisation based on user profiles and personalisation
• CPC Class Codes: G06F 16/9538 Presentation of query results

Abstract

A method of identifying one or more particular images from an image collection, includes indexing the image collection to provide image descriptors for each image in the image collection such that each image is described by one or more of the image descriptors; receiving a query from a user specifying at least one keyword for an image search; and using the keyword(s) to search a second collection of tagged images to identify co-occurrence keywords. The method further includes using the identified co-occurrence keywords to provide an expanded list of keywords; using the expanded list of keywords to search the image descriptors to identify a set of candidate images satisfying the keywords; grouping the set of candidate images according to at least one of the image descriptors, and selecting one or more representative images from each grouping; and displaying the representative images to the user.

Claims

1. A method of identifying one or more particular images from an image collection, comprising using a data processing system for:

(a) indexing the image collection to provide image descriptors for each image in the image collection such that each image is described by one or more image descriptor, with or without user input;

(b) receiving a query from a user specifying at least one keyword for an image search, wherein the at least one keyword is not an image descriptor;

(c) using the keyword(s) to search a second collection of tagged images or a lexical database to identify keywords that are co-occurrent with the user-provided keyword(s);

(d) using identified co-occurrence keywords to provide an expanded list of keywords;

(e) using the expanded list of keywords to search the image descriptors to identify a set of candidate images satisfying the keywords;

(f) grouping the set of candidate images according to at least one of the image descriptors, and selecting one or more representative images from each grouping; and

(g) displaying the representative images to the user.

2. The method of claim 1 further comprising using a processor for receiving user confirmation that the set of candidate images is an appropriate result.

3. The method of claim 2 wherein upon receiving said confirmation, the method further comprises using a processor for incorporating the expanded list of keywords as image descriptors for each image in the set of candidate images.

4. The method of claim 1, wherein prior to said step (c), the method further comprises differentiating between subsets of images within the second collection based on tagged demographic information, and in said step (c), using the keyword(s) to search one of the differentiated subsets of the second collection to identify keywords that are co-occurrent with the user-provided keyword(s).

5. A method of identifying one or more particular images from an image collection, comprising using a data processing system for:

(a) indexing the image collection to provide one or more image descriptors for each image in the image collection wherein each image is described by one or more of the image descriptors and wherein each image descriptor belongs to one or more categories;

(b) receiving a query from a user containing at least one keyword for an image search;

(c) extracting a set of one or more keywords from the user-provided query;

(d) expanding the set of keywords by adding tagged keywords associated with tagged images from a tagged image collection or by adding related keywords from a lexical database, wherein one or more of the added tagged keywords or related keywords co-occurs with one or more of the keywords extracted from the user-provided query;

(e) using the expanded set of keywords to search the image descriptors;

(f) identifying a set of candidate images having image descriptors that match the expanded set of keywords;

(g) grouping the set of candidate images according to at least one of the image descriptors;

(h) selecting a representative image for each grouping of candidate images; and

(i) displaying the representative images to the user.

6. The method of claim 5, wherein the mechanism for expanding a given category considers one or more keywords belonging to other categories.

7. The method of claim 6, wherein the categories include geographic location or feature type, scenery or material type, color, activity, object, or people type.

8. The method of claim 5, wherein the grouping in step (g) is determined by temporal information, geospatial information, people information, or visual similarity.

9. The method of claim 5 wherein the displaying of representative images in step (i) is responsive to the categories.

10. The method of claim 5 wherein indexing the image collection is performed according to categories based on image capture date and time.

11. The method of claim 5 wherein indexing the image collection is performed according to a category selected from the group consisting of:

scene classification, material and texture classification, object classification, and color classification.

12. The method of claim 5 wherein indexing the image collection comprises:

using an algorithm to gather personal features data of one or more persons appearing in the images in the image collection, wherein personal features data comprises facial shape data, hair and clothing style data, and social context data;

analyzing the personal features data of the one or more persons to estimate age and gender; and

associating image descriptors for images in the image collection according to the estimated age and gender of the one or more persons appearing in the images.

13. The method of claim 5 wherein indexing is based on geographic location or features of the images in the image collection, the geographic location determined by reverse geocoding.

14. The method of claim 5 further comprising using a data processing system for computing a co-occurrency matrix to determine co-occurrency frequency, the co-occurrency frequency being the frequency at which the tagged keywords co-occur with one or more of the keywords in the set of keywords extracted from the user-provided query.

15. The method of claim 14, wherein a tagged keyword is added to the set of keywords extracted from the user-provided query only if the tagged keyword'"'"'s co-occurrency frequency exceeds a threshold value.

16. The method of claim 5, wherein after extracting a set of keywords from the user-provided query, the method further comprises using a data processing system for filtering the set of keywords.

17. The method of claim 5, wherein selecting a representative image for each grouping of candidate images is performed according to a scoring function.

18. The method of claim 5, wherein after identifying a set of candidate images, the method further comprises using a data processing system for ranking each image in the set of candidate images according to an estimated strength of the match between the one or more image descriptors associated with each image and the expanded set of keywords.

19. The method of claim 5 further comprising using a processor for receiving user confirmation that the set of candidate images is an appropriate result.

20. The method of claim 19 wherein upon receiving said confirmation, the method further comprises using a processor for incorporating the expanded set of keywords as image descriptors for each image in the set of candidate images.

21. A user interface system configured to interact with a data processing system in performing the method of claim 5, the user interface system comprising:

a primary user interface design displayed on a display device comprising;

a query entry area for accepting user-provided search queries;

a search term area for displaying an expanded set of keywords used to search image descriptors;

a first scrollable region for displaying representative images; and

a second scrollable region for displaying at least file pathnames and search scores corresponding to the representative images displayed in the first scrollable region.

22. The user interface system of claim 21 further comprising:

a secondary user interface design displayed on the display device comprising;

a third scrollable region for displaying images of a specific set of images represented by one of the representative images;

an image display region for displaying an image of a specific set of images that is selected by the user; and

two navigation buttons adapted to enable a user to cycle images of the specific set of images into view in the image display region;

wherein the user interface system is adapted to cease display of the primary user interface design and begin display of the secondary user interface design upon a user selecting a representative image displayed in the first scrollable region.

23. The user interface system of claim 21, wherein the query entry area is adapted to accept user-provided queries entered using a method chosen from the group consisting of:

typing the query into the query entry area, speaking the query into a microphone, and indicating the query by selecting items from a checklist.

24. The user interface system of claim 21, wherein the user interface system is adapted to enable a user to refine the expanded set of keywords used to search the image descriptors by modifying the expanded set of keywords as displayed in the search term area.

25. The user interface system of claim 24, wherein the expanded set of keywords is displayed as a set of checkboxes and the interface system is adapted to enable a user to refine the expanded set of keywords by deselecting one or more checkboxes.