Image object retrieval based on aggregation of visual annotations

• US 8,527,564 B2
• Filed: 12/16/2010
• Issued: 09/03/2013
• Est. Priority Date: 12/16/2010
• Status: Active Grant

• Current Assignee: R2 Solutions LLC (Acacia Research Corporation)
• Original Assignee: Yahoo! Inc. (Apollo Global Management, Inc.)
• Inventors: Olivares, Ximena, Van Zwol, Roelof
• Primary Examiner(s): AL HASHEMI, SANA A
• Application Number:
  US12/970,844
• Publication Number:
  US 20120158716A1
• Time in Patent Office: 992 Days
• Field of Search: 707/769, 707/803, 707/758, 707/722, 707/953
• US Class Current: 707/953
• CPC Class Codes: G06F 16/58 Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
• CPC Class Codes: G06F 16/583 using metadata automatically derived from the content

Abstract

An approach for responding to a text-based query for a digital image is provided. A request that identifies one or more keywords is received. A number of annotated digital images are selected based on a previously determined optimum quantity of annotated digital images. Composite data is gathered from each annotated digital image and a set of candidate digital images is selected based on the composite data. The set of candidate images are the digital images, of a set of digital images, which have a visual appearance that is most similar to the composite data. A response is generated that identifies those digital images which are most responsive to the one or more keywords. Alternatively, a partitioned response is generated which identifies dissimilar sets of digital images.

Claims

1. A computer-implemented method comprising steps of:

receiving a query, wherein the query identifies a set of one or more query terms;

in response to receiving the query that identifies the set of one or more query terms, performing the steps of;

selecting, from an image corpus, a set of three or more sample images that are associated with annotations that match one or more of the query terms;

producing multiple separate sets of candidate images by selecting from the image corpus, for each particular sample image in the set of sample images, a separate set of candidate images that are visually similar to that particular sample image;

from among images that belong to the multiple separate sets of candidate images, determining a set of duplicate images;

wherein each image in the set of duplicate images is a member of at least two of the multiple separate sets of candidate images; and

generating a response to the query, wherein the response is based, at least in part, on the set of duplicate images;

wherein the steps are performed by at least one computing device.

2. The method of claim 1, wherein the multiple separate sets of candidate images include a first set of candidate images, a second set of candidate images, and a third set of candidate images, and wherein:

the step of determining the set of duplicate images comprises;

determining that the first set of candidate images and the second set of candidate images both contain at least a threshold number of candidate images in common;

determining that the third set of candidate images contains less than the threshold number of candidate images in common with the first and second sets of candidate images; and

the method further comprises, in response to determining that the third set of candidate images contains less than the threshold number of candidate images in common with the first and second sets of candidate images, generating an aggregated set of images based on the first and second sets of candidate images;

wherein the response includes at least a subset of the images in the aggregated set of images.

3. The method of claim 2, further comprising associating a rank with each image contained in the first, second, and third sets of candidate images, wherein the determination that the third set of candidate images contain less than the threshold number of candidate images in common with the first and second sets of candidate images is based, at least in part, on the rank.

4. The method of claim 2, wherein the aggregated set of images is generated based on the first, second, and third sets of candidate images, and wherein the response is a partitioned response to the query, wherein the partitioned response identifies the third set of candidate images as distinguishable from the first and second sets of candidate images.

5. A non-transitory computer-readable medium storing one or more sequences of instructions, wherein processing of the one or more sequences of instructions by one or more processors causes performance of the method recited in claim 1.

6. A non-transitory computer-readable medium storing one or more sequences of instructions, wherein processing of the one or more sequences of instructions by one or more processors causes performance of the method recited in claim 2.

7. A computer-implemented method comprising steps of:

receiving a set of one or more query terms;

selecting, from an image corpus, a set of sample images that are associated with annotations that match one or more of the query terms;

based on the sample images in the set of sample images, generating composite data that represents at least one duplicate visual characteristic;

wherein the at least one duplicate visual characteristic is a visual characteristic that is possessed by multiple sample images in the set of sample images;

determining a set of candidate images that possess visual characteristics that are similar to the at least one visual characteristic represented by the composite data; and

returning at least a subset of candidate images, from the set of candidate images, as results of a query that contained the set of query terms;

wherein the steps are performed by at least one computing device.

8. The method of claim 7, wherein the results are generated without determining whether any image in the image corpus is visually similar to any one of the sample images in the set of sample images.

9. The method of claim 7, wherein a number of sample images in the set of sample images to be selected is determined prior to receiving the set of one or more query terms.

10. The method of claim 7, wherein generating composite data further comprises associating the at least one visual characteristic with a value, wherein the value represents a number of occurrences of the at least one visual characteristic within the set of sample images.

11. The method of claim 10, wherein the method further comprises storing the composite data in association with the set of one or more query terms.

12. The method of claim 10, further comprising:

filtering the composite data to remove at least one visual characteristic associated with a value less which is less than a threshold value.

13. A non-transitory computer-readable medium storing one or more sequences of instructions, wherein processing of the one or more sequences of instructions by one or more processors causes performance of the method recited in claim 7.

14. A non-transitory computer-readable medium storing one or more sequences of instructions, wherein processing of the one or more sequences of instructions by one or more processors causes performance of the method recited in claim 9.

15. A computer-implemented method comprising steps of:

for each particular sample image of a first set of sample images, selecting, from an image corpus, a separate set of candidate images that are visually similar to that particular sample image, thereby producing a first plurality of sets of candidate images;

aggregating the first plurality of sets of candidate images to produce a first aggregated list of images;

determining a first precision score that is based on a measure of relevance of images in the first aggregated list of images to a specified set of query terms;

for each particular sample image of a second set of sample images, selecting, from the image corpus, a separate set of candidate images that are visually similar to that particular sample image, thereby producing a second plurality of sets of candidate images;

aggregating the second plurality of sets of candidate images to produce a second aggregated list of images;

determining a second precision score that is based on a measure of relevance of images in the second aggregated list of images to the specified set of query terms;

wherein a quantity of sample images in the second set of sample images is greater than a quantity of sample images in the first set of sample images; and

based at least in part on the first precision score and the second precision score, determining a quantity of sample images to be used in performing future searches for images in the image corpus;

wherein the steps are performed by at least one computing device.

16. The method of claim 15, wherein the measure of relevance is based, at least in part, on dividing a quantity of relevant images in a set of images by a quantity of at least a subset of images in a set of images.

17. The method of claim 15, wherein the method further comprises:

prior to determining the first and second precision scores, ranking the first and second aggregated list of images to produce a first and second ranked list of images, wherein each measure of relevance for the first and second precision scores is based on each respective first and second ranked list of images.

18. The method of claim 15, wherein the quantity of sample images to be used in performing future searches is selected based, at least in part, on determining whether a difference value between the first precision score and the second precision score is below a threshold difference value.

19. A non-transitory computer-readable medium storing one or more sequences of instructions, wherein processing of the one or more sequences of instructions by one or more processors causes performance of the method recited in claim 15.

20. A non-transitory computer-readable medium storing one or more sequences of instructions, wherein processing of the one or more sequences of instructions by one or more processors causes performance of the method recited in claim 16.