Digital image retrieval by aggregating search results based on visual annotations

• US 7,962,500 B2
• Filed: 10/24/2008
• Issued: 06/14/2011
• Est. Priority Date: 10/24/2008
• Status: Active Grant

• Current Assignee: R2 Solutions LLC (Acacia Research Corporation)
• Original Assignee: Yahoo! Inc. (Apollo Global Management, Inc.)
• Inventors: Olivares, Ximena, van Zwol, Roelof
• Primary Examiner(s): Lu; Kuen S
• Assistant Examiner(s): Pham; Tuan
• Application Number:
  US12/258,349
• Publication Number:
  US 20100114888A1
• Time in Patent Office: 963 Days
• Field of Search: None
• US Class Current: 707/758
• CPC Class Codes: G06F 16/58 Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually

Abstract

An approach for responding to a text-based query for a digital image is provided. A request that identifies one or more keywords is received. A number of annotated digital images are selected. Each selected annotated digital image has a bounded region, on its appearance, that has an annotation associated with at least one of the keywords. A set of candidate digital images is selected for each annotated digital image. The set of candidate images, for a particular annotated digital image, are the digital images, of a set of digital images, which have an appearance that is most similar to the particular annotated digital image. The sets of candidate images are aggregated into a single set of digital images. A response is generated that identifies those digital images in the single set of digital images which are most responsive to the one or more keywords.

Claims

1. A method, comprising:

receiving a request for a digital image, wherein said request identifies one or more keywords;

selecting a set of annotated digital images, wherein each member of said set of annotated digital images has a bounded region associated therewith, wherein each bounded region has been assigned an annotation that is associated with at least one of said one or more keywords, and wherein each bounded region identifies a portion of the appearance of the annotated digital image to which the bounded region is associated;

upon determining that a certain annotated digital image of said set of annotated digital images is dissimilar in appearance to other annotated digital images of said set of annotated digital images, removing said certain annotated digital image from said set of annotated digital images;

for each annotated digital image of said set of annotated digital images, selecting, from a set of digital images, a set of candidate digital images, wherein digital images in a particular set of candidate digital images are selected for inclusion in said particular set of candidate digital images based on visual characteristics shared by members of said particular set of candidate digital images and a particular annotated digital image, of said set of annotated digital images, to which said particular set of candidate digital images is associated;

aggregating the sets of candidate digital images for said each annotated digital image of said set of annotated digital images into a single set of digital images; and

generating a response to said request, wherein said response identifies a predetermined number of response digital images, wherein said response digital images are those digital images in said single set of digital images which are most responsive to said request;

wherein a human made the association between at least one bounded region and at least one annotated digital image of said set of annotated digital images;

wherein the method is performed by one or more computing devices.

2. The method of claim 1, wherein the number of annotated digital images to be selected in said selecting said set of annotated digital images is determined prior to receiving said request.

3. The method of claim 1, wherein the number of annotated digital images to be selected in said selecting said set of annotated digital images is a configurable number, and wherein the method further comprises:

upon receiving input that identifies a particular number, updating said configurable number to be said particular number.

4. The method of claim 1, wherein said request is sent from a requester, and wherein each annotated digital image of said set of annotated digital images has been annotated by a different person that said requestor.

5. The method of claim 1, wherein said response identifies said predetermined number of response digital images by including each response digital image of said predetermined number of response digital images in said response.

6. The method of claim 1, wherein the step of selecting, from said set of digital images, a set of candidate digital images for said each annotated digital image of said set of annotated digital images further comprises:

ranking said each set of candidate images in order of perceived responsiveness to said request.

7. The method of claim 1, wherein said response identifies a ranking for each response digital image of said predetermined number of response digital images in order of perceived responsiveness to said request.

8. The method of claim 1, wherein said request is sent from a requester, and wherein the method further comprises:

sending, to said requester, said response.

9. The method of claim 1, wherein said visual characteristics that form a basis for inclusion in said particular set of candidate digital images includes a similarity in appearance between the bounded region identified on the visual appearance of each member of said particular set of candidate digital images and said particular annotated digital image.

10. A non-transitory machine-readable medium storing one or more sequences of instructions which when executed cause:

receiving a request for a digital image, wherein said request identifies one or more keywords;

selecting a set of annotated digital images, wherein each member of said set of annotated digital images has a bounded region associated therewith, wherein each bounded region has been assigned an annotation that is associated with at least one of said one or more keywords, and wherein each bounded region identifies a portion of the appearance of the annotated digital image to which the bounded region is associated;

upon determining that a certain annotated digital image of said set of annotated digital images is dissimilar in appearance to other annotated digital images of said set of annotated digital images, removing said certain annotated digital image from said set of annotated digital images;

for each annotated digital image of said set of annotated digital images, selecting, from a set of digital images, a set of candidate digital images, wherein digital images in a particular set of candidate digital images are selected for inclusion in said particular set of candidate digital images based on visual characteristics shared by members of said particular set of candidate digital images and a particular annotated digital image, of said set of annotated digital images, to which said particular set of candidate digital images is associated;

aggregating the sets of candidate digital images for said each annotated digital image of said set of annotated digital images into a single set of digital images; and

generating a response to said request, wherein said response identifies a predetermined number of response digital images, wherein said response digital images are those digital images in said single set of digital images which are most responsive to said request;

wherein a human made the association between at least one bounded region and at least one annotated digital image of said set of annotated digital images.

11. The machine-readable medium of claim 10, wherein the number of annotated digital images to be selected in said selecting said set of annotated digital images is determined prior to receiving said request.

12. The machine-readable medium of claim 10, wherein the number of annotated digital images to be selected in said selecting said set of annotated digital images is a configurable number, and wherein execution of said one or more sequences of instructions further causes:

upon receiving input that identifies a particular number, updating said configurable number to be said particular number.

13. The machine-readable medium of claim 10, wherein said request is sent from a requester, and wherein each annotated digital image of said set of annotated digital images has been annotated by a different person that said requestor.

14. The machine-readable medium of claim 10, wherein said response identifies said predetermined number of response digital images by including each response digital image of said predetermined number of response digital images in said response.

15. The machine-readable medium of claim 10, wherein the step of selecting, from said set of digital images, a set of candidate digital images for said each annotated digital image of said set of annotated digital images further comprises:

ranking said each set of candidate images in order of perceived responsiveness to said request.

16. The machine-readable medium of claim 10, wherein said response identifies a ranking for each response digital image of said predetermined number of response digital images in order of perceived responsiveness to said request.

17. The machine-readable medium of claim 10, wherein said request is sent from a requester, and wherein execution of said one or more sequences of instructions further causes:

sending, to said requester, said response.

18. The machine-readable medium of claim 10, wherein said visual characteristics that form a basis for inclusion in said particular set of candidate digital images includes a similarity in appearance between the bounded region identified on the visual appearance of each member of said particular set of candidate digital images and said particular annotated digital image.