Any-image labeling engine

• US 9,037,600 B1
• Filed: 01/28/2011
• Issued: 05/19/2015
• Est. Priority Date: 01/28/2011
• Status: Active Grant

• Current Assignee: Verizon Patent and Licensing Incorporated (Verizon Communications Inc.)
• Original Assignee: Yahoo! Inc. (Apollo Global Management, Inc.)
• Inventors: Culpepper, Benjamin Jackson, Nguyen, Huy X., Pesavento, Gerry, Garrigues, Pierre, Cadieu, Charles, Koepsell, Killian, Stadlen, Andrew, Warland, David
• Primary Examiner(s): Kuddus, Daniel
• Application Number:
  US13/016,776
• Time in Patent Office: 1,572 Days
• Field of Search: 707/772, 707/769, 707/999.107, 707/915
• US Class Current: 707/769
• CPC Class Codes: G06F 16/50 of still image data
• CPC Class Codes: G06F 16/58 Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
• CPC Class Codes: G06F 16/5866 using information manually generated, e.g. tags, keywords, comments, manually generated location and time information

Abstract

A system for tagging an image comprises a processor and a memory. The processor is configured to analyze an image associated with an image query using one or more computer vision analysis types to determine zero or more computer vision matches. Each computer vision match has one or more associated computer vision tags. In the event that it is determined that there are zero computer vision matches, the processor is further configured to analyze the image associated with the image query using a human vision analysis system to determine zero or more human vision matches. Each human vision match has one or more associated human vision tags. A memory coupled to the processor and configured to provide the processor with instructions.

Claims

1. A system, comprising:

a processor; and

a memory, at least one of the processor or the memory being configured to;

analyze an image associated with an image query using one or more computer vision analysis types to determine computer vision matches, wherein each computer vision match includes one or more computer vision tags associated with the image;

determine whether any computer vision matches for the analyzed image have been determined;

in the event that one or more computer vision matches have been determined, add the image associated with the image query to one or more databases and for each of the one or more computer vision matches associating the image with the one or more associated computer vision tags; and

in the event that there are no computer vision matches or the computer vision matches are determined to be inadequate, analyze the image associated with the image query using a human vision analysis system to determine one or more human vision matches, wherein each human vision match includes one or more human vision tags associated with the image;

wherein a computer vision match is determined to be inadequate when the computer vision match lacks specificity or a probability associated with the computer vision match indicates that the computer vision match is not a strong match.

2. The system as recited in claim 1, at least one of the processor or the memory being further configured to receive the image query.

3. The system as recited in claim 1, at least one of the processor or the memory being further configured to determine one or more computer vision analysis types for the image query.

4. The system as recited in claim 1, at least one of the processor or the memory being further configured to:

in the event that there are no human vision matches, indicate that there are no matches identified.

5. The system as recited in claim 1, at least one of the processor or the memory being further configured to select from the one or more associated computer vision tags or the one or more associated human vision tags, one or more tag results for the image query.

6. The system as recited in claim 5, at least one of the processor or the memory being further configured to provide the one or more tag results.

7. The system as recited in claim 1, at least one of the processor or the memory being further configured to, in the event that there are one or more human vision matches, add the image associated with the image query to one or more databases and for each of the one or more human vision matches associating the image with the one or more human vision tags.

8. The system as recited in claim 1, wherein the human vision analysis system is configured to verify the computer vision tags associated with each of the computer vision matches.

9. The system as recited in claim 8, at least one of the processor or the memory being further configured to provide at least one of the human vision tags.

10. The system as recited in claim 9, wherein the at least one of the human vision tags comprises a set of verified computer vision tags, the set of verified computer vision tags comprising at least one of the computer vision tags.

11. The system as recited in claim 1, wherein the human vision analysis system is configured to communicate with one or more human taggers.

12. The system as recited in claim 11, wherein the human taggers comprise expert human taggers.

13. The system as recited in claim 11, further comprising:

a dashboard system configured to provide information pertaining to each of the human taggers, wherein the information indicates a performance of the corresponding one of the human taggers.

14. The system as recited in claim 11, at least one of the processor or the memory being further configured to:

provide the image and the human vision tags to the human taggers.

15. The system as recited in claim 11, at least one of the processor or the memory being further configured to:

assign a tagger status to one of the human taggers based upon a validity of human vision tags submitted by the corresponding one of the human taggers.

16. The system as recited in claim 11, at least one of the processor or the memory being further configured to:

provide a user interface to the human taggers, the user interface including a tagging guideline that includes one or more tag entry boxes and corresponding indicators specifying specific information requested for the image.

17. A method for tagging an image, comprising:

analyzing an image associated with an image query using one or more computer vision analysis types to determine computer vision matches, wherein each computer vision match has one or more associated computer vision tags;

determining whether any computer vision matches for the analyzed image have been determined;

in the event that one or more computer vision matches have been determined, add the image associated with the image query to one or more databases and for each of the one or more computer vision matches associating the image with the one or more associated computer vision tags; and

in the event that there are no computer vision matches or the computer vision matches are determined to be inadequate, analyze the image associated with the image query using a human vision analysis system to determine one or more human vision matches, wherein each human vision match includes one or more human vision tags associated with the image.

18. A non-transitory computer-readable storage medium storing thereon computer-readable instructions, comprising:

instructions for analyzing an image associated with an image query using one or more computer vision analysis types to determine computer vision matches, wherein each computer vision match has one or more associated computer vision tags;

instructions for determining whether any computer vision matches for the analyzed image have been determined;

instructions for, in the event that one or more computer vision matches are determined, adding the image associated with the image query to one or more databases and for each of the one or more computer vision matches associating the image with the one or more associated computer vision tags; and

instructions for, in the event that there are no computer vision matches or the computer vision matches are determined to be inadequate, analyzing the image associated with the image query using a human vision analysis system based on the context information to determine one or more human vision matches, wherein each human vision match includes one or more human vision tags associated with the image;

wherein a computer vision match is determined to be inadequate when the computer vision match lacks specificity or a probability associated with the computer vision match indicates that the computer vision match is not a strong match.

19. The non-transitory computer-readable storage medium as recited in claim 18, further comprising:

instructions for determining whether the computer vision matches are specific enough, wherein determining whether the computer vision matches are specific enough is performed based, at least in part, on context information, wherein the context information comprises one or more of the following;

information from an originating web site, information from an originating web service, associated text information, a specific user request, a specific request from a site submitting the image, metadata information from a service, information about a user submitting the image query, or geolocation.

20. The non-transitory computer-readable storage medium as in claim 18, further comprising instructions for determining one or more computer vision analysis types for the image query.

21. The non-transitory computer-readable storage medium as recited in claim 18, further comprising instructions for selecting from the one or more associated computer vision tags or the one or more associated human vision tags, one or more tag results for the image query.

22. The non-transitory computer-readable storage medium as recited in claim 21, further comprising instructions for providing the one or more tag results.

23. The non-transitory computer-readable storage medium as recited in claim 18, further comprising instructions for, in the event that there are one or more human vision matches, adding the image associated with the image query to one or more databases and for each of the one or more human vision matches associating the image with the one or more associated human vision tags.

24. A method, comprising:

analyzing an image associated with an image query using one or more computer vision analysis types to determine one or more computer vision matches, wherein each computer vision match has one or more associated computer vision tags;

determining, based at least in part on context information, whether the computer vision matches for the analyzed image are sufficiently specific; and

in the event that the computer vision matches are determined to lack specificity based at least in part on the context information, analyze the image associated with the image query using a human vision analysis system based on the context information to determine one or more human vision matches, wherein each human vision match includes one or more human vision tags associated with the image, wherein the human vision analysis system is configured to communicate with one or more human taggers, wherein a user interface is provided to the human taggers, the user interface including a tagging guideline that includes one or more tag entry boxes and corresponding indicators specifying specific information requested for the image.