System and method for recognizing objects from images and identifying relevancy amongst images and information

• US 7,809,192 B2
• Filed: 10/07/2005
• Issued: 10/05/2010
• Est. Priority Date: 05/09/2005
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Like.Com (Alphabet Inc.)
• Inventors: Shah, Munjal, Lee, Kuang-Chih, Gokturk, Salih Burak, Vu, Diem, Anguelov, Dragomir, Vanhoucke, Vincent, Khan, Azhar, Yang, Danny
• Primary Examiner(s): Strege; John B
• Application Number:
  US11/246,589
• Publication Number:
  US 20060251292A1
• Time in Patent Office: 1,824 Days
• Field of Search: 382/100, 382/176, 382/190, 382/209, 382/218, 382/224, 382/278, 382/282, 382/289, 382/290
• US Class Current: 382/176
• CPC Class Codes: G06F 16/58 Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
• CPC Class Codes: G06F 16/5846 using extracted text
• CPC Class Codes: G06V 40/107 Static hand or arm

Abstract

An embodiment provides for enabling retrieval of a collection of captured images that form at least a portion of a library of images. For each image in the collection, a captured image may be analyzed to recognize information from image data contained in the captured image, and an index may be generated, where the index data is based on the recognized information. Using the index, functionality such as search and retrieval is enabled. Various recognition techniques, including those that use the face, clothing, apparel, and combinations of characteristics may be utilized. Recognition may be performed on, among other things, persons and text carried on objects.

Claims

1. A method for analyzing a collection of captured images, the method comprising:

using one or more processors to perform steps comprising;

(a) performing analysis on each image in the collection to determine analysis information that is associated with one or more images from the collection;

(b) maintaining a data structure that associates the one or more individual images from the collection with the analysis information that is associated with that image;

(c) identifying an object carrying a text in the one or more images from the collection;

(d) making a determination that the identified text comprises only a portion of a word, including determining that the identified text is one of (i) a word that is in skew on the object of the image, or (ii) a word that appears to omit one or more letters in the image; and

(e) enabling a search of the data structure using a word from the identified text as search criteria for individual images in the collection that are associated with analysis information that match or are relevant to the determined word.

2. The method of claim 1, wherein enabling a search of the data structure using a word from the identified text comprises determining, for a first image in the one or more images, that the text indicates information about at least one of (i) a location of where the first image was captured, (ii) a time of when the first image was captured, or (iii) an event relating to the first image being captured.

3. The method of claim 2, further comprising identifying other images in the collection that are related to the first image having the object, the identified other images and the first image forming a set.

4. The method of claim 3, further comprising associating the set of images with the information indicated by the text.

5. The method of claim 4, further comprising retrieving at least one or more images in the set in response to an input that includes the information indicated by the text.

6. The method of claim 4, wherein identifying other images in the collection includes identifying other images taken within a designated duration of when the first image was captured.

7. The method of claim 4, further comprising determining a recognition signature of one or more other objects that appear with the detected object.

8. The method of claim 7, wherein the one or more other objects correspond to at least a portion of a person.

9. The method of claim 8, wherein determining the recognition signature of one or more other objects includes determining the recognition signature based at least in part on a face of the person.

10. The method of claim 8, wherein determining the recognition signature of one or more other objects includes determining the recognition signature based on a clothing or apparel of the person.

11. A system for performing recognition on captured images, the system comprising:

a data structure for associating individual images in a set of captured images with analysis information that is associated with one or more individual images from the set of captured images;

an image analysis module configured to perform analysis on each image in the set of captured images in order to (i) generate analysis information for a person or an object that appears in one or more images in the set, and (ii) detect text carried on an object that appears in skew in the one or more images in the set;

wherein the image analysis module (i) analyzes the image data to determine a word that corresponds to the text that is carried on the text and appears in skew, and (ii) enables a search of the data structure using the word as search criteria for individual images in the set of captured images that are associated with analysis information that matches or are relevant to the determined word.

12. The system of claim 11, wherein the data structure is an index that stores the recognition information.

13. The system of claim 12, wherein the recognition information includes text data that is stored in the index.

14. The system of claim 13, wherein the text data represents at least one of (i) identities of persons recognized by the image analysis module, and (ii) text recognized from objects.

15. The system of claim 12, wherein the recognition information includes quantitative data that is stored in the index, wherein the quantitative data represents recognition signatures of persons recognized by the image analysis module.

16. The system of claim 11, further comprising a search module that is configured to perform a search operation based on a user-specified criteria using the data structure.

17. The system of claim 12, further comprising a search module that is configured to perform a search operation based on a user-specified criteria, and wherein the user-specified criteria originates from an image, and is compared against quantitative information stored in the index.

18. The system of claim 17, wherein the user-specified criteria is based on an image specified by the user, and wherein the image analysis module generates a recognition signature for the specified image of the user, and wherein the search module identifies a result by comparing the recognition signature to other recognition signatures stored in the index.

19. The system of claim 18, wherein the search module identifies the result by identifying recognition signatures that are similar but not matching to the recognition signature for the image specified by the user.

20. The system of claim 18, wherein the search module identifies the result by identifying recognition signatures that are matching to the recognition signature for the image specified by the user.

21. A method for organizing a collection of captured images, the method comprising:

using one or more processors to perform steps comprising;

(a) performing image analysis on each image in the collection to determine analysis information that is associated with one or more individual images from the collection, wherein the image analysis includes identifying, from the one or more of the individual images, an object carrying a text that is in skew;

(b) maintaining a data structure that associates the one or more individual images from the collection with the analysis information that is associated with each individual image;

(c) determining a word that corresponds to the text that is in skew and carried on the object;

(d) making a determination that the word is relevant to the data structure; and

(e) enabling a search of the data structure using a word from the identified text as search criteria for individual images in the collection that are associated with analysis information that match or are relevant to the determined word.