System and method for enabling the use of captured images through recognition

• US 20060251339A1
• Filed: 10/07/2005
• Published: 11/09/2006
• Est. Priority Date: 05/09/2005
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Like.Com (Alphabet Inc.)
• Inventors: Shah, Munjal, Lee, Kuang-Chih, Gokturk, Salih Burak, Vu, Diem, Anguelov, Dragomir, Vanhoucke, Vincent, Khan, Azhar, Yang, Danny
• Granted Patent: US 7,519,200 B2
• Time in Patent Office: Days
• US Class Current: 382/305
• CPC Class Codes: G06F 16/50 of still image data
• CPC Class Codes: G06F 16/53 Querying
• CPC Class Codes: G06F 16/58 Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
• CPC Class Codes: G06F 16/5846 using extracted text
• CPC Class Codes: G06F 16/587 using geographical or spatial information, e.g. location
• CPC Class Codes: G06V 40/107 Static hand or arm

Abstract

An embodiment provides for enabling retrieval of a collection of captured images that form at least a portion of a library of images. For each image in the collection, a captured image may be analyzed to recognize information from image data contained in the captured image, and an index may be generated, where the index data is based on the recognized information. Using the index, functionality such as search and retrieval is enabled. Various recognition techniques, including those that use the face, clothing, apparel, and combinations of characteristics may be utilized. Recognition may be performed on, among other things, persons and text carried on objects.

Claims

1. A method for enabling retrieval of a collection of captured images that form at least a portion of a library of images, the method comprising:

for each image in the collection, performing steps comprising;

analyzing a captured image to recognize information from image data contained in the captured image; and

generating an index containing index data that is based on the recognized information.

2. The method of claim 1, further comprising enabling the index to be used for searching images in the collection using input.

3. The method of claim 2, wherein enabling the index to be searched includes enabling the index to be searched using input that corresponds to a portion of an image.

4. The method of claim 1, wherein analyzing the captured image includes detecting a person in the image.

5. The method of claim 4, wherein analyzing the captured image includes:

detecting a face of the person, and

generating a recognition signature that identifies the person from other persons recognized in the collection based at least in part on the face of the person.

6. The method of claim 5, wherein analyzing the captured image includes:

detecting one or more characteristics selected from a group consisting of (i) an apparel of the person, (ii) a clothing item of the person, (iii) a time when the image was captured, (iv) recognizing another person in the captured image, and

wherein generating the recognition signature includes generating the recognition signature based at least in part on the detected one or more characteristics.

7. The method of claim 1, wherein analyzing the captured image includes detecting a text image contained in the captured image.

8. The method of claim 7, wherein analyzing the captured image includes recognizing text from the text image, and wherein generating an index includes indexing the recognized text in association with the captured image.

9. The method of claim 8, further comprising identifying one or more text items by extrapolating the recognized text, and wherein generating an index includes indexing the one or more text items extrapolated from the recognized text.

10. The method of claim 9, wherein identifying one or more text items by extrapolating the recognized text includes identifying the one or more text items based on one or more (i) a geographic relationship, (ii) a dictionary, and (iii) a thesaurus.

11. The method of claim 8, wherein analyzing the captured image includes determining a relevance of the text image based on other information determined from the captured image.

12. The method of claim 2, wherein analyzing the captured image includes:

identifying clothing or apparel of the person detected, and

generating a recognition signature that identifies the person from other persons recognized in the collection based at least in part on the.

13. The method of claim 1, further comprising matching one or more captured images from the collection to a given input specified by the user, wherein matching the one or more captured images is performed using the index.

14. A method for enabling retrieval of captured images, the method comprising:

analyzing image data of a captured image to determine one or more identifiers of at least one of (i) the captured image, or (ii) an object within the captured image; and

generating index data for an index that correlates the captured image with the one or more identifiers.

15. The method of claim 14, further comprising enabling the captured image to be retrieved in response to an input that corresponds to the one or more identifiers.

16. The method of claim 14, further comprising:

receiving image data as a search request;

determining one or more search criteria from the image data; and

using the index to identify one or more captured images that can be correlated to the one or more search criteria.

17. The method of claim 16, wherein determining one or more search criteria from the image data includes:

determining a recognition signature of a person contained in the image data of the search request, and

basing the one or more search criteria at least in part on the recognition signature of the person.

18. The method of claim 17, wherein determining a recognition signature of a person contained in the image data of the search request is based at least in part on a face of the person.

19. The method of claim 17, determining a recognition signature of a person contained in the image data is based at least in part on one or more of a clothing or apparel of the person.

20. The method of claim 17, wherein determining an identifier of a person contained in the image data of the search request includes determining one or more of (i) an identity recognition value of a face of the person, (ii) information about apparel of the person;

(iii) information about a clothing of the person, and (iv) a time when the image was captured, and (v) a location of where the image was captured.

21. The method of claim 14, wherein analyzing image data of a captured image to determine one or more identifiers includes determining one or more of (i) a person in the image, (ii) a text in the image, or (iii) an object other than the person in the image.

22. The method of claim 21, wherein the object corresponds to the object selected from a group consisting of (i) a landmark, (ii) an animal, or (iii) a place of business.

23. The method of claim 14, wherein the input corresponds to a selection of a portion of an image.

24. The method of claim 15, wherein the input corresponds to a text input.

25. The method of claim 14, wherein analyzing image data includes detecting a person in the captured image.

26. The method of claim 25, wherein analyzing image data includes generating a recognition signature for the detected person, wherein the recognition signature enables the person to be recognized from other detected persons in a collection that includes the captured image.

27. The method of claim 25, wherein analyzing image data includes identifying a classification of the person in the captured image.

28. The method of claim 27, wherein identifying a classification of the person in the captured image includes determining at least one of a classification selected from a group consisting of:

(i) a race of the person, (ii) a gender of the person, (iii) an age group of the person, (iv) skin pigmentation of the person, (v) a hair color of the person, and (vi) an eye color of the person.

29. A system for enabling retrieval of a collection of captured images that form at least a portion of a library of images, the system comprising:

an image analysis module configured to analyze each image in the collection to determine recognition information about one or more objects contained in the captured image;

an indexer that generates index data, wherein the index data associates the recognition information determined from the one or more objects in each captured image with the captured image that contained the one or more objects.

30. The system of claim 29, wherein the recognition information includes text data, and wherein the system further comprises an index that maintains the text data in association with captured images that contained the one or more objects from which the recognition information is determined from.

31. The system of claim 29, wherein the recognition information includes quantitative data, and wherein the system comprises an index that maintains recognition signatures in association with captured images that contained the one or more objects from which the recognition information is determined from.

32. The system of claim 29, wherein the image analysis module is configured to generate recognition information for persons and for text carried on objects.

33. The system of claim 29, wherein the image analysis module is configured to generate recognition information for persons using one or more facial features of the person.

34. The system of claim 33, wherein the image analysis module is configured to generate a recognition signature for persons appearing in each image in the collection using the one or more features of the person and at least one of (i) a clothing or apparel worn by the person, (ii) metadata associated with each image in the collection from which a person is recognized.

35. A computer-readable medium carrying instructions for enabling retrieval of captured images, wherein the instructions include instructions, that when executed by one-or more processors, cause the one or more processors to perform steps comprising:

analyzing image data of a captured image to determine one or more identifiers of at least one of (i) the captured image, or (ii) an object within the captured image; and

generating index data for an index that correlates the captured image with the one or more identifiers.