System and method for recognizing objects from images and identifying relevancy amongst images and information

• US 20060251292A1
• Filed: 10/07/2005
• Published: 11/09/2006
• Est. Priority Date: 05/09/2005
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Like.Com (Alphabet Inc.)
• Inventors: Vanchoucke, Vincent, Shah, Munjal, Lee, Kuang-Chih, Gokturk, Salih Burak, Vu, Diem, Anguelov, Dragomir, Khan, Azhar, Yang, Danny
• Granted Patent: US 7,809,192 B2
• Time in Patent Office: Days
• US Class Current: 382/103
• CPC Class Codes: G06F 16/58 Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
• CPC Class Codes: G06F 16/5846 using extracted text
• CPC Class Codes: G06V 40/107 Static hand or arm

Abstract

An embodiment provides for enabling retrieval of a collection of captured images that form at least a portion of a library of images. For each image in the collection, a captured image may be analyzed to recognize information from image data contained in the captured image, and an index may be generated, where the index data is based on the recognized information. Using the index, functionality such as search and retrieval is enabled. Various recognition techniques, including those that use the face, clothing, apparel, and combinations of characteristics may be utilized. Recognition may be performed on, among other things, persons and text carried on objects.

Claims

1. A method for determining information about captured images, the method comprising:

for each of one or more captured images in the collection,

detecting a person in the captured image, and

determining recognition information from at least one of clothing or apparel on the person.

2. The method of claim 1, further comprising generating a recognition signature for the person based at least in part on the identified information.

3. The method of claim 2, wherein the recognition signature is used to identify the person and the captured image in a searchable index.

4. The method of claim 1, further comprising indexing the recognition information in reference to at least the captured image.

5. The method of claim 2, wherein the recognition signature is used as a search criteria to identify other captured images in the collection that have a corresponding recognition signature that is sufficiently similar to the search criteria to indicate a match.

6. The method of claim 2, wherein detecting a person in the captured image includes detecting a face of the person.

7. The method of claim 6, wherein generating a recognition signature based at least in part on the identified information includes generating the recognition signature based at least in part on the face of the person.

8. The method of claim 6, wherein detecting a person in the captured image includes determining one or more characteristics of the person selected from a group consisting of (i) a hair color of the person, (ii) a hair pattern of the person, (iii) a gender of the person, (iv) a skin color of the person.

9. The method of claim 8, wherein generating a recognition signature is based at least in part on the one or more characteristics selected from the group.

10. The method of claim 1, wherein determining the recognition information includes determining a color present in the captured image at a location determined to be a possible location for the clothing or apparel.

11. The method of claim 10, wherein the location is determined based on a pre-designated relationship with a feature identified from the detected person.

12. A method for determining information about captured images, the method comprising:

identifying a cluster of images from a collection of captured images based on a common characteristic shared by the cluster of images;

determining a recognition signature of a given person appearing in one of the cluster of images; and

using the determined recognition signature in determining a recognition signature of one or more persons appearing in other images in the cluster of images.

13. The method of claim 12, wherein identifying a cluster of images includes identifying each image in the cluster from a time in which the image was captured.

14. The method of claim 12, wherein identifying a cluster of images includes identifying each image in the cluster from a location of which the image was captured.

15. The method of claim 12, wherein determining a recognition signature of a given person includes determining the recognition signature based on a recognition of a face of the person.

16. The method of claim 12, wherein determining a recognition signature of a given person includes determining the recognition signature based on a recognition of a clothing of the person.

17. A method for organizing a collection of captured images, the method comprising:

identifying an object carrying a text in one or more image from the collection;

recognizing the text for each of the one or more images; and

organizing the images, based at least in part, on the recognized text from each of the one or more images.

18. The method of claim 17, wherein organizing the images includes determining, for a first image in the one or more images, that that the text indicates information about at least one of (i) a location of where the first image was captured, (ii) a time of when the first image was captured, or (iii) an event relating to the first image being captured.

19. The method of claim 18, further comprising identifying other images in the collection that are related to the first image having the object, the identified other images and the first image forming a set.

20. The method of claim 19, further comprising associating the set of images with the identified information.

21. The method of claim 20, further comprising retrieving at least one or more images in the set in response to an input that indicates the information.

22. The method of claim 20, wherein identifying other images in the collection include identifying other images taken within a designated duration of when the first image was captured.

23. The method of claim 20, further comprising determining a recognition signature of one or more other objects that appear with the detected object.

24. The method of claim 23, wherein the one or more other objects correspond to at least a portion of a person.

25. The method of claim 24, wherein determining the recognition signature of one or more other objects includes determining the recognition signature based at least in part on a face of the person.

26. The method of claim 24, wherein determining the recognition signature of one or more other objects includes determining the recognition signature based on a clothing or apparel of the person.

27. A method for performing recognition on a captured image, the method comprising:

analyzing image data from the captured image to detect a face of a person in the image;

normalizing a portion of the image data corresponding to a location in which the face appears; and

generating a recognition signature for the person using the normalized portion of the image.

28. The method of claim 27, wherein normalizing a portion of the image includes generating image data of the face in a pre-determined orientation.

29. The method of claim 27, wherein normalizing a portion of the image includes adjusting image data of the face to have a normalized lighting.

30. The method of claim 27, wherein normalizing a portion of the image includes scaling image data of the face with a pre-determined size.

38. The system of claim 27, wherein the text data represents at least one of (i) identities of persons recognized by the image analysis module, and (ii) text recognized from objects.

31. A method for performing recognition on a captured image, the method comprising:

analyzing image data from at least the captured image to detect a person in the image;

determining recognition information from a face of the person;

determining additional recognition information from performing at least one of (i) determining recognition information of a clothing or apparel worn by the person, (ii) determining a time in which the image was captured, (iii) identifying a location or region in which the image was captured; and

determining a recognition signature of the person based on the recognition information from the face and the additional recognition information.

32. The method of claim 31, wherein determining recognition information from the face includes normalizing a portion of the image data corresponding to a location in which a face of the person appears.

33. The method of claim 31, wherein determining additional recognition information includes using a recognition signature from one or more other images in a set of images that are determined to be relevant.

34. The method of claim 33, further comprising programmatically determining the set of images that are determined to be relevant using one or more of (i) a time in which each image in the set was captured, (ii) a location in which each image in the set was captured, and (iii) one or more other persons recognized in one or more images in the set.

35. A system for performing recognition on captured images, the system comprising:

an image analysis module configured to analyze image data from at a set of captured images in order to generate recognition information for (i) persons that appear in images in the set of captured images, and (ii) text carried on objects that appear in images in the set of images.

36. The system of claim 35, further comprising an index that stores the recognition information.

37. The system of claim 36, wherein the recognition information includes text data that is stored in the index.

39. The system of claim 36, wherein the recognition information includes quantitative data that is stored in the index, wherein the quantitative data represents recognition signatures of persons recognized by the image analysis module.

40. The system of claim 35, further comprising a search module that is configured to perform a search operation based on a user-specified criteria using the index.

41. The system of claim 36, further comprising a search module that is configured to perform a search operation based on a user-specified criteria, and wherein the user-specified criteria originates from an image, and is compared against quantitative information stored in the index.

42. The system of claim 41, wherein the user-specified criteria is based on an image specified by the user, and wherein the image analysis module generates a recognition signature for the specified image of the user, and wherein the search module identifies a result by comparing the recognition signature to other recognition signatures stored in the index.

43. The system of claim 42, wherein the search module identifies the result by identifying recognition signatures that are similar but not matching to the recognition signature for the image specified by the user.

44. The system of claim 42, wherein the search module identifies the result by identifying recognition signatures that are matching to the recognition signature for the image specified by the user.

45. A system for performing recognition on captured images, the system comprising:

an image analysis module configured to analyze image data from at a set of captured images in order to generate recognition information for persons that appear in images in the set of captured images; and

a content and data inference module that enhances the recognition information generated for a given image in the set by recognition information from other images in the set.

46. The system of claim 45, wherein the content and data inference module enhances the recognition information generated for the given image using recognition information from other images in the set that are about one or more persons not appearing in the given image.

47. The system of claim 45, wherein the content and data inference module enhances the recognition information generated for the given image using statistical calculations.

48. The system of claim 45, wherein the content and data inference module enhances the recognition information generated for the given image by determining a frequency in which a given person appears in images in the set.

49. The system of claim 45, wherein for a given image comprising two or more individuals, the content and analysis module is configured to identify two or more persons who stand close together in one or more other images in the set.

50. The system of claim 45, wherein the image analysis module generates recognition information corresponding to a clothing worn by persons that appear in the set of images, and wherein the content and data inference module enhances the recognition information generated for the given image by identifying recognition information corresponding to clothing of persons that appear in one or more other images in the set.

51. The system of claim 45, wherein at least one of the image analysis module and the content and data inference module are configured to identify the set of images based by identifying the images being part of an event designated by at least one of a time period or a geographic location.