System and method for providing objectified image renderings using recognition information from images

• US 9,430,719 B2
• Filed: 09/16/2015
• Issued: 08/30/2016
• Est. Priority Date: 05/09/2005
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Gokturk, Salih Burak, Anguelov, Dragomir, Vanhoucke, Vincent O., Lee, Kuang-chih, Vu, Diem Thanh, Yang, Danny, Shah, Munjal, Khan, Azhar
• Primary Examiner(s): Kassa, Yosef
• Application Number:
  US14/855,826
• Publication Number:
  US 20160125270A1
• Time in Patent Office: 349 Days
• Field of Search: 382/115, 382/170, 382/176, 382/177, 382/180, 382/185, 382/190, 382/209, 358/1.11, 358/1.16, 358/538, 358/435, 358/462
• US Class Current: 1/1
• CPC Class Codes: G06F 16/58 Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
• CPC Class Codes: G06F 16/583 using metadata automatically derived from the content
• CPC Class Codes: G06F 16/5838 using colour
• CPC Class Codes: G06F 16/5846 using extracted text
• CPC Class Codes: G06F 16/5854 using shape and object relationship
• CPC Class Codes: G06F 18/22 Matching criteria, e.g. proximity measures
• CPC Class Codes: G06F 18/23 Clustering techniques
• CPC Class Codes: G06F 18/24 Classification techniques
• CPC Class Codes: G06F 3/0482 Interaction with lists of selectable items, e.g. menus
• CPC Class Codes: G06V 20/20 in augmented reality scenes
• CPC Class Codes: G06V 20/70 Labelling scene content, e.g. deriving syntactic or semantic representations
• CPC Class Codes: G06V 40/107 Static hand or arm
• CPC Class Codes: G06V 40/161 Detection; Localisation; Normalisation
• CPC Class Codes: G06V 40/172 Classification, e.g. identification

Abstract

An embodiment provides for enabling retrieval of a collection of captured images that form at least a portion of a library of images. For each image in the collection, a captured image may be analyzed to recognize information from image data contained in the captured image, and an index may be generated, where the index data is based on the recognized information. Using the index, functionality such as search and retrieval is enabled. Various recognition techniques, including those that use the face, clothing, apparel, and combinations of characteristics may be utilized. Recognition may be performed on, among other things, persons and text carried on objects.

Claims

1. A method, comprising:

identifying a plurality of images each associated with metadata;

determining a group of images from the plurality of images based on the metadata associated with each image, wherein a value of at least one parameter of the metadata associated with each image of the group of images are similar;

for each image of the group of images, detecting one or more faces included in the image;

determining clusters of faces from the detected faces, wherein each cluster of faces includes only detected faces that meet a similarity threshold to each other detected face in the cluster of faces; and

for each cluster of faces, matching the detected faces of the cluster of faces to an identity of a person.

2. The method of claim 1, wherein the parameter is a location-based parameter, and wherein a location value of the location-based parameter of the metadata associated with each image of the group of images are similar.

3. The method of claim 1, wherein the parameter is a time-based parameter, and wherein a time value of the time-based parameter of the metadata associated with each image of the group of images are similar.

4. The method of claim 1, further comprising providing a user interface that identifies a particular cluster of faces of the cluster of faces, wherein matching further comprises receiving input to the user interface assigning correlation information to the detected faces of the particular cluster of faces, the correlation information identifying the identity of the person associated with the detected faces of the particular cluster of faces.

5. The method of claim 1, further comprising providing a user interface that identifies each cluster of faces, wherein matching further comprises receiving, for each cluster of faces, a single user input to the user interface assigning correlation information to the detected faces of the cluster of faces, the correlation information identifying the identity of the person associated with the detected faces of the cluster of faces.

6. The method of claim 1, further comprising providing a user interface that identifies i) each cluster of faces and ii) a plurality of identities, each identity associated with a person associated with a user, wherein matching further comprises receiving input to the user interface assigning, for one or more cluster of faces, an identify of a person from the plurality of identities to the cluster of faces.

7. The method of claim 1, wherein matching further comprises matching, for each cluster of faces, the detected faces of the cluster of faces to a name of the person.

8. A system comprising:

one or more computers and one or more storage devices storing instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform operations comprising;

identifying a plurality of images each associated with metadata;

determining a group of images from the plurality of images based on the metadata associated with each image, wherein a value of at least one parameter of the metadata associated with each image of the group of images are similar;

for each image of the group of images, detecting one or more faces included in the image;

determining clusters of faces from the detected faces, wherein each cluster of faces includes only detected faces that meet a similarity threshold to each other detected face in the cluster of faces; and

for each cluster of faces, matching the detected faces of the cluster of faces to an identity of a person.

9. The system of claim 8, wherein the parameter is a location-based parameter, and wherein a location value of the location-based parameter of the metadata associated with each image of the group of images are similar.

10. The system of claim 8, wherein the parameter is a time-based parameter, and wherein a time value of the time-based parameter of the metadata associated with each image of the group of images are similar.

11. The system of claim 8, the operations further comprising providing a user interface that identifies a particular cluster of faces of the cluster of faces, wherein matching further comprises receiving input to the user interface assigning correlation information to the detected faces of the particular cluster of faces, the correlation information identifying the identity of the person associated with the detected faces of the particular cluster of faces.

12. The system of claim 8, the operations further comprising providing a user interface that identifies each cluster of faces, wherein matching further comprises receiving, for each cluster of faces, a single user input to the user interface assigning correlation information to the detected faces of the cluster of faces, the correlation information identifying the identity of the person associated with the detected faces of the cluster of faces.

13. The system of claim 8, the operations further comprising providing a user interface that identifies i) each cluster of faces and ii) a plurality of identities, each identity associated with a person associated with a user, wherein matching further comprises receiving input to the user interface assigning, for one or more cluster of faces, an identify of a person from the plurality of identities to the cluster of faces.

14. The system of claim 8, wherein matching further comprises matching, for each cluster of faces, the detected faces of the cluster of faces to a name of the person.

15. A non-transitory computer-readable medium storing software comprising instructions executable by one or more computers which, upon such execution, cause the one or more computers to perform operations comprising:

identifying a plurality of images each associated with metadata;

determining a group of images from the plurality of images based on the metadata associated with each image, wherein a value of at least one parameter of the metadata associated with each image of the group of images are similar;

for each image of the group of images, detecting one or more faces included in the image;

determining clusters of faces from the detected faces, wherein each cluster of faces includes only detected faces that meet a similarity threshold to each other detected face in the cluster of faces; and

for each cluster of faces, matching the detected faces of the cluster of faces to an identity of a person.

16. The computer-readable medium of claim 15, wherein the parameter is a location-based parameter, and wherein a location value of the location-based parameter of the metadata associated with each image of the group of images are similar.

17. The computer-readable medium of claim 15, wherein the parameter is a time-based parameter, and wherein a time value of the time-based parameter of the metadata associated with each image of the group of images are similar.

18. The computer-readable medium of claim 15, the operations further comprising providing a user interface that identifies a particular cluster of faces of the cluster of faces, wherein matching further comprises receiving input to the user interface assigning correlation information to the detected faces of the particular cluster of faces, the correlation information identifying the identity of the person associated with the detected faces of the particular cluster of faces.

19. The computer-readable medium of claim 15, the operations further comprising providing a user interface that identifies each cluster of faces, wherein matching further comprises receiving, for each cluster of faces, a single user input to the user interface assigning correlation information to the detected faces of the cluster of faces, the correlation information identifying the identity of the person associated with the detected faces of the cluster of faces.

20. The computer-readable medium of claim 15, the operations further comprising providing a user interface that identifies i) each cluster of faces and ii) a plurality of identities, each identity associated with a person associated with a user, wherein matching further comprises receiving input to the user interface assigning, for one or more cluster of faces, an identify of a person from the plurality of identities to the cluster of faces.