System and method for enabling the use of captured images through recognition

• US 7,519,200 B2
• Filed: 10/07/2005
• Issued: 04/14/2009
• Est. Priority Date: 05/09/2005
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Like.Com (Alphabet Inc.)
• Inventors: Shah, Munjal, Lee, Kuang-Chih, Gokturk, Salih Burak, Vu, Diem, Anguelov, Dragomir, Vanhoucke, Vincent, Khan, Azhar, Yang, Danny
• Primary Examiner(s): KIM, CHONG R
• Application Number:
  US11/246,742
• Publication Number:
  US 20060251339A1
• Time in Patent Office: 1,285 Days
• Field of Search: None
• US Class Current: 382/118
• CPC Class Codes: G06F 16/50 of still image data
• CPC Class Codes: G06F 16/53 Querying
• CPC Class Codes: G06F 16/58 Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
• CPC Class Codes: G06F 16/5846 using extracted text
• CPC Class Codes: G06F 16/587 using geographical or spatial information, e.g. location
• CPC Class Codes: G06V 40/107 Static hand or arm

Abstract

An embodiment provides for enabling retrieval of a collection of captured images that form at least a portion of a library of images. For each image in the collection, a captured image may be analyzed to recognize information from image data contained in the captured image, and an index may be generated, where the index data is based on the recognized information. Using the index, functionality such as search and retrieval is enabled. Various recognition techniques, including those that use the face, clothing, apparel, and combinations of characteristics may be utilized. Recognition may be performed on, among other things, persons and text carried on objects.

Claims

1. A method for enabling retrieval of a collection of captured images that form at least a portion of a collection of images, the method comprising:

for individual images in the collection, using a processor to perform steps comprising;

(a) detecting one or more persons depicted in the captured image;

(b) performing normalization on a face of the detected one or more persons, including normalizing effects of lighting when the image was captured using at least one of a histogram or linear ramp intensity analysis;

(c) performing image analysis on the captured image to determine recognition information about the one or more persons in the captured image using image data contained in the captured image that depicts the face of the one or more persons, further including determining information about a clothing or apparel of the detected one or more persons;

(d) using a probabilistic model and the recognition information to determine an identity assignment for the one or more persons in the captured image, wherein the probability model is based at least on a time when the image was captured and the information about the clothing or apparel, wherein the identify assignment is determined according to a combination that includes at least the time information and the clothing or apparel information.

2. The method of claim 1, further comprising:

generating an index that correlates at least one of each captured image or depicted persons to a determined identity assignment, and

enabling the index to be used for searching images in the collection using input.

3. The method of claim 2, wherein enabling the index to be searched includes enabling the index to be searched using input that corresponds to a portion of an image containing a person.

4. The method of claim 1, wherein performing image analysis includes:

determining a recognition signature that identifies at least one of the one or more persons from other persons recognized in the collection.

5. The method of claim 1, further comprising:

generating a recognition signature based at least in part on the recognition information and information used in the probabilistic model.

6. The method of claim 1, further comprising:

receiving an input that includes an image depicting a person

determining an identity assignment of the depicted person of the input by performing steps (a) through (d); and

performing a search using the input by associating the image of the input with one or more other images in the collection that have the same identity assignment.

7. A system for enabling retrieval of a collection of captured images that form at least a portion of a library of images, the system comprising:

an image analysis module that operates to;

receive images for the library, the images being captured by one or more consumer-level users that operate corresponding image capturing devices;

detect whether individual images depict persons;

normalize the individual images for lighting and pose of the depicted persons using at least one of a histogram or linear ramp intensity analysis;

perform recognition on the normalized images to determine recognition information for the depicted persons, the recognition information including information about a clothing or apparel of the detected one or more persons;

use a probabilistic model and the recognition information to determine an identity assignment for the one or more persons in the captured image, wherein the probabilistic model is based at least on a time when the image captured and the information about the clothing or apparel, wherein the identity assignment is determined according to a combination that includes at least the time information and the clothing or apparel information;

an indexer that generates index data, wherein the index data associates the recognition information determined from the one or more persons depicted in the received images with a corresponding captured image that contained that person.

8. The system of claim 7, wherein the recognition information includes recognition signatures that include quantitative data, and wherein the system comprises an index that maintains recognition signatures in association with captured images that contained the one or more persons from which the recognition information is determined from.

9. The system of claim 7, wherein the image analysis module generates one or more modules are configured to generate recognition information for persons using one or more facial features of the person.

10. The system of claim 9, wherein the image analysis module generates one or more modules are configured to generate a recognition signature for persons appearing in each image in the collection using the one or more features of the person and at least one of (i) a clothing or apparel worn by the person, (ii) metadata associated with each image in the collection from which a person is recognized.