Object recognition and database population for video indexing

• US 8,315,430 B2
• Filed: 12/03/2007
• Issued: 11/20/2012
• Est. Priority Date: 11/07/2007
• Status: Active Grant

• Current Assignee: Google Technology Holdings LLC (Alphabet Inc.), Viewdle, Inc.
• Original Assignee: Viewdle, Inc.
• Inventors: Kovtun, Ivan, Zhukov, Oleksandr, Musatenko, Yuriy, Schlesinger, Mykhailo
• Primary Examiner(s): Akhavannik, Hadi
• Application Number:
  US11/949,258
• Publication Number:
  US 20090141988A1
• Time in Patent Office: 1,814 Days
• Field of Search: 382/100, 382/118
• US Class Current: 382/103
• CPC Class Codes: G06F 16/784 the detected or recognised objects being people
• CPC Class Codes: G06F 18/232 Non-hierarchical techniques
• CPC Class Codes: G06V 10/763 Non-hierarchical techniques, e.g. based on statistics of modelling distributions
• CPC Class Codes: G06V 20/41 Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items segmenting video sequences G06V20/49
• CPC Class Codes: G06V 40/172 Classification, e.g. identification

Abstract

A method for processing digital media is described. The method, in one example embodiment, includes identification of objects in a video stream by detecting, for each video frame, an object in the video frame and selectively associating the object with an object cluster. The method may further include comparing the object in the object cluster to a reference object and selectively associating object data of the reference object with all objects within the object cluster based on the comparing. The method may further include manually associating the object data of the reference object with all objects within the object cluster having no associated reference object and populating a reference database with the reference object for the object cluster.

Claims

1. A method of processing a video stream including a plurality of video frames, the method comprising:

detecting appearances of a person in one or more of the plurality of video frames in the video stream;

responsive to detecting an appearance of the person in a video frame;

extracting an image of the person from the video frame;

identifying first metadata from at least one of a video frame or audio track associated with the plurality of video frames; and

associating the first metadata with the image of the person;

determining a distance between the extracted image and at least one image from an object cluster;

associating the image with the object cluster responsive to the distance, the object cluster comprising images of the person;

comparing at least one image from the object cluster to a reference image of a known person;

determining whether the image from the object cluster is of the known person based on the comparison; and

associating second metadata with the object cluster, the second metadata identifying the object cluster as comprising images of the known person.

2. The method of claim 1, wherein determining whether the image from the object cluster is of the known person and associating the metadata with the object cluster is performed automatically.

3. The method of claim 1, wherein determining whether the image from the object cluster is of the known person is performed manually.

4. The method of claim 2, wherein comparing the at least one image from the object cluster to a reference image comprises comparing the image with a plurality of reference images from a database of reference objects.

5. The method of claim 1, wherein comparing the at least one image from the object cluster to a reference image comprises:

providing the at least one image from the object cluster to a user interface;

providing the reference image to the user interface; and

receiving a user input indicating that a user views the image from the object cluster as being associated with the known person.

6. The method of claim 1, wherein detecting appearances of the person comprises:

partitioning the video stream into a plurality of frame groups;

detecting a facial image of the person in a first frame and a last frame of a frame group;

in each of one or more video frames that fall between the first frame and the last frame in the frame group, extrapolating approximate regions in which the a facial image of the person is likely to be located; and

limiting the detecting of facial images to the approximate regions.

7. The method of claim 1, wherein the comparing the at least one image from the object cluster to a reference image of a known person comprises:

performing a first action that compares the image from the object cluster with a bigger set of reference images, producing a smaller set of reference images with an increased likelihood of identifying the image from the cluster as an image of a known person; and

performing a second action that compares the image from the cluster with the smaller set of reference images.

8. The method of claim 1, wherein the reference image is selected from a collection of high-resolution facial images.

9. The method of claim 1, wherein the extracted image is associated with the object cluster when a distance between a vector representing the extracted image and an image from the object cluster is above a first predetermined threshold but below a second predetermined threshold.

10. The method of claim 1, the method further comprising creating a new object cluster when a distance between a vector representing the extracted image and images from any object cluster is greater than a predetermined threshold and associating the extracted image with the new object cluster.

11. The method of claim 1, wherein the object cluster is an object folder, the method further comprising saving the extracted image and the metadata to files in the object folder.

12. The method of claim 1, wherein the images are facial images and the method further comprises one or more of normalizing facial images to represent a horizontal orientation of the facial images, normalizing an intensity of lighting, normalizing distance between eyes of the facial images, or normalizing a size of the facial images.

13. The method of claim 1, the method further comprising selecting images in which a distance between eyes in a normalized facial image is not less than 40 pixels.

14. The method of claim 1, further comprising:

comparing images of the person in subsequent video frames; and

using data derived from appearances of the person in one frame to identify an object or properties of the object in another frame.

15. The method of claim 1, further comprising:

for an appearance of the person in a video frame in the video stream, saving to an index database at least one of the following;

a time of the appearance, duration of the appearance based on the appearance of the person in subsequent video frames, and spatial location of the image of the person within the video frame; and

providing searching capabilities within the index database to display the video stream starting from the appearance of the person in the video frame in the video stream.

16. A system for processing a video stream including a plurality of video frames, the system comprising:

a non-transitory computer-readable storage medium storing executable computer program instructions that when executed by one or more processors cause the processors to;

detect appearances of a person in one or more of the plurality of video frames in the video stream;

responsive to detecting an appearance of the person in a video frame;

extract an image of the person from the video frame;

identify first metadata from at least one of a video frame or audio track associated with the plurality of video frames; and

associate the first metadata with the image of the person;

determine a distance between the extracted image and at least one image from an object cluster;

associate the extracted facial image with the object cluster responsive to the distance, the object cluster comprising images of the person;

compare at least one image from the object cluster to a reference image of a known person;

determine whether the image from the object cluster is of the known person based on the comparison; and

associate second metadata with the object cluster, the second metadata identifying the object cluster as comprising images of the known person.

17. The system of claim 16, further comprising instructions causing the processors to populate a reference database with a reference image from every object cluster having associated metadata identifying the object cluster as comprising images of a known person.

18. The system of claim 16, wherein the comparing module is to compare the at least one object to the at least one reference object from a database of reference objects.

19. The system of claim 16, wherein comparing the at least one image from the object cluster to a reference image comprises:

providing the at least one image from the object cluster to a user interface;

providing the reference images to the user interface; and

receiving module a user input indicating that a user views the image from the object cluster as being associated with the known person.

20. The system of claim 16, wherein detecting appearances of the person comprises:

partitioning the video stream into a plurality of frame groups;

detecting a facial image of the person in a first frame and a last frame of a frame group;

extrapolating approximate regions in one or more video frames in the frame group between the first frame and the last frame in which the facial image of the person is likely to be located; and

limiting the detecting of facial images to the approximate regions.

21. The system of claim 16, wherein the extracted image is associated with the object cluster when a distance between a vector representing the extracted image and a image from the object cluster is above a first predetermined threshold but below a second predetermined threshold.

22. The system of claim 16, further comprising instructions causing the processors to:

compare images of the person in subsequent frames; and

use data derived from appearances of the person in one face to identify an object or properties of the object in another frame.

23. The system of claim 16, further comprising instructions causing the processors to:

for an appearance of the person in a video frame in the video stream, save to an index database one or more of the following;

a time of the appearance, duration of the appearance based on the appearance of the person in subsequent video frames, and spatial location of the image of the person within the video frame; and

provide searching capabilities within the index database to display the video stream starting from the appearance of the person in the video frame in the video stream.

24. A method to process a video stream including a plurality of video frames, the method comprising:

means for detecting appearances of a person in one or more of the plurality of video frames in the video stream, and responsive to detecting an appearance of the person in a video frame;

extracting an image of the person from the video frame;

identifying first metadata from at least one of a video frame or audio track associated with the plurality of video frames; and

associating the first metadata with the image of the person;

means for determining a distance between the extracted image and at least one image from an object cluster;

means for associating the extracted facial image with the object cluster responsive to the distance, the object cluster comprising images of the person;

means for comparing at least one image from the object cluster to a reference image of a known person, and determining whether the image from the object cluster is of the known person based on the comparison; and

means for associating second metadata with the object cluster, the second metadata identifying the object cluster as comprising images of the known person.

25. A non-transitory computer-readable storage medium storing executable computer program instructions that when executed by one or more processors cause the processors to:

detect appearances of a person in one or more of the plurality of video frames in the video stream;

responsive to detecting an appearance of the person in a video frame;

extract an image of the person from the video frame;

identify first metadata from at least one of a video frame or audio track associated with the plurality of video frames; and

associate the first metadata with the image of the person;

determine a distance between the extracted image and at least one image from an object cluster;

associate the extracted facial image with the object cluster responsive to the distance, the object cluster comprising images of the person;

compare at least one image from the object cluster to a reference image of a known person;

determine whether the image from the object cluster is of the known person based on the comparison; and

associate second metadata with the object cluster, the second metadata identifying the object cluster as comprising images of the known person.