System and method of object recognition and database population for video indexing

• US 8,457,368 B2
• Filed: 10/18/2012
• Issued: 06/04/2013
• Est. Priority Date: 11/07/2007
• Status: Active Grant

• Current Assignee: Viewdle, Inc. (Alphabet Inc.)
• Original Assignee: Viewdle, Inc. (Alphabet Inc.)
• Inventors: Kovtun, Ivan, Zhukov, Oleksandr, Musatenko, Yuriy, Schlesinger, Mykhailo
• Primary Examiner(s): AKHAVANNIK, HADI
• Application Number:
  US13/654,541
• Publication Number:
  US 20130039545A1
• Time in Patent Office: 229 Days
• Field of Search: 382/118
• US Class Current: 382/118
• CPC Class Codes: G06F 16/784 the detected or recognised objects being people
• CPC Class Codes: G06F 18/232 Non-hierarchical techniques
• CPC Class Codes: G06V 10/763 Non-hierarchical techniques, e.g. based on statistics of modelling distributions
• CPC Class Codes: G06V 20/41 Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items segmenting video sequences G06V20/49
• CPC Class Codes: G06V 40/172 Classification, e.g. identification

Abstract

A method for processing digital media is described. The method, in one example embodiment, includes identification of objects in a video stream by detecting, for each video frame, an object in the video frame and selectively associating the object with an object cluster. The method may further include comparing the object in the object cluster to a reference object and selectively associating object data of the reference object with all objects within the object cluster based on the comparing. The method may further include manually associating the object data of the reference object with all objects within the object duster having no associated reference object and populating a reference database with the reference object for the object cluster.

Claims

1. A method of processing a video stream including a plurality of video frames, the method comprising:

detecting appearances of an object in one or more of the plurality of video frames in the video stream;

responsive to detecting an appearance of the object in a video frame;

extracting an image of the object from the video frame;

identifying first metadata from at least one of a video frame or audio track associated with the plurality of video frames; and

associating the first metadata with the image of the object;

determining a distance between the extracted image and at least one image from an object cluster;

associating the image with the object duster responsive to the distance, the object cluster comprising images of the object;

comparing at least one image from the object cluster to a reference image of a known object in same category;

determining whether the image from the object cluster is of the known object based on the comparison.

2. The method of claim 1, further comprising automatically determining whether the image from the object cluster is of the known object based on metadata associated with the object duster.

3. The method of claim 1, wherein the image of the object is selected from a group consisting of cars, buildings, animals, and helicopters.

4. The method of claim 2, wherein comparing the at least one image from the object cluster to a reference image comprises comparing the image with a plurality of reference images from a database of reference objects.

5. The method of claim 1, wherein comparing the at least one image from the object cluster to a reference image comprises:

providing the at least one image from the object cluster to a user interface;

providing the reference image to the user interface; and

receiving a user input indicating that a user views the image from the object cluster as being associated with the known object.

6. The method of claim 1, wherein detecting appearances of the object comprises:

partitioning the video stream into a plurality of frame groups;

detecting an image of the object in a first frame and a last frame of a frame group;

in each of one or more video frames that fall between the first frame and the last frame in the frame group, extrapolating approximate regions in which the image of the object is likely to be located; and

limiting the detecting of images of the object to the approximate regions.

7. The method of claim 1, wherein the comparing the at least one image from the object cluster to a reference image of a known person comprises:

performing a first action that compares the image from the object cluster with a bigger set of reference images, producing a smaller set of reference images with an increased likelihood of identifying the image from the cluster as an image of a known person; and

performing a second action that compares the image from the cluster with the smaller set of reference images.

8. The method of claim 1, wherein the reference image is selected from a collection of high-resolution images of categories of the object.

9. The method of claim 1, wherein the extracted image is associated with the object cluster when a distance between a vector representing the extracted image and an image from the object cluster is above a first predetermined threshold but below a second predetermined threshold.

10. The method of claim 1, the method further comprising creating, a new object cluster when a distance between a vector representing the extracted image and images from any object cluster is greater than a predetermined threshold and associating the extracted image with the new object cluster.

11. The method of claim 1, wherein the object cluster is an object folder, the method further comprising saving the extracted image and the metadata to files in the object folder.

20. The method of claim 1, the method further comprising associating second metadata with the object cluster, the second metadata identifying the object cluster as comprising images of the known object.

12. A system for processing a video stream including a plurality of video frames, the system comprising:

a non-transitory computer-readable storage medium storing executable computer program instructions that when executed by one or more processors cause the processors to;

detect appearances of an object in one or more of the plurality of video frames in the video stream;

responsive to detecting an appearance of the object in a video frame;

extract an image of the object from the video frame;

identify first metadata from at least one of as video frame or audio track associated with the plurality of video frames; and

associate the first metadata with the image of the object;

determine a distance between the extracted image and at least one image from an object cluster;

associate the extracted object image with the object cluster responsive to the distance, the object cluster comprising images of the person;

compare at least one image from the object cluster to a reference image of a known object in same category;

determine whether the image from the object cluster is of the known object based on the comparison; and

associate second metadata with the object cluster, the second metadata identifying the object cluster as comprising images of the known object.

13. The system of claim 12, further comprising instructions causing the processors to populate a reference database with a reference image from every object cluster having associated metadata identifying the object duster as comprising images of the known object.

14. The system of claim 12, wherein the comparing module is to compare the at least one object to the at least one reference object from a database of reference objects.

15. The system of claim 12, wherein comparing the at least one image from the object cluster to a reference image comprises:

providing the at least one image from the object cluster to a user interface;

providing the reference images to the user interface; and

receiving module a user input indicating that a user views the image from the object duster as being associated with the known object.

16. The system of claim 12, wherein detecting appearances of the object comprises:

partitioning the video stream into a plurality of frame groups;

detecting an image of the object in a first frame and as last frame of a frame group;

extrapolating approximate regions in one or more video frames in the frame group between the first frame and the last frame in which the image of the object is likely to be located; and

limiting the detecting of object images to the approximate regions.

17. The system of claim 12, wherein the extracted imago is associated with the object cluster when a distance between a vector representing the extracted image and an image from the object cluster is above a first predetermined threshold but below a second predetermined threshold.

18. The system of claim 12, further comprising instructions causing the processors to:

compare images of the object in subsequent frames; and

use data derived from appearances of the object in one frame to identify the object or properties of the object in another frame.

19. The system of claim 12, further comprising instructions causing the processors to:

for an appearance of the person in a video frame in the video stream, save to an index database one or more of the following;

a time of the appearance, duration of the appearance based on the appearance of the person in subsequent video frames, and spatial location of the image of the person within the video frame; and

provide searching capabilities within the index database to display the video stream starting from the appearance of the object in the video frame in the video stream.