SYSTEM AND METHOD OF OBJECT RECOGNITION AND DATABASE POPULATION FOR VIDEO INDEXING

• US 20090141988A1
• Filed: 12/03/2007
• Published: 06/04/2009
• Est. Priority Date: 11/07/2007
• Status: Active Grant

• Current Assignee: Google Technology Holdings LLC (Alphabet Inc.), Viewdle, Inc.
• Original Assignee: Viewdle, Inc.
• Inventors: Zhukov, Oleksandr, Musatenko, Yuriy, Kovtun, Ivan, Schlezinger, Mykhailo
• Granted Patent: US 8,315,430 B2
• Time in Patent Office: Days
• US Class Current: 382/225
• CPC Class Codes: G06F 16/784 the detected or recognised objects being people
• CPC Class Codes: G06F 18/232 Non-hierarchical techniques
• CPC Class Codes: G06V 10/763 Non-hierarchical techniques, e.g. based on statistics of modelling distributions
• CPC Class Codes: G06V 20/41 Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items segmenting video sequences G06V20/49
• CPC Class Codes: G06V 40/172 Classification, e.g. identification

Abstract

A method for processing digital media is described. The method, in one example embodiment, includes identification of objects in a video stream by detecting, for each video frame, an object in the video frame and selectively associating the object with an object cluster. The method may further include comparing the object in the object cluster to a reference object and selectively associating object data of the reference object with all objects within the object cluster based on the comparing. The method may further include manually associating the object data of the reference object with all objects within the object cluster having no associated reference object and populating a reference database with the reference object for the object cluster.

Claims

1. A method to process a video stream including a plurality of video frames, the method comprising;

for each video frame, detecting at least one object;

selectively associating the at least one object with an object cluster,

comparing the at least one object associated with the object cluster to at least one reference object; and

selectively associating object data of the at least one reference object with all objects within the object cluster based on the comparing.

2. The method of claim 1, wherein the associating the object data of the at least one reference object is performed automatically.

3. The method of claim 1, further comprising manually associating the object data of the at least one reference object with all objects within the object cluster having no associated reference object and populating a reference database with the at least one reference object for the object cluster.

4. The method of claim 2, wherein the comparing the at least one object utilizes at least one reference object from a database of reference objects.

5. The method of claim 1, wherein the comparing the at least one object to the at least one reference object comprises:

providing an image including the at least one object to a user interface;

providing the at least one reference object to the user interface; and

receiving a user input to associate the object data with the object cluster.

6. The method of claim 1, wherein the detecting the at least one object comprises:

partitioning the video stream into a plurality of frame groups;

detecting a facial image in a first frame and a last frame of every frame group;

in each of one or more video frames that fall between the first frame and the last frame, extrapolating approximate regions in which the at least one object is likely to be located; and

limiting the detecting of the facial image to the approximate regions.

7. The method of claim 1, wherein the comparing the at least one object to at least one reference object comprises:

performing a first action that compares the at least one object with a bigger set of reference objects, producing a smaller set of reference objects having an increased likelihood of the reference objects and the at least one object being the same object; and

performing a second action that compares the at least one object with the smaller set of reference objects.

8. The method of claim 1, wherein the at least one reference object is selected from a collection of high-resolution facial images.

9. The method of claim 1, wherein the object data is associated with all objects in a data cluster when a distance between a vector representing the at least one object and an at least one processed object in the object cluster is above a first predetermined threshold but below a second predetermined threshold.

10. The method of claim 1, wherein the object data is associated with a new data cluster when a distance between a vector representing the at least one object and any object already in any object cluster from the video stream is greater than a predetermined threshold.

11. The method of claim 1, wherein the object cluster is an object folder, the method further comprising saving the at least one object and the object data to files in the object folder.

12. The method of claim 1, further comprising one or more of normalizing facial images associated with the at least one object to represent a horizontal orientation of the facial images, normalizing an intensity of lighting, normalizing distance between eyes of the facial images, or normalizing a size of the facial images.

13. The method of claim 1, wherein the at least one object is a facial image, the method further comprising selecting images in which a distance between eyes in a normalized facial image is not less than 40 pixels.

14. The method of claim 1, further comprising:

comparing objects in subsequent video frames; and

using data derived from an object in one frame to identify an object or properties of the object in another frame.

15. The method of claim 1, further comprising:

for every object appearance in the video stream, saving to an index database at least one of the following;

a time of the object appearance, duration of the object appearance, and spatial location of the object within the video stream; and

providing object searching capabilities within the index database to display the video stream starting from where an object being searched appeared in the video stream.

16. A system to process a video stream including a plurality of video frames, the system comprising:

a detecting module to detect, for each video frame, at least one object responsive to the at least one object being present in the video frame;

an associating module to selectively associate the at least one object with an object cluster;

a comparing module to compare, for each object cluster, the at least one object in the object cluster to at least one reference object; and

an assigning module to selectively assign object data of the at least one reference object with all objects within the object cluster based on the comparing.

17. The system of claim 16, further comprising a populating module to populate a reference database with a reference object for every object cluster having no associated object data.

18. The system of claim 16, wherein the comparing module is to compare the at least one object to the at least one reference object from a database of reference objects.

19. The system of claim 16, wherein the comparing module is to compare the at least one object to the at least one reference object, the system further comprising:

a client module to provide an image including the at least one object to a user interface, and to provide at least one known object to the user interface; and

a receiving module to receive a user input to associate the object data with the object cluster.

20. The system of claim 16, wherein the detecting module is to detect the at least one object associated with the video frame, the system further comprising:

a partitioning module to partition the video stream into a plurality of frame groups;

a discovering module to discover a facial image in a first frame and a last frame of each of the plurality of frame groups;

an extrapolating module to approximate regions in one or more video frames between the first frame and the last frame in which the facial image is likely to be located; and

a limiting module to limit the detecting of facial images to the approximate regions.

21. The system of claim 16, wherein the associating module to associate the object data with all objects in the object cluster when a distance between a vector representing the at least one object and at least one processed object in the object cluster is above a first predetermined threshold but below a second predetermined threshold.

22. The system of claim 16, further comprising:

an evaluating module to evaluate objects in subsequent frames; and

an identifying module to identify an object or property of the object in one frame using data derived from the object or the property of the object in another frame.

23. The system of claim 16, further comprising:

a saving module to save, for every object appearance in the video stream, to an index database, one or more of the following;

a time of the object appearance, duration of the object appearance, and spatial location of the object within the video stream; and

a searching module to provide object searching capabilities within the index database to facilitate displaying of a video stream starting from where an object being searched appeared in the video stream.

24. A method to process a video stream including a plurality of video frames, the method comprising:

means for detecting, for each video frame, at least one object responsive to the at least one object being present in the video frame;

means for selectively associating the at least one object with an object cluster;

means for comparing the at least one object in the object cluster to at least one reference object; and

means for selectively associating object data of the at least one reference object with all objects within the object cluster based on the comparing.

25. A machine-readable medium comprising instructions, which when implemented by one or more processors, perform the following operations:

for each of a plurality of video frames, detect at least one object responsive to the at least one object being present in the video frame;

selectively associate the at least one object with an object cluster;

compare the at least one object in the object cluster to at least one reference object; and

selectively associate object data of the at least one reference object with all objects within the object cluster based on the comparing.