Integrated systems and methods for video-based object modeling, recognition, and tracking

• US 8,170,280 B2
• Filed: 12/03/2008
• Issued: 05/01/2012
• Est. Priority Date: 12/03/2007
• Status: Active Grant

• Current Assignee: TiVo Solutions Inc. (Adeia Inc.)
• Original Assignee: Digital Smiths Incorporated
• Inventors: Berry, Matthew G., Zhao, Liang
• Primary Examiner(s): Azarian, Seyed
• Application Number:
  US12/327,589
• Publication Number:
  US 20090141940A1
• Time in Patent Office: 1,245 Days
• Field of Search: 382/100, 382/103, 382/106, 382/107, 382/155, 382/162, 382/168, 382/173, 382/181, 382/190, 382/209, 382/219, 382/232, 382/254, 382/274, 382/276, 382/291, 382/305, 382/312, 1/1, 345/619, 707/802, 715/723
• US Class Current: 382/103
• CPC Class Codes: G06F 18/28 Determining representative reference patterns, e.g. by averaging or distorting; Generating dictionaries
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/20081 Training; Learning
• CPC Class Codes: G06T 2207/30201 Face
• CPC Class Codes: G06T 7/248 involving reference images or patches
• CPC Class Codes: G06T 7/40 Analysis of texture depth or shape recovery from texture G06T7/529
• CPC Class Codes: G06V 10/772 Determining representative reference patterns, e.g. averaging or distorting patterns; Generating dictionaries
• CPC Class Codes: G06V 20/40 in video content extracting overlay text G06V20/62; video retrieval G06F16/70; processing of video elementary streams in video servers H04N21/234; processing of video elementary streams in video clients H04N21/44
• CPC Class Codes: G06V 40/173 face re-identification, e.g. recognising unknown faces across different face tracks

Abstract

The present disclosure relates to systems and methods for modeling, recognizing, and tracking object images in video files. In one embodiment, a video file, which includes a plurality of frames, is received. An image of an object is extracted from a particular frame in the video file, and a subsequent image is also extracted from a subsequent frame. A similarity value is then calculated between the extracted images from the particular frame and subsequent frame. If the calculated similarity value exceeds a predetermined similarity threshold, the extracted object images are assigned to an object group. The object group is used to generate an object model associated with images in the group, wherein the model is comprised of image features extracted from optimal object images in the object group. Optimal images from the group are also used for comparison to other object models for purposes of identifying images.

Claims

1. A method for tracking object images in video files, comprising the steps of:

receiving a video file, wherein the video file comprises a plurality of frames;

extracting an image of an object from a particular frame in the video file;

extracting a subsequent image of an object from a subsequent frame in the video file;

calculating a similarity value between the extracted object image from the particular frame and the subsequent extracted object image from the subsequent frame;

if the calculated similarity value exceeds a predetermined similarity threshold, collecting the extracted object images from the video file into an object group;

extracting a plurality of object features from the extracted object images collected in the object group;

generating an object model based on the plurality of extracted object features, where the object model is associated with an object identifier; and

updating the object model with additional object features extracted from one or more additional object images collected in the object group but extracted from an additional video file.

2. The method of claim 1, further comprising the steps of:

identifying one or more optimal object images from the images collected in the object group, where the plurality of object features are extracted from the one or more optimal object images identified from the images collected in the object group and where the plurality of object features comprise image data associated with the one or more optimal object images.

3. The method of claim 2, wherein the plurality of object features in the object model are weighted based on the relative importance of each feature.

4. The method of claim 2, wherein the one or more optimal object images are identified based on properties of the image.

5. The method of claim 4, wherein the image properties comprise one or more of resolution, occlusion, brightness, scale, and pose.

6. The method of claim 2, wherein the one or more optimal object images are identified by calculating a similarity score between the images in the object group and learned examples of optimal object images.

7. The method of claim 1, further comprising the steps of:

identifying one or more optimal object images from the images in the object group;

retrieving one or more predefined object models, wherein each object model is associated with an object identifier;

calculating an average similarity value between the one or more identified optimal object images and each of the one or more predefined object models; and

if at least one of the calculated average similarity values exceeds a predetermined average similarity threshold, labeling the object images in the object group according to the respective object identifier.

8. The method of claim 7, further comprising the step of if none of the calculated average similarity values exceeds a predetermined average similarity threshold, labeling the object images in the object group as unknown.

9. The method of claim 7, wherein the average similarity value is calculated based on a predefined algorithm.

10. The method of claim 7, wherein the average similarity value is calculated by comparing object features of the optimal object images to object features of the one or more predefined object models.

11. The method of claim 1, wherein the similarity value, is calculated based on a predefined algorithm.

12. The method of claim 1, wherein the similarity value is calculated by comparing object features, spatial features, and contextual features of the extracted object image from the particular frame to object features, spatial features, and contextual features of the subsequent extracted object image from the subsequent frame.

13. The method of claim 12, wherein the spatial features comprise data associated with physical distances in images.

14. The method of claim 12, wherein the contextual features comprise data associated with elements surrounding an object image in a frame.

15. The method of claim 1, wherein the object group is stored in a database.

16. The method of claim 1, wherein the object group comprises a plurality of object, images similar to the extracted images.

17. The method of claim 1, wherein the object images comprise images of faces.

18. A method of tracking a specific object through a plurality of frames of a video wherein each frame of the video includes a plurality of objects, comprising the steps of:

dividing the video into one or more scenes;

identifying, an image of the specific object in a first frame of one of the one or more scenes;

assigning the image of the specific object from the first frame of the respective scene to an object group;

analyzing a subsequent frame of the respective scene to identify a potential additional image of the specific object;

calculating a similarity value between the image of the specific object from the first frame and the potential additional image of the specific object from the subsequent frame;

if the calculated similarity value exceeds a predetermined similarity threshold, assigning the potential additional image to the object group; and

repeating the steps of analyzing and calculating for additional subsequent frames of the respective scene whereby the object group has assigned thereto all images of the specific object in the analyzed frames of the video.

19. The method of claim 18, further comprising the step of selecting the specific object, from the plurality of objects in the, first frame of the one or more scenes.

20. The method of claim 18, wherein the step of assigning the image of the specific object to the object group includes associating in a database the image with the frame from which the image was identified.

21. The method of claim 18, wherein the subsequent frame is a frame immediately following the first frame.

22. The method of claim 18, wherein there is at least one interposing frame between the first frame and the subsequent frame.

23. The method of claim 18, wherein the step of calculating a similarity value comprises comparing object features of the image of the specific object from the first frame and the potential additional image of the specific object from the subsequent frame.

24. A method for generating an object model from one or more images in a video, comprising the steps of:

receiving a video file, wherein the video file comprises a plurality of video frames;

detecting an object image from one of the plurality of video frames;

if the detected object image is an optimal object image, extracting a plurality of object features from the optimal object image;

determining whether an object model already exists for the object represented by the optimal object image;

if an object model does not exist for the object, generating an object model comprising the plurality of object features extracted from the optimal object image; and

updating the object model with additional object features extracted from one or more additional optimal object images associated with the object extracted from one or more additional video files.

25. The method of claim 24, further comprising the step of if an object model already exists for the object, adding the plurality of object features extracted from the optimal object image to the already-existing object model.

26. The method of claim 24, farther comprising the step of determining that the detected object image is an optimal object image based on properties of the image.

27. The method of claim 26, wherein the image properties comprise one or more of resolution, occlusion, brightness, scale, and pose.

28. The method of claim 24, further comprising the step of determining that the detected object image is an optimal object image by calculating a similarity value between the detected object image and learned examples of optimal object images.

29. A system for identifying object images from videos, comprising:

a processor;

a computer program product that includes a computer-readable medium that is usable by the processor, the medium having stored thereon a sequence of instructions that when executed by the processor causes the execution of the steps of;

retrieving a video, wherein the video comprises a plurality of frames;

extracting a plurality of object images from the plurality of frames;

grouping similar object images together into a group based on a similarity score calculated between features in the plurality of object images;

calculating an average similarity score for the plurality of object images in the group as compared to one or more stored object models, where each object model is associated with a respective object identifier;

if the calculated average similarity score does not exceed a predetermined threshold, labeling the plurality of object images in the group as unknown; and

if the calculated average similarity score exceeds the predefined threshold, identifying the plurality of object images in the group based on the respective object identifier of the most similar stored object model.

30. The system of claim 29, further comprising a database for storing one or more videos and the one or more object models.

31. The system of claim 29, wherein the average similarity score is calculated by comparing object features of the plurality of object images to object features of the one or more stored object models.