Method of Performing Motion-Based Object Extraction and Tracking in Video

• US 20080118107A1
• Filed: 03/16/2007
• Published: 05/22/2008
• Est. Priority Date: 11/20/2006
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: VideoSurf, Inc. (Microsoft Corporation)
• Inventors: Brandt, Achi, Sharon, Eitan
• Granted Patent: US 8,379,915 B2
• Time in Patent Office: Days
• US Class Current: 382/103
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/30241 Trajectory
• CPC Class Codes: G06T 7/215 Motion-based segmentation
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments

Abstract

A method of extracting objects from a video includes the identification of regions to be tracked, tracking the regions across several frames, calculating motions of the regions, and identifying new regions to be tracked. Regions of a selected frame of the video that are appropriate for tracking are selected. Tracking of the identified regions is then performed across frames subsequent to the selected frame of the video. Motions of the identified regions are calculated, and new regions for tracking are identified.

Claims

1. A method of extracting objects from a video comprising the steps of:

identifying regions of a selected frame of the video appropriate for tracking;

tracking identified regions across frames subsequent to said selected frame of the video;

calculating motions of the identified regions; and

identifying new regions for tracking.

2. The method according to claim 1 wherein said step of identifying new regions for tracking is responsive to said step of calculating motions of said identified regions.

3. The method according to claim 1 wherein said step of identifying new regions for tracking is responsive to said step of tracking identified regions across frames.

4. The method according to claim 1 wherein said step of identifying regions of said selected frame includes a step of segmenting at least one frame of the video to partition the frame into coherent regions.

5. The method according to claim 1 wherein said step of identifying regions of said selected frame includes a step of extracting local motion estimations for various points in the selected frame across said frames subsequent to said selected frame.

6. The method according to claim 1 wherein said step of tracking identified regions across said frames subsequent to said selected frame includes a step of determining a motion of each of the identified regions of the selected frame across said frames subsequent to said selected frame.

7. The method according to claim 6 wherein said step of determining a motion of each of the identified regions includes fitting said identified regions to said frames subsequent to said selected frame.

8. The method according to claim 6 wherein said step of determining a motion of each of the identified regions includes a step of estimating local motion within said selected frame.

9. The method according to claim 1 wherein said step of tracking identified regions across said frames subsequent to said selected frame includes a step of determining whether a moveable object is trackable.

10. The method according to claim 1 wherein said step of tracking identified regions across said frames subsequent to said selected frame includes a step of determining whether one of said frames subsequent to said selected frame is substantially different from said selected frame.

11. The method according to claim 1 wherein said step of calculating motions of the identified regions includes a step of determining a motion of each of the identified regions of the selected frame across said frames subsequent to said selected frame.

12. The method according to claim 1 wherein said step of calculating motions of the identified regions includes a step of grouping together ones of said identified regions based on their common motion to generate more complex regions of interest.

13. The method according to claim 1 wherein said step of calculating motions of the identified regions includes steps of determining a camera motion and factoring said camera motion out to identify salient objects that move differently from said camera motion.

14. The method according to claim 1 wherein said step of calculating motions of the identified regions includes steps of identifying a salient object and continuing tracking of said salient object.

15. The method according to claim 1 wherein said step of calculating motions of the identified regions includes a step of receiving user input designating a salient object of interest.

16. A method of generating an indexed video comprising the steps of:

identifying regions of a frame of the video appropriate for tracking;

tracking the identified regions across subsequent frames of the video;

calculating motions of the identified regions including identifying coherent video objects;

identifying new regions for tracking in response to said calculated motions;

creating tubes that relate to said coherent video objects; and

indexing and storing said tubes in a searchable database.

17. The method according to claim 16 wherein said step of tracking includes generating tracking information and said step of creating tubes includes a step of collecting said tracking information into space-time threads.

18. The method according to claim 16 wherein said step of creating tubes includes steps of identifying common salient objects and joining ones of said tubes of respective ones of said common salient objects.

19. The method according to claim 16 further comprising steps of tagging and assigning weights to each of said tubes.

20. The method according to claim 19 further comprising a step of analyzing tube features to identify said weights.