Method of performing motion-based object extraction and tracking in video

• US 8,379,915 B2
• Filed: 03/16/2007
• Issued: 02/19/2013
• Est. Priority Date: 11/20/2006
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: VideoSurf, Inc. (Microsoft Corporation)
• Inventors: Sharon, Eitan, Brandt, Achi
• Primary Examiner(s): Lu, Tom Y
• Assistant Examiner(s): CONWAY, THOMAS A
• Application Number:
  US11/687,261
• Publication Number:
  US 20080118107A1
• Time in Patent Office: 2,167 Days
• Field of Search: None
• US Class Current: 382/103
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/30241 Trajectory
• CPC Class Codes: G06T 7/215 Motion-based segmentation
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments

Abstract

A method of extracting objects from a video includes the identification of regions to be tracked, tracking the regions across several frames, calculating motions of the regions, and identifying new regions to be tracked. Regions of a selected frame of the video that are appropriate for tracking are selected. Tracking of the identified regions is then performed across frames subsequent to the selected frame of the video. Motions of the identified regions are calculated, and new regions for tracking are identified.

Claims

1. A computer-implemented method of extracting objects from a video comprising the steps of:

identifying, by a processor, regions of a selected frame of the video appropriate for tracking movement of one or more salient objects appearing in the video;

tracking, by a processor, identified regions across frames subsequent to said selected frame of the video;

calculating, by a processor, motions of the identified regions including reacquiring tracking of a previously identified salient object for which tracking had been lost;

identifying, by a processor, new regions for tracking, said identifying based on determining whether one or more previously tracked salient objects have become untrackable, changes in the movement of the salient objects, new salient objects and changes in interactions between the salient objects;

collecting, by a processor, tracking information related to the movement of the salient objects into one or more object-related tubes;

assigning a weight, by a processor, to each object-related tube, wherein the weight is determined based on the amount of time or the number of frames by which the salient objects are tracked relative to the total video length in comparison to one or more other tracked objects;

receiving, by a processor, search criteria specifying a target object and a corresponding weight associated with the target object;

searching, by a processor, one or more videos having object-related tubes that match the target object and the corresponding weight associated with the target object; and

displaying, by a processor, the one or more videos to a user.

2. The method according to claim 1 wherein said step of identifying new regions for tracking is responsive to said step of calculating motions of said identified regions.

3. The method according to claim 1 wherein said step of identifying new regions for tracking is responsive to said step of tracking identified regions across frames.

4. The method according to claim 1 wherein said step of identifying regions of said selected frame includes a step of segmenting at least one frame of the video to partition the at least one frame into coherent regions.

5. The method according to claim 1 wherein said step of identifying regions of said selected frame includes a step of extracting local motion estimations for various points in the selected frame across said frames subsequent to said selected frame.

6. The method according to claim 1 wherein said step of tracking identified regions across said frames subsequent to said selected frame includes a step of determining whether a moveable object is trackable.

7. The method according to claim 1 wherein said step of tracking identified regions across said frames subsequent to said selected frame includes a step of determining whether one of said frames subsequent to said selected frame is substantially different from said selected frame.

8. The method according to claim 1 wherein said step of calculating motions of the identified regions includes a step of determining a motion of each of the identified regions of the selected frame across said frames subsequent to said selected frame.

9. The method according to claim 1 wherein said step of calculating motions of the identified regions includes a step of grouping together ones of said identified regions based on their common motion to generate more complex regions of interest.

10. The method according to claim 1 wherein said step of calculating motions of the identified regions includes steps of determining a camera motion and factoring said camera motion out to identify salient objects that move differently from said camera motion.

11. The method according to claim 1 wherein said step of calculating motions of the identified regions includes steps of identifying a salient object and continuing tracking of said salient object.

12. The method according to claim 1 wherein said step of calculating motions of the identified regions includes a step of receiving user input designating a salient object of interest.

13. The method according to claim 1 wherein said step of tracking identified regions across said frames subsequent to said selected frame includes a step of determining a motion of each of the identified regions of the selected frame across said frames subsequent to said selected frame.

14. The method according to claim 13 wherein said step of determining a motion of each of the identified regions includes fitting said identified regions to said frames subsequent to said selected frame.

15. The method according to claim 13 wherein said step of determining a motion of each of the identified regions includes a step of estimating local motion within said selected frame.

16. A computer-implemented method of extracting objects from a video comprising the steps of:

identifying, by a processor, regions of a selected frame of the video appropriate for tracking movement of one or more salient objects appearing in the video;

tracking, by a processor, identified regions across frames subsequent to said selected frame of the video;

calculating, by a processor, motions of the identified regions including reacquiring tracking of a previously identified salient object for which tracking had been lost;

identifying, by a processor, new regions for tracking;

assigning a weight, by a processor, to one or more object-related tubes comprising tracking information related to the movement of the salient objects, said weight determined based on the amount of time or the number of frames by which the salient objects are tracked relative to the total video length in comparison to one or more other tracked objects;

searching, by a processor, one or more videos having object-related tubes that match a target object and a corresponding weight associated with the target object, specified by search criteria; and

displaying, by a processor, the one or more videos to a user.

17. A computer-implemented method of extracting objects from a video comprising the steps of:

identifying, by a processor, regions of a selected frame of the video appropriate for tracking movement of one or more salient objects appearing in the video;

tracking, by a processor, identified regions across frames subsequent to said selected frame of the video;

calculating, by a processor, motions of the identified regions including reacquiring tracking of a previously identified salient object for which tracking had been lost; and

identifying, by a processor, new regions for tracking, said identifying based on determining whether one or more previously tracked salient objects have become untrackable, changes in the movement of the salient objects, new salient objects and changes in interactions between the salient objects.

18. The method of claim 17, further comprising collecting, by a processor, tracking information related to the movement of the salient objects into one or more object-related tubes.

19. The method of claim 18, wherein the step of collecting comprises assigning a weight, by a processor, to each object-related tube, wherein the weight is determined based on the amount of time or the number of frames by which the salient objects are tracked relative to the total video length in comparison to one or more other tracked objects.

20. The method of claim 19, further comprising searching, by a processor, one or more videos having object-related tubes that match a target object and a corresponding weight associated with the target object, specified by search criteria.