Tracking semantic objects in vector image sequences

• US 20050240629A1
• Filed: 06/29/2005
• Published: 10/27/2005
• Est. Priority Date: 09/10/1998
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Lee, Ming-Chieh, Gu, Chuang
• Granted Patent: US 7,162,055 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06T 7/174 involving the use of two or more images
• CPC Class Codes: G06T 7/215 Motion-based segmentation
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G06V 10/24 Aligning, centring, orientation detection or correction of the image
• CPC Class Codes: H04N 19/543 using regions
• CPC Class Codes: H04N 19/80 Details of filtering operations specially adapted for video compression, e.g. for pixel interpolation H04N19/635, H04N19/86 take precedence

Abstract

A semantic object tracking method tracks general semantic objects with multiple non-rigid motion, disconnected components and multiple colors throughout a vector image sequence. The method accurately tracks these general semantic objects by spatially segmenting image regions from a current frame and then classifying these regions as to which semantic object they originated from in the previous frame. To classify each region, the method perform a region based motion estimation between each spatially segmented region and the previous frame to computed the position of a predicted region in the previous frame. The method then classifies each region in the current frame as being part of a semantic object based on which semantic object in the previous frame contains the most overlapping points of the predicted region. Using this method, each region in the current image is tracked to one semantic object from the previous frame, with no gaps or overlaps. The method propagates few or no errors because it projects regions into a frame where the semantic object boundaries are previously computed rather than trying to project and adjust a boundary in a frame where the object'"'"'s boundary is unknown.

Claims

1-20. -20. (canceled)

21. A system for tracking video objects in video frames, the system comprising one or more modules configured for:

performing spatial segmentation on a current video frame to identify plural regions of pixels with homogenous intensity values;

performing motion estimation between each of the plural regions in the current video frame and a previous video frame in which a boundary of a video object was previously computed;

using the motion estimation for each of the plural regions to warp pixel locations in the region to locations in the previous video frame;

determining whether the warped pixel locations are within the previously computed boundary of the video object in the previous video frame to identify a set of the plural regions that are likely to be part of the video object in the current video frame; and

forming a boundary of the video object in the current video frame as a combination of each of the plural regions in the current video frame that are in the set.

22. The system of claim 21 wherein the one or more modules are further configured for:

repeating claim 21 for a subsequent video frame using the boundary of the video object in the current video frame as a reference boundary for the subsequent video frame.

23. The system of claim 21 wherein the one or more modules are further configured for:

filtering the current video frame to remove noise from the current video frame before performing the spatial segmentation.

24. The system of claim 21 wherein the spatial segmentation includes, for each of the plural regions, ensuring that a difference between a maximum intensity value in the region and a minimum intensity value in the region is below a threshold.

25. The system of claim 21 wherein the spatial segmentation is a sequential region growing comprising:

starting with a first pixel location in the current video frame, growing a first region of connected pixels around the first pixel location by adding pixel locations to the first region such that a homogeneity criteria is satisfied;

when no boundary pixels around the first region satisfy the homogeneity criteria, repeating the growing for another region with a pixel location outside the first region; and

continuing the growing until each of the pixel locations in the current video frame is identified as being part of one of the plural regions.

26. The system of claim 21 wherein the motion estimation comprises:

for each of the plural regions identified through spatial segmentation in the current video frame, performing a region-based motion estimation including matching only pixels within the region with pixels in the previous video frame; and

applying a motion model to approximate motion of the pixels within the region to corresponding locations in the previous frame.

27. The system of claim 26 wherein the motion model is used to find a motion vector for each of the plural regions.

28. The system of claim 21 wherein the determining includes, for each of the plural regions:

counting the warped pixel locations that are within the boundary of the video object in the previous video frame;

when a majority of the warped pixel locations are within the boundary of the video object, classifying the region as being in the set that are likely to be part of the video object in the current video frame.

29. A system for tracking an object in a vector image sequence using backward region-based classification, the system comprising:

means for computing a boundary of an object in a first image frame of a vector image sequence;

means for segmenting a second image frame of the vector image sequence into plural regions;

means for, based upon motion estimates, warping each of the plural regions backward into the first image frame;

means for, for each of the plural regions of the second image frame, if a threshold portion of the warped region lies within the previously computed boundary in the first image frame, classifying the region of the second image frame as part of the object.

30. The system of claim 29 further comprising:

means for forming a new boundary of the object in the second image frame from one or more of the plural regions classified as part of the object.

31. The system of claim 29 wherein the segmenting comprises:

growing a first region of the plural regions from a first image point by adding neighboring image points that satisfy a homogeneity criterion for the first region, wherein the homogeneity criterion constrains the difference between a maximum image point value and a minimum image point value in the first region; and

repeating the growing for other regions of the plural regions until each image point of the second image frame is part of one of the plural regions.

32. The system of claim 29 wherein the first image frame includes plural objects having boundaries, and wherein the classifying indicates one of the plural objects for each of the plural regions of the second image frame.

33. The system of claim 29 wherein the segmenting includes performing spatial segmentation.

34. The system of claim 29 further comprising:

means for filtering the second image frame before the segmenting.

35. The system of claim 29 wherein the threshold portion is a majority.

36. A system for tracking an object in a vector image sequence using backward region-based classification, the system comprising one or more modules adapted for:

computing a boundary of an object in a first image frame of a vector image sequence;

segmenting a second image frame of the vector image sequence into plural regions;

for each of the plural regions of the second image frame,

associating the region with a corresponding region of the first image frame based upon motion estimation from the second image frame back to the first image frame; and

if a threshold portion of the associated corresponding region of the first image frame lies within the previously computed boundary in the first image frame, classifying the region of the second image frame as part of the object.

37. The system of claim 36 wherein the one or more modules are further adapted for:

forming a new boundary of the object in the second image frame from one or more of the plural regions classified as part of the object.

38. The system of claim 37 wherein the one or more modules are further adapted for:

repeating the segmenting, associating, and classifying for a third image frame, wherein the associating is based upon motion estimation from the third image frame back to the second image frame, and wherein the classifying occurs by comparison to the new boundary.

39. The system of claim 37 wherein the one or more modules are further adapted for filtering the second image frame before the segmenting.

40. The system of claim 37 wherein the threshold portion is a majority.