Object tracking

• US 10,178,396 B2
• Filed: 12/30/2011
• Issued: 01/08/2019
• Est. Priority Date: 09/04/2009
• Status: Active Grant

• Current Assignee: STMicroelectronics International N.V. (STMicroelectronics NV)
• Original Assignee: STMicroelectronics International N.V. (STMicroelectronics NV)
• Inventors: Chaudhury, Santanu, Tripathi, Subarna, Roy, Sumantra Dutta
• Primary Examiner(s): Lee, Y
• Assistant Examiner(s): Carter, Richard B
• Application Number:
  US13/341,697
• Publication Number:
  US 20120170659A1
• Time in Patent Office: 2,566 Days
• Field of Search: 37524016, 382103, 382107, 382190, 348168
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/20021 Dividing image into blocks, subimages or windows
• CPC Class Codes: G06T 7/194 involving foreground-background segmentation
• CPC Class Codes: G06T 7/215 Motion-based segmentation
• CPC Class Codes: H04N 19/12 Selection from among a plurality of transforms or standards, e.g. selection between discrete cosine transform [DCT] and sub-band transform or selection between H.263 and H.264
• CPC Class Codes: H04N 19/132 Sampling, masking or truncation of coding units, e.g. adaptive resampling, frame skipping, frame interpolation or high-frequency transform coefficient masking
• CPC Class Codes: H04N 19/139 Analysis of motion vectors, e.g. their magnitude, direction, variance or reliability
• CPC Class Codes: H04N 19/14 Coding unit complexity, e.g. amount of activity or edge presence estimation H04N19/146 takes precedence
• CPC Class Codes: H04N 19/154 Measured or subjectively estimated visual quality after decoding, e.g. measurement of distortion use of rate-distortion criteria H04N19/147
• CPC Class Codes: H04N 19/159 Prediction type, e.g. intra-frame, inter-frame or bidirectional frame prediction
• CPC Class Codes: H04N 19/17 the unit being an image region, e.g. an object
• CPC Class Codes: H04N 19/20 using video object coding
• CPC Class Codes: H04N 19/36 Scalability techniques involving formatting the layers as a function of picture distortion after decoding, e.g. signal-to-noise [SNR] scalability
• CPC Class Codes: H04N 19/436 using parallelised computational arrangements
• CPC Class Codes: H04N 19/46 Embedding additional information in the video signal during the compression process H04N19/517, H04N19/68, H04N19/70 take precedence
• CPC Class Codes: H04N 19/527 Global motion vector estimation
• CPC Class Codes: H04N 19/543 using regions
• CPC Class Codes: H04N 19/61 in combination with predictive coding
• CPC Class Codes: H04N 19/70 characterised by syntax aspects related to video coding, e.g. related to compression standards
• CPC Class Codes: H04N 19/82 involving filtering within a prediction loop

Abstract

A video compression framework based on parametric object and background compression is proposed. At the encoder, an object is detected and frames are segmented into regions corresponding to the foreground object and the background. The encoder generates object motion and appearance parameters. The motion or warping parameters may include at least two parameters for object translation; two parameters for object scaling in two primary axes and one object orientation parameter indicating a rotation of the object. Particle filtering may be employed to generate the object motion parameters. The proposed methodology is the formalization of the concept and usability for perceptual quality scalability layer for Region(s) of Interest. A coded video sequence format is proposed which aims at “network friendly” video representation supporting appearance and generalized motion of object(s).

Claims

1. A digital video encoder, comprising:

a segmentation module configured to segment frames in a digital video sequence into background regions and object regions; and

an object parameter generator configured to generate object motion and object appearance parameters based on segmented object regions, the object parameter generator including a tracking module configured to track object motion and changes in object appearance of segmented object regions, the tracking object motion including;

identifying object motion parameter states based on particle filtering; and

tracking one or more object regions based on one or more object motion parameter states selected from the identified object motion parameter states,

wherein the tracking module is configured to;

estimate a target state by selecting one of the identified motion parameter states;

determine a similarity of a normalized color histogram for a region associated with the estimated target state and an initial color histogram for a region associated with an object;

compare the similarity to a first similarity threshold;

when the similarity exceeds the first similarity threshold, set a target state for the region associated with the object to the estimated target state;

when the similarity does not exceed the first similarity threshold, selectively, identify new motion parameter states; and

estimate a new target state by selecting one of the identified new motion parameter states; and

when the similarity does not exceed the first similarity threshold and a number of iterations of identifying new motion parameter states exceeds an iteration threshold;

determine whether the object is visible based on a comparison of the similarity to a second similarity threshold and a comparison of a reconstruction error associated with the estimated target state to a reconstruction error threshold; and

when it is determined that the object is visible, setting the target state for the region associated with the object to an identified state having a lowest reconstruction error.

2. The digital video encoder of claim 1 wherein the object regions comprise oriented rectangles and at least one of the object motion parameters defines an orientation of an object region.

3. The digital video encoder of claim 1 wherein the particle filtering includes importance sampling.

4. The digital video encoder of claim 3 wherein the importance sampling is based on a normalized color histogram.

5. The digital encoder of claim 1 wherein the tracking module is configured to select a motion parameter state from the identified motion parameter states that is a minimum distance from a learnt subspace.

6. The digital encoder of claim 1 wherein the tracking module is configured to select a motion parameter state having a minimum reconstruction error with respect to a learnt subspace.

7. The digital encoder of claim 1 wherein the tracking module is configured to determine whether an object is occluded in a frame.

8. The encoder of claim 1, further comprising:

an auto regressive model generator configured to generate an auto regressive background model based on segmented background regions.

9. A method, comprising:

segmenting frames in a digital video sequence into background regions and object regions;

generating object motion parameters of a plurality of segmented object regions based on particle filtering;

generating object appearance parameters of the plurality of segmented object regions, the generating object appearance parameters including tracking changes in appearance of one or more of the plurality of segmented object regions; and

encoding the object motion parameters and the object appearance parameters in a bit stream,

wherein the generating object motion parameters comprises selecting one of the identified motion parameter states as an estimated target state,

and the method comprises;

determining a similarity of a normalized color histogram for a region associated with the estimated target state and an initial color histogram for the segmented object region;

comparing the similarity to a first similarity threshold;

when the similarity exceeds the first similarity threshold, setting a target state for the region associated with the object to the estimated target state; and

when the similarity does not exceed the first similarity threshold, selectively, identifying new motion parameter states; and

estimating a new target state by selecting one of the identified new motion parameter states; and

when the similarity does not exceed the first similarity threshold and a number of iterations of identifying new motion parameter states exceeds an iteration threshold, the method comprises;

determining whether the object is visible based on a comparison of the similarity to a second similarity threshold and a comparison of a reconstruction error associated with the estimated target state to a reconstruction error threshold; and

when it is determined that the object is visible, setting the target state for the region associated with the object to an identified state having a lowest reconstruction error.

10. The method of claim 9 wherein the generating object motion parameters comprises:

identifying motion parameter states based on the particle filtering; and

tracking the segmented object regions based on one or more motion parameter states selected from the identified motion parameter states.

11. The method of claim 9 wherein the object regions comprise oriented rectangles and the object motion parameters comprise a translation parameter, a scaling parameter and a parameter indicating a difference of an orientation angle between a width axis of the object region and an axis of a bounding box for successive frames in the digital video sequence.

12. The method of claim 9 wherein the particle filtering includes importance sampling.

13. The method of claim 12 wherein the importance sampling is based on a normalized color histogram.

14. The method of claim 9 wherein the estimated target state is one of the identified sample motion parameter states that is a minimum distance from a learnt subspace.

15. The method of claim 9 wherein the selecting one of the motion parameter states comprises determining a reconstruction error.

16. A non-transitory computer-readable medium whose contents cause an encoder to perform a method of encoding a video image sequence, the method comprising:

segmenting frames in a digital video sequence into background regions and object regions;

generating object motion parameters of a plurality of segmented object regions based on particle filtering;

generating object appearance parameters of the plurality of segmented object regions, the generating object appearance parameters including tracking changes in appearance of one or more of the plurality of segmented object regions; and

encoding the object motion parameters and the object appearance parameters in a bit stream,

wherein the generating object motion parameters comprises selecting one of the identified motion parameter states as an estimated target state, and the method comprises;

determining a similarity of a normalized color histogram for a region associated with the estimated target state and an initial color histogram for the segmented object region;

comparing the similarity to a first similarity threshold;

when the similarity exceeds the first similarity threshold, setting a target state for the region associated with the object to the estimated target state; and

when the similarity does not exceed the first similarity threshold, selectively,

identifying new motion parameter states; and

estimating a new target state by selecting one of the identified new motion parameter states; and

when the similarity does not exceed the first similarity threshold and a number of iterations of identifying new motion parameter states exceeds an iteration threshold, the method comprises;

determining whether the object is visible based on a comparison of the similarity to a second similarity threshold and a comparison of a reconstruction error associated with the estimated target state to a reconstruction error threshold; and

when it is determined that the object is visible, setting the target state for the region associated with the object to an identified state having a lowest reconstruction error.

17. The non-transitory computer-readable medium of claim 16 wherein the generating object motion parameters comprises:

identifying motion parameter states based on the particle filtering; and

tracking the segmented object regions based on one or more motion parameter states selected from the identified motion parameter states.

18. The non-transitory computer-readable medium of claim 16 wherein the object regions comprise oriented rectangles and the object motion parameters comprise a translation parameter, a scaling parameter and a parameter indicating a difference of an orientation angle between a width axis of the object region and an axis of a bounding box for successive frames in the digital video sequence.

19. The non-transitory computer-readable medium of claim 16 wherein the particle filtering includes importance sampling.