Digital video encoder system, method, and non-transitory computer-readable medium for tracking object regions

• US 9,626,769 B2
• Filed: 12/30/2011
• Issued: 04/18/2017
• Est. Priority Date: 09/04/2009
• Status: Active Grant

• Current Assignee: STMicroelectronics International N.V. (STMicroelectronics NV)
• Original Assignee: STMicroelectronics International N.V. (STMicroelectronics NV)
• Inventors: Tripathi, Subarna, Mathur, Mona, Chaudhury, Santanu
• Primary Examiner(s): Philippe, Gims
• Assistant Examiner(s): CARTER, RICHARD BRUCE
• Application Number:
  US13/341,687
• Publication Number:
  US 20120177121A1
• Time in Patent Office: 1,936 Days
• Field of Search: 37524016, 382103, 382107, 382190, 382154, 348168, 348155
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 7/10 Segmentation; Edge detection motion-based segmentation G06T7/215
• CPC Class Codes: G06T 7/20 Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51
• CPC Class Codes: G06T 7/215 Motion-based segmentation
• CPC Class Codes: H04N 19/12 Selection from among a plurality of transforms or standards, e.g. selection between discrete cosine transform [DCT] and sub-band transform or selection between H.263 and H.264
• CPC Class Codes: H04N 19/132 Sampling, masking or truncation of coding units, e.g. adaptive resampling, frame skipping, frame interpolation or high-frequency transform coefficient masking
• CPC Class Codes: H04N 19/139 Analysis of motion vectors, e.g. their magnitude, direction, variance or reliability
• CPC Class Codes: H04N 19/14 Coding unit complexity, e.g. amount of activity or edge presence estimation H04N19/146 takes precedence
• CPC Class Codes: H04N 19/154 Measured or subjectively estimated visual quality after decoding, e.g. measurement of distortion use of rate-distortion criteria H04N19/147
• CPC Class Codes: H04N 19/159 Prediction type, e.g. intra-frame, inter-frame or bidirectional frame prediction
• CPC Class Codes: H04N 19/17 the unit being an image region, e.g. an object
• CPC Class Codes: H04N 19/20 using video object coding
• CPC Class Codes: H04N 19/36 Scalability techniques involving formatting the layers as a function of picture distortion after decoding, e.g. signal-to-noise [SNR] scalability
• CPC Class Codes: H04N 19/436 using parallelised computational arrangements
• CPC Class Codes: H04N 19/527 Global motion vector estimation
• CPC Class Codes: H04N 19/543 using regions
• CPC Class Codes: H04N 19/61 in combination with predictive coding
• CPC Class Codes: H04N 19/70 characterised by syntax aspects related to video coding, e.g. related to compression standards

Abstract

A video compression framework based on parametric object and background compression is proposed. At the encoder, an object is detected and frames are segmented into regions corresponding to the foreground object and the background. The encoder generates object motion and appearance parameters. The motion or warping parameters may include at least two parameters for object translation; two parameters for object scaling in two primary axes and one object orientation parameter indicating a rotation of the object. Particle filtering may be employed to generate the object motion parameters. The proposed methodology is the formalization of the concept and usability for perceptual quality scalability layer for Region(s) of Interest. A coded video sequence format is proposed which aims at “network friendly” video representation supporting appearance and generalized motion of object(s).

Claims

1. A digital video encoder, comprising:

a segmentation module configured to segment frames in a digital video sequence into background regions and object regions; and

an object parameter generator configured to generate object motion and object appearance parameters based on segmented object regions, the motion parameters defining a location of an object and including at least one motion parameter defining an orientation of a segmented object region with respect to an image wherein, the object regions comprise oriented rectangles and the at least one motion parameter comprises a translation parameter, a scaling parameter and a parameter indicating a difference of an orientation angle between a width axis of the object region and an axis of a bounding box for successive frames in the digital video sequence.

2. The encoder of claim 1, further comprising:

an encoding module configured to encode an indication of prioritization of an object.

3. The encoder of claim 2 wherein the encoding module is configured to encode the indication of prioritization as a perceptual quality scalability of a scalable video coding bit stream.

4. The encoder of claim 2 wherein the encoding module is configured to encode the indication of prioritization in a network abstraction layer unit of a video compression standard.

5. The encoder of claim 4 wherein the video compression standard is an H.264/SVC standard.

6. The encoder of claim 1, further comprising an encoding module configured to encode the object motion and appearance parameters of a frame into a raw byte sequence payload.

7. The encoder of claim 6 wherein the motion parameters include restricted affine motion parameters.

8. The encoder of claim 6 wherein the motion parameters include general motion parameters.

9. The encoder of claim 6 wherein the encoding module is configured to multiplex object and background data in the raw byte sequence payload.

10. The encoder of claim 6 wherein a format of the raw byte sequence data is specified in a network abstraction layer format unit.

11. The encoder of claim 1, further comprising:

an auto regressive model generator configured to generate an auto regressive background model based on segmented background regions;

an encoding module configured to compress background I frames, wherein the segmentation module is configured to provide background I frames to the encoding module; and

a block categorizer configured to categorize blocks in background P frames regions into edge blocks and non-edge blocks, wherein the encoding module is further configured to compress the edge blocks and the auto regressive model generator is configured to generate the auto regressive background model based on non-edge blocks of the segmented background regions.

12. The encoder of claim 1 wherein the object parameter generator comprises an object tracking module.

13. The encoder of claim 12 wherein the object parameter generator comprises a residual calculator and an H.264 I slice encoder and the object tracking module is configured to generate the object motion and appearance parameters based on the received object region information and a decoded object received from the residual calculator.

14. The encoder of claim 1, further comprising an H.264/AVC (Advanced Video Coding) compatible encoding module.

15. The encoder of claim 14 wherein the encoding module is configured to employ a raw byte sequence payload in a network abstraction layer unit to specify a bit stream structure.

16. The encoder of claim 15 wherein the network abstraction layer unit is a code slice extension unit.

17. A method, comprising:

segmenting frames in a digital video sequence into background regions and object regions;

generating object motion parameters of a segmented object region, the object motion parameters including at least two motion parameters defining a location of a segmented object region inside the video frame, including at least one motion parameter indicating an orientation of the segmented object region with respect to the video frame;

generating object appearance parameters of the segmented object region; and

encoding the object motion parameters and the object appearance parameters in a bit stream, wherein the object regions comprise oriented rectangles and the at least one motion parameter comprises a translation parameter, a scaling parameter and a parameter indicating a difference of an orientation angle between a width axis of the object region and an axis of a bounding box for successive frames in the digital video sequence.

18. The method of claim 17, further comprising:

encoding an indication of prioritization of the segmented object region in the bit stream.

19. The method of claim 18 wherein encoding the indication of prioritization comprises encoding the indication of prioritization as a perceptual quality scalability of a scalable video coding bit stream.

20. The method of claim 18 wherein encoding the indication of prioritization comprises encoding the indication of prioritization in a reserved field of network abstraction layer unit of video compression standard.

21. The method of claim 20 wherein the video compression standard is an H.264/SVC standard.

22. The method of claim 17 wherein encoding the object motion parameters and the object appearance parameters in a bit stream comprises encoding the object motion parameters and the object appearance parameters of a frame into a raw byte sequence payload.

23. The method of claim 17 wherein encoding the object motion parameters and the object appearance parameters in the bit stream comprises multiplexing object and background data in the bit stream.

24. A non-transitory computer-readable medium whose contents cause an encoder to perform a method of encoding a video image sequence, the method comprising:

segmenting frames in a digital video sequence into background regions and object regions;

generating object motion parameters of a segmented object region, the object motion parameters including at least one motion parameter defining an orientation of the segmented object region with respect to an image frame;

generating object appearance parameters of the segmented object region; and

encoding the object motion parameters and the object appearance parameters in a bit stream, wherein the object regions comprise oriented rectangles and the at least one motion parameter comprises a translation parameter, a scaling parameter and a parameter indicating a difference of an orientation angle between a width axis of the object region and an axis of a bounding box for successive frames in the digital video sequence.

25. The non-transitory computer-readable medium of claim 24, wherein the method further comprises encoding an indication of prioritization of the segmented object region in the bit stream.

26. The non-transitory computer-readable medium of claim 24 wherein encoding the object motion parameters and the object appearance parameters in the bit stream comprises multiplexing object and background data in the bit stream.

27. A digital video decoder, comprising:

a decoding block configured to decode background I frames in an encoded video sequence;

an object appearance estimator configured to estimate an object appearance based on object appearance parameters in the encoded video sequence; and

an object motion compensator configured to compensate for object motion based on object motion parameters in the encoded video sequence, the object motion parameters including at least one motion parameter defining an orientation of a segmented object region with respect to an image frame, wherein the object region is an oriented rectangle and the at least one motion parameter comprises a translation parameter, a scaling parameter and a parameter indicating a difference of an orientation angle between a width axis of the object region and an axis of a bounding box for successive frames in the encoded video sequence.

28. The decoder of claim 27, further comprising:

an object correction module configured to modify the estimated object appearance based on object residual information in the encoded video sequence.

29. The decoder of claim 27 wherein the object motion parameters include general motion parameters.

30. The decoder of claim 27, further comprising:

a demultiplexer configured to split the encoded video sequence into;

encoded H.264 information;

AR coefficients and thread information; and

object appearance and motion parameters and compressed object residue information.

31. The decoder of claim 27 wherein the decoder is configured to extract an indication of prioritization of an object from the encoded video sequence.

32. A method, comprising:

decoding background I frames in an encoded video sequence;

estimating an object appearance based on object appearance parameters in the encoded video sequence; and

compensation for object motion based on object motion parameters in the encoded video sequence, the object motion parameters including at least one motion parameter defining an orientation of a segmented object region with respect to an image frame, wherein the object region is an oriented rectangle and the at least one motion parameter comprises a translation parameter, a scaling parameter and a parameter indicating a difference of an orientation angle between a width axis of the object region and an axis of a bounding box for successive frames in the encoded video sequence.

33. The method of claim 32, further comprising:

demultiplexing the encoded video sequence into;

encoded H.264/AVC information;

AR coefficients and thread information; and

object appearance and motion parameters and compressed object residue information.

34. A system comprising:

means for segmenting frames in a digital video sequence into background regions and object regions;

means for generating object motion parameters of a segmented object region, the object motion parameters including at least one parameter defining an orientation of the segmented object region with respect to an image frame;

means for generating object appearance parameters of the segmented object region; and

means for encoding the object motion parameters and the object appearance parameters in a bit stream, wherein the object region is an oriented rectangle and the at least one motion parameter comprises a translation parameter, a scaling parameter and a parameter indicating a difference of an orientation angle between a width axis of the object region and an axis of a bounding box for successive frames in the digital video sequence.

35. The system of claim 34, further comprising:

means for decoding background I frames in an encoded video sequence; and

means for estimating an object appearance based on object appearance parameters in the encoded video sequence.