SYSTEM AND METHOD FOR OBJECT BASED PARAMETRIC VIDEO CODING

• US 20110058609A1
• Filed: 09/04/2009
• Published: 03/10/2011
• Est. Priority Date: 09/04/2009
• Status: Active Grant

• Current Assignee: STMicroelectronics International N.V. (STMicroelectronics NV)
• Original Assignee: STMicroelectronics Pvt Ltd. (STMicroelectronics NV)
• Inventors: Chaudhury, Santanu, Gorecha, Saurabh, Roy, Sumantra Dutta, Mathur, Mona, Tripathi, Subarna, Khandelia, Aditya, Lall, Brejesh
• Granted Patent: US 8,848,802 B2
• Time in Patent Office: Days
• US Class Current: 375/240.16
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 7/215 Motion-based segmentation
• CPC Class Codes: H04N 19/12 Selection from among a plurality of transforms or standards, e.g. selection between discrete cosine transform [DCT] and sub-band transform or selection between H.263 and H.264
• CPC Class Codes: H04N 19/132 Sampling, masking or truncation of coding units, e.g. adaptive resampling, frame skipping, frame interpolation or high-frequency transform coefficient masking
• CPC Class Codes: H04N 19/139 Analysis of motion vectors, e.g. their magnitude, direction, variance or reliability
• CPC Class Codes: H04N 19/14 Coding unit complexity, e.g. amount of activity or edge presence estimation H04N19/146 takes precedence
• CPC Class Codes: H04N 19/159 Prediction type, e.g. intra-frame, inter-frame or bidirectional frame prediction
• CPC Class Codes: H04N 19/17 the unit being an image region, e.g. an object
• CPC Class Codes: H04N 19/20 using video object coding
• CPC Class Codes: H04N 19/23 with coding of regions that are present throughout a whole video segment, e.g. sprites, background or mosaic
• CPC Class Codes: H04N 19/436 using parallelised computational arrangements
• CPC Class Codes: H04N 19/543 using regions
• CPC Class Codes: H04N 19/587 involving temporal sub-sampling or interpolation, e.g. decimation or subsequent interpolation of pictures in a video sequence
• CPC Class Codes: H04N 19/61 in combination with predictive coding

Abstract

A video compression framework based on parametric object and background compression is proposed. At the encoder, an embodiment detects objects and segments frames into regions corresponding to the foreground object and the background. The object and the background are individually encoded using separate parametric coding techniques. While the object is encoded using the projection of coefficients to the orthonormal basis of the learnt subspace (used for appearance based object tracking), the background is characterized using an auto-regressive (AR) process model. An advantage of the proposed schemes is that the decoder structure allows for simultaneous reconstruction of object and background, thus making it amenable to the new multi-thread/multi-processor architectures.

Claims

1. A digital video encoder, comprising:

a segmentation module configured to segment frames in a digital video sequence into object regions and background regions;

an object parameter generator configured to generate object motion and appearance parameters based on segmented object regions; and

an auto regressive model generator configured to generate an auto regressive background model based on segmented background regions.

2. The encoder of claim 1, further comprising an encoding module configured compress background I frames, wherein the segmentation module is configured to provide background I frames to the encoding module.

3. The encoder of claim 2, further comprising a block categorizer configured to categorize blocks in background P frames regions into edge blocks and non-edge blocks, wherein the encoding module is further configured to compress the edge blocks and the auto regressive model generator is configured to generate the auto regressive background model based on non-edge blocks of the segmented background regions.

4. The encoder of claim 3, further comprising a motion threader configured to generate motion thread information based on non-edge blocks of the segmented background region, wherein the auto regressive model generator is configured to generate the auto regressive background model based on the motion thread information.

5. The encoder of claim 4, further comprising a block removal module configured to remove the background blocks based on an error criteria.

6. The encoder of claim 3 wherein the block categorizer comprises a gradient-based edge detector.

7. The encoder of claim 1 wherein the object parameter generator comprises an object tracking module, a residual calculator and an H.264 I slice encoder and the object tracking module is configured to generate the object motion and appearance parameters based on the received object region information and a decoded object received from the residual calculator.

8. The encoder of claim 2 wherein the encoding module comprises an H.264/AVG encoder.

9. A method of encoding a video image sequence in a digital video encoder, the method comprising:

segmenting frames in the digital video sequence into object regions and background regions;

generating object motion and appearance parameters based on segmented object regions; and

generating auto regressive model parameters based on segmented background regions.

10. The method of claim 9, further comprising:

compressing background I frames using transform-based encoding.

11. The method of claim 9, further comprising:

compressing background I frames using H.264/AVC encoding.

12. The method of claim 9, further comprising categorizing blocks in background P frames into edge blocks and non-edge blocks, wherein the generating the auto regressive background model is based on non-edge blocks of the background P frames.

13. The method of claim 12, further comprising:

encoding the edge blocks using H.264/AVC encoding.

14. The method of claim 12, further comprising:

generating motion thread information based on non-edge blocks of the background P frames.

15. The method of claim 12 wherein the categorizing blocks comprises gradient-based edge detection.

16. The method of claim 9 wherein the generating object motion and appearance parameters comprises calculating and compressing object residuals.

17. A computer-readable memory medium whose contents cause a processor to perform a method of encoding a video image sequence, the method comprising:

segmenting frames in the digital video sequence into object regions and background regions;

generating object motion and appearance parameters based on segmented object regions; and

generating auto regressive model parameters based on segmented background regions.

18. A digital video decoder, comprising:

a decoding block configured to decode background I frames in an encoded video sequence;

an object appearance estimator configured to estimate an object appearance based on object appearance parameters in the encoded video sequence; and

a texture synthesizer configured to generate background P blocks based on auto regressive coefficients and thread information in the encoded video sequence.

19. The decoder of claim 18, further comprising:

an object correction module configured to modify the estimated object appearance based on object residual information in the encoded video sequence.

20. The decoder of claim 19, further comprising:

an object motion compensator configured to compensate for object motion based on object motion parameters in the encoded video sequence.

21. The decoder of claim 18 wherein the decoding block is configured to decode edge blocks of background P-frames and the texture synthesizer is configured to generate background P frames based on the auto regressive coefficients, the thread information and the decoded edge blocks.

22. The decoder of claim 18, further comprising:

a demultiplexer configured to split the encoded video sequence into;

encoded H.264 information;

AR coefficients and thread information; and

object appearance and motion parameters and compressed object residue information.

23. The decoder of claim 18 wherein when an error criteria is satisfied, the texture synthesizer is configured to output a previous background P block as a current background P block.

24. A method of decoding an encoded digital video sequence, the method comprising:

decoding background I frames in the encoded video sequence;

estimating an object appearance based on object appearance parameters in the encoded video sequence; and

generating background P frames based on auto regressive coefficients and thread information in the encoded video sequence.

25. The method of claim 24, further comprising:

modifying the estimated object appearance based on object residual information in the encoded video sequence.

26. The method of claim 25, further comprising:

compensating for object motion based on object motion parameters in the encoded video sequence.

27. The method of claim 24, further comprising:

H.264/AVC decoding edge blocks of background P-frames, wherein the generating background P frames is based on the auto regressive coefficients, the thread information and the decoded edge blocks.

28. The method of claim 24, further comprising:

demultiplexing the encoded video sequence into;

encoded H.264/AVC information;

AR coefficients and thread information; and

object appearance and motion parameters and compressed object residue information.

29. The method of claim 24 wherein, when an error criteria is satisfied, generating a background P frame comprises outputting a previous background P block as a current background P block.

30. A computer-readable memory medium whose contents cause a processor to perform a method of decoding an encoded video image sequence, the method comprising:

decoding background I frames in the encoded video sequence;

estimating an object appearance based on object appearance parameters in the encoded video sequence; and

generating background P frames based on auto regressive coefficients and thread information in the encoded video sequence.

31. The computer-readable medium of claim 30 wherein the method further comprises:

modifying the estimated object appearance based on object residual information in the encoded video sequence.

32. The computer-readable medium of claim 30 wherein the method further comprises:

compensating for object motion based on object motion parameters in the encoded video sequence.

33. The computer-readable medium of claim 30 wherein the method further comprises:

H.264/AVC decoding edge blocks of background P-frames, wherein the generating background P frames is based on the auto regressive coefficients, the thread information and the decoded edge blocks.

34. The computer-readable memory medium of claim 30 wherein the method further comprises:

demultiplexing the encoded video sequence into;

encoded H.264/AVC information;

AR coefficients and thread information; and

object appearance and motion parameters and compressed object residue information.

35. The computer-readable memory medium of claim 30 wherein the method further comprises:

when an error criteria is satisfied, outputting a previous background P block as a current background P block.

36. A system comprising:

means for decoding background I frames in an encoded video sequence;

means for estimating an object appearance based on object appearance parameters in the encoded video sequence; and

means for generating background P frames based on auto regressive coefficients and thread information in the encoded video sequence.

37. The system of claim 36, further comprising:

means for modifying the estimated object appearance based on object residual information in the encoded video sequence.

38. The system of claim 37, further comprising:

means for compensating for object motion based on object motion parameters in the encoded video sequence.

39. The system of claim 36 wherein the means for decoding background I frames comprises an H.264 I slice decoder configured to decode background I frames and an H.264 P slice decoder configured to decode edge blocks of background P-frames and the means for generating background P frames is configured to generate background P frames based on the auto regressive coefficients, the thread information and the decoded edge blocks.

40. The system of claim 36, further comprising:

means for splitting the encoded video sequence into;

encoded H.264/AVC information;

AR coefficients and thread information; and

object appearance and motion parameters and compressed object residue information.

41. The system of claim 36 wherein the means for generating background P frames is configured to, when an error criteria is satisfied, output a previous background P block as a current background P block.

42. The system of claim 36, further comprising:

a digital video encoder, including;

means for segmenting frames in a digital video sequence into object regions and background regions;

means for generating object motion and appearance parameters based on segmented object regions; and

means for generating an auto regressive background model based on segmented background regions.