System and method for object based parametric video coding

• US 9,813,731 B2
• Filed: 08/06/2014
• Issued: 11/07/2017
• Est. Priority Date: 09/04/2009
• Status: Active Grant

• Current Assignee: STMicroelectronics International N.V. (STMicroelectronics NV)
• Original Assignee: STMicroelectronics International N.V. (STMicroelectronics NV)
• Inventors: Chaudhury, Santanu, Mathur, Mona, Khandelia, Aditya, Tripathi, Subarna, Lall, Brejesh, Dutta Roy, Sumantra, Gorecha, Saurabh
• Primary Examiner(s): Rahaman, Mohammed
• Assistant Examiner(s): CARTER, RICHARD BRUCE
• Application Number:
  US14/453,006
• Publication Number:
  US 20140348231A1
• Time in Patent Office: 1,189 Days
• Field of Search: 37524008, 37524002, 37524016, 382181, 382199
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 7/215 Motion-based segmentation
• CPC Class Codes: H04N 19/12 Selection from among a plurality of transforms or standards, e.g. selection between discrete cosine transform [DCT] and sub-band transform or selection between H.263 and H.264
• CPC Class Codes: H04N 19/132 Sampling, masking or truncation of coding units, e.g. adaptive resampling, frame skipping, frame interpolation or high-frequency transform coefficient masking
• CPC Class Codes: H04N 19/139 Analysis of motion vectors, e.g. their magnitude, direction, variance or reliability
• CPC Class Codes: H04N 19/14 Coding unit complexity, e.g. amount of activity or edge presence estimation H04N19/146 takes precedence
• CPC Class Codes: H04N 19/159 Prediction type, e.g. intra-frame, inter-frame or bidirectional frame prediction
• CPC Class Codes: H04N 19/17 the unit being an image region, e.g. an object
• CPC Class Codes: H04N 19/20 using video object coding
• CPC Class Codes: H04N 19/23 with coding of regions that are present throughout a whole video segment, e.g. sprites, background or mosaic
• CPC Class Codes: H04N 19/436 using parallelised computational arrangements
• CPC Class Codes: H04N 19/543 using regions
• CPC Class Codes: H04N 19/587 involving temporal sub-sampling or interpolation, e.g. decimation or subsequent interpolation of pictures in a video sequence
• CPC Class Codes: H04N 19/61 in combination with predictive coding

Abstract

A video compression framework based on parametric object and background compression is proposed. At the encoder, an embodiment detects objects and segments frames into regions corresponding to the foreground object and the background. The object and the background are individually encoded using separate parametric coding techniques. While the object is encoded using the projection of coefficients to the orthonormal basis of the learnt subspace (used for appearance based object tracking), the background is characterized using an auto-regressive (AR) process model. An advantage of the proposed schemes is that the decoder structure allows for simultaneous reconstruction of object and background, thus making it amenable to the new multi-thread/multi-processor architectures.

Claims

1. A digital video encoder, comprising:

one or more memories; and

digital signal processing circuitry, which, in operation;

segments frames in a digital video sequence into object regions and background regions;

generates object motion and appearance parameters based on segmented object regions;

categorizes blocks in background P frames regions into edge blocks and non-edge blocks; and

generates an auto regressive background model based on the non-edge blocks of the segmented background regions.

2. The digital video encoder of claim 1 wherein the digital signal processing circuitry comprises:

a segmentation module which, in operation, segments frames in the digital sequence into object regions and background regions;

an object parameter generator which, in operation, generates object motion and appearance parameters; and

an auto-regressive model generator, which, in operation, generates the auto regressive background model based on non-edge blocks of the segmented background regions.

3. The encoder of claim 2 wherein the digital signal processing circuitry comprises an encoding module which, in operation, compresses background I frames, and wherein the segmentation module, in operation, provides background I frames to the encoding module.

4. The encoder of claim 2 wherein the object parameter generator comprises an object tracking module, a residual calculator and an H.264 I slice encoder and the object tracking module, in operation, generates object motion and appearance parameters based on received object region information and decoded objects received from the residual calculator.

5. The encoder of claim 3 wherein the digital signal processing circuitry comprises a block categorizer which, in operation, categorizes blocks in background P frames regions into edge blocks and non-edge blocks, wherein the encoding module, in operation, compresses the edge blocks and the auto regressive model generator, in operation, generates the auto regressive background model based on non-edge blocks of the segmented background regions.

6. The encoder of claim 3 wherein the encoding module comprises an H.264/AVG encoder.

7. The encoder of claim 5, wherein the digital signal processing circuitry comprises a motion threader, which, in operation, generates motion thread information based on non-edge blocks of segmented background regions, wherein the auto regressive model generator, in operation, generates the auto regressive background model based on the motion thread information.

8. The encoder of claim 5 wherein the block categorizer comprises a gradient-based edge detector.

9. The encoder of claim 7, wherein the digital signal processing circuitry comprises a block removal module, which, in operation, removes background blocks based on an error criteria.

10. The digital video encoder of claim 1 wherein the auto regressive background model includes background texture parameters.

11. A method comprising:

segmenting frames in a digital video sequence into object regions and background regions;

generating object motion and appearance parameters based on segmented object regions;

categorizing blocks in background P frames regions into edge blocks and non-edge blocks; and

generating auto regressive background model parameters based on the non-edge blocks of the segmented background regions.

12. The method of claim 11, comprising:

compressing background I frames using transform-based encoding.

13. The method of claim 11, comprising:

compressing background I frames using H.264/AVC encoding.

14. The method of claim 11, comprising:

encoding the edge blocks using H.264/AVC encoding.

15. The method of claim 11, comprising:

generating motion thread information based on non-edge blocks of the background P frames.

16. The method of claim 11 wherein the categorizing blocks comprises gradient-based edge detection.

17. The method of claim 11 wherein the generating object motion and appearance parameters comprises calculating and compressing object residuals.

18. The method of claim 11 wherein the auto regressive background model parameters include background texture parameters.

19. A non-transitory, computer-readable memory medium whose contents cause a digital signal processor to perform a method of encoding a video image sequence, the method comprising:

segmenting frames in the digital video sequence into object regions and background regions;

generating object motion and appearance parameters based on segmented object regions;

categorizing blocks in background P frames regions into edge blocks and non-edge blocks; and

generating auto regressive background model parameters based on the non-edge blocks of the segmented background regions.

21. The medium of claim 19 wherein the method comprises:

generating motion thread information based on non-edge blocks of the background P frames.

22. The medium of claim 19 wherein the categorizing blocks comprises gradient-based edge detection.

23. The medium of claim 19 wherein the auto regressive background model parameters include background texture parameters.

20. The medium of claim medium 19 wherein the contents are instructions, which, when executed by the digital signal processor, cause the digital signal processor to perform the method.