Texture replacement in video sequences and images

• US 7,564,465 B2
• Filed: 10/29/2007
• Issued: 07/21/2009
• Est. Priority Date: 10/11/2001
• Status: Expired due to Term

• Current Assignee: AT&T Intellectual Property II LP (AT&T, Inc.)
• Original Assignee: AT&T Intellectual Property II LP (AT&T, Inc.)
• Inventors: Haskell, Barin Geoffry, Dumitras, Adriana
• Primary Examiner(s): Tung; Kee M
• Assistant Examiner(s): Richer; Aaron M
• Application Number:
  US11/926,868
• Publication Number:
  US 20080055332A1
• Time in Patent Office: 631 Days
• Field of Search: 345582-587
• US Class Current: 345/582
• CPC Class Codes: G06T 11/001 Texturing; Colouring; Generation of texture or colour inpainting G06T5/77
• CPC Class Codes: G06T 7/40 Analysis of texture depth or shape recovery from texture G06T7/529
• CPC Class Codes: G06T 7/41 based on statistical description of texture
• CPC Class Codes: G06T 9/00 Image coding bandwidth or redundancy reduction for static pictures H04N1/41; coding or decoding of static colour picture signals H04N1/64; methods or arrangements for coding, decoding, compressing or decompressing digital video signals H04N19/00
• CPC Class Codes: G06T 9/001 Model-based coding, e.g. wire frame
• CPC Class Codes: H04N 19/14 Coding unit complexity, e.g. amount of activity or edge presence estimation H04N19/146 takes precedence
• CPC Class Codes: H04N 19/167 Position within a video image, e.g. region of interest [ROI]
• CPC Class Codes: H04N 19/17 the unit being an image region, e.g. an object
• CPC Class Codes: H04N 19/186 the unit being a colour or a chrominance component
• CPC Class Codes: H04N 19/23 with coding of regions that are present throughout a whole video segment, e.g. sprites, background or mosaic
• CPC Class Codes: H04N 19/27 involving both synthetic and natural picture components, e.g. synthetic natural hybrid coding [SNHC]

Abstract

Systems and methods for reducing bit rates by replacing original texture in a video sequence with synthesized texture. Reducing the bit rate of the video sequence begins by identifying and removing selected texture from frames in a video sequence. The removed texture is analyzed to generate texture parameters. New texture is synthesized using the texture parameters in combination with a set of constraints. Then, the newly synthesized texture is mapped back into the frames of the video sequence from which the original texture was removed. The resulting frames are then encoded. The bit rate of the video sequence with the synthesized texture is less than the bit rate of the video sequence with the original texture. Also, the ability of a decoder to decode the new video sequence is not compromised because no assumptions are made about the texture synthesis capabilities of the decoder.

Claims

1. A computer-implemented method for processing video data, the method comprising:

an encoder executing the following steps;

removing texture from at least some frames of a set of frames in a video sequence;

synthesizing new texture based on texture parameters associated with the removed texture and at least one of overall color, color saturation, and whether an original texture associated with the set of frames is structured and busy or unstructured and smooth, wherein;

if the original texture is structured and busy, then synthesizing the new texture based on a first set of constraints, the first set of constraints further comprising marginal statistics, coefficient correlations, coefficient magnitude correlations, cross-scale statistics, overall color and color saturation; and

if the original texture is unstructured and smooth, then synthesizing the new texture based on a second set of constraints; and

inserting the synthesized new texture into the set of frames to enable display of the set of frames.

2. The computer-implemented method of claim 1, wherein the step of synthesizing new texture is further based on at least one of coefficient magnitude correlations, cross scale statistics, coefficient correlations, marginal statistics, and dominant texture orientation.

3. The computer-implemented method of claim 1, wherein the second set of constraints comprises coefficient magnitude correlations, cross-scale statistics, overall color and color saturation.

4. The computer-implemented method of claim 1, wherein inserting the synthesized new texture into the set of frames enables the display of the set of frames at a decoder.

5. The computer-implemented method of claim 1, wherein a bit rate of a video sequence that includes the synthesized texture is less than a bit rate of the original video sequence.

6. The computer-implemented method of claim 1, wherein a region of interest is identified within the texture of an initial frame.

7. A computing device for processing video data, the computing device comprising:

a encoder;

a module configured to control the encoder to remove texture from at least some frames of a set of frames in a video sequence;

a module configured to control the encoder to synthesize new texture based on texture parameters associated with the removed texture and at least one of overall color, color saturation, and whether an original texture associated with the set of frames is structured and busy or unstructured and smooth;

a module configured, if the original texture is structured and busy, to control the encoder to synthesize the new texture based on a first set of constraints, the first set of constraints further comprising marginal statistics, coefficient correlations, coefficient magnitude correlations, cross-scale statistics, overall color and color saturation;

a module configured, if the original texture is unstructured and smooth, to control the encoder to synthesize the new texture based on a second set of constraints; and

a module configured to control the processor to insert the synthesized new texture into the set of frames.

8. The computing device of claim 7, wherein the module configured to control the encoder to synthesize new texture further synthesizes new texture based on at least one of coefficient magnitude correlations, cross scale statistics, coefficient correlations, marginal statistics, and dominant texture orientation.

9. The computing device of claim 7, wherein the second set of constraints comprises coefficient magnitude correlations, cross-scale statistics, overall color and color saturation.

10. The computing device of claim 7, wherein the module configured to control the encoder to insert the synthesized new texture into the set of frames further enables the display of the set of frames at a decoder.

11. The computing device of claim 7, wherein a bit rate of a video sequence that includes the synthesized texture is less than a bit rate of the original video sequence.

12. The computing device of claim 7, wherein a region of interest is identified within the texture of an initial frame.

13. A tangible computer-readable medium for storing instructions for controlling a computing device to process video data, the instructions comprising:

removing texture from at least some frames of a set of frames in a video sequence;

synthesizing new texture based on texture parameters associated with the removed texture and at least one of overall color, color saturation, and whether an original texture associated with the set of frames is structured and busy or unstructured and smooth, wherein;

if the original texture is structured and busy, then synthesizing the new texture based on a first set of constraints, the first set of constraints further comprising marginal statistics, coefficient correlations, coefficient magnitude correlations, cross-scale statistics, overall color and color saturation; and

if the original texture is unstructured and smooth, then synthesizing the new texture based on a second set of constraints; and

inserting the synthesized new texture into the set of frames to enable display of the set of frames.

14. The computer-readable medium of claim 13, wherein the step of synthesizing new texture is further based on at least one of coefficient magnitude correlations, cross scale statistics, coefficient correlations, marginal statistics, and dominant texture orientation.

15. The computer-readable medium of claim 13, wherein the second set of constraints comprises coefficient magnitude correlations, cross-scale statistics, overall color and color saturation.

16. The computer-readable medium of claim 13, wherein inserting the synthesized new texture into the set of frames enables the display of the set of frames at a decoder.

17. The computer-readable medium of claim 13, wherein a bit rate of a video sequence that includes the synthesized texture is less than a bit rate of the original video sequence.

18. The computer-readable medium of claim 13, wherein a region of interest is identified within the texture of an initial frame.