Texture replacement in video sequences and images

• US 6,977,659 B2
• Filed: 09/09/2002
• Issued: 12/20/2005
• Est. Priority Date: 10/11/2001
• Status: Expired due to Term

• Current Assignee: AT&T Corporation (AT&T, Inc.)
• Original Assignee: AT&T Corporation (AT&T, Inc.)
• Inventors: Haskell, Barin Geoffry, Dumitras, Adriana
• Primary Examiner(s): Bella, Matthew C.
• Assistant Examiner(s): TRAN, TAM D
• Application Number:
  US10/237,489
• Publication Number:
  US 20030076334A1
• Time in Patent Office: 1,198 Days
• Field of Search: 345/582, 345/583, 345/600, 345/605
• US Class Current: 345/582
• CPC Class Codes: G06T 11/001 Texturing; Colouring; Generation of texture or colour inpainting G06T5/77
• CPC Class Codes: G06T 7/40 Analysis of texture depth or shape recovery from texture G06T7/529
• CPC Class Codes: G06T 7/41 based on statistical description of texture
• CPC Class Codes: G06T 9/00 Image coding bandwidth or redundancy reduction for static pictures H04N1/41; coding or decoding of static colour picture signals H04N1/64; methods or arrangements for coding, decoding, compressing or decompressing digital video signals H04N19/00
• CPC Class Codes: G06T 9/001 Model-based coding, e.g. wire frame
• CPC Class Codes: H04N 19/14 Coding unit complexity, e.g. amount of activity or edge presence estimation H04N19/146 takes precedence
• CPC Class Codes: H04N 19/167 Position within a video image, e.g. region of interest [ROI]
• CPC Class Codes: H04N 19/17 the unit being an image region, e.g. an object
• CPC Class Codes: H04N 19/186 the unit being a colour or a chrominance component
• CPC Class Codes: H04N 19/23 with coding of regions that are present throughout a whole video segment, e.g. sprites, background or mosaic
• CPC Class Codes: H04N 19/27 involving both synthetic and natural picture components, e.g. synthetic natural hybrid coding [SNHC]

Abstract

Systems and methods for reducing bit rates by replacing original texture in a video sequence with synthesized texture. Reducing the bit rate of the video sequence begins by identifying and removing selected texture from frames in a video sequence. The removed texture is analyzed to generate texture parameters. New texture is synthesized using the texture parameters in combination with a set of constraints. Then, the newly synthesized texture is mapped back into the frames of the video sequence from which the original texture was removed. The resulting frames are then encoded. The bit rate of the video sequence with the synthesized texture is less than the bit rate of the video sequence with the original texture. Also, the ability of a decoder to decode the new video sequence is not compromised because no assumptions are made about the texture synthesis capabilities of the decoder.

Claims

1. A method for replacing original texture in a set of frames included in a video sequence with synthesized texture such that a bit rate of the video sequence that includes the synthesized texture is less than a bit rate of the original video sequence, the method comprising:

selecting original texture from an initial frame within the set of frames;

removing the selected original texture from the set of frames in the video sequence;

determining texture parameters from an analysis of the removed texture; and

inserting synthesized texture, based on the texture parameters, into the set of frames, the inserting synthesized texture into the set of frames, further comprising;

synthesizing the removed texture by;

applying, in a prescribed order, coefficient magnitude correlations, cross scale statistics, coefficient correlations, marginal statistics, overall color, and color saturation qualitative constraints if the original texture is structured and busy; and

applying, in a prescribed order, coefficient magnitude correlations, cross scale statistics, overall color, and color saturation qualitative constraints if the original texture is unstructured and smooth.

2. A method for replacing original texture in a set of frames included in a video sequence with synthesized texture such that a bit rate of the video sequence that includes the synthesized texture is less than a bit rate of the original video sequence, the method comprising:

selecting original texture from an initial frame within the set of frames;

removing the selected original texture from the set of frames in the video sequence;

analyzing the removed texture to compute texture parameter;

synthesizing new texture using the texture parameters; and

inserting the synthesized texture into the set of frames, wherein synthesizing new texture using the texture parameters further comprises;

applying qualitative constraints to the new texture, wherein the qualitative constraints include at least one of;

marginal statistics,

coefficient correlations,

coefficient magnitude correlations,

cross scale statistics,

overall color, and

color saturation; and

synthesizing new texture using the texture parameters further comprises one of;

applying, in a prescribed order, coefficient magnitude correlations, cross scale statistics, coefficient correlations, marginal statistics, overall color, and color saturation qualitative constraints if the original texture is structured and busy; and

applying, in a prescribed order, coefficient magnitude correlations, cross scale statistics, overall color, and color saturation qualitative constraints if the original texture is unstructured and smooth.

3. In a system that distributes a video sequence over a network, a method for replacing original texture of a video sequence with synthesized texture in order to reduce a bit rate of the video sequence, the method comprising:

removing original texture from a set of frames included in the video sequence, wherein the original texture is identified from a single frame included in the video sequence;

analyzing the removed texture by computing texture parameters that include statistical texture descriptors;

generating synthesized texture using the texture parameters and qualitative constraints; and

replacing the original texture in the set of frames with the synthesized texture, wherein;

the qualitative constraints include one or more of marginal statistics, coefficient correlations, coefficient magnitude correlations, cross-scale statistics, overall color and color saturation, and wherein;

generating synthesized texture using the texture parameters and qualitative constraints further comprises;

applying, in a prescribed order, coefficient magnitude correlations, cross-scale statistics, coefficient correlations, marginal statistics, overall color, and color saturation qualitative constraints if the original texture is structured and busy; and

applying, in a prescribed order, coefficient magnitude correlations, cross-scale statistics, overall color, and color saturation qualitative constraints if the original texture is unstructured and smooth.

4. A method for replacing original texture in a plurality of frames that are included in a video sequence with synthesized texture, the method comprising:

selecting a region of interest in an initial frame, wherein the region of interest has color characteristics;

identifying an original texture in the initial frame using the color characteristics of the region of interest, wherein some of the pixels in the frame that have color characteristics that are similar to the color characteristics of the region of interest are included in the original texture;

removing the original texture from the frame;

computing texture parameters from the original texture;

creating synthesized texture using the texture parameters, wherein the synthesized texture is distinguishable from the original texture; and

inserting the synthesized texture into a set of frames that includes the initial frame, wherein;

creating synthesized texture using the texture parameters further comprises;

applying qualitative constraints to the synthesized texture, which further comprises;

applying, in a prescribed order, coefficient magnitude correlations, cross scale statistics, coefficient correlations, marginal statistics, overall color,

and color saturation qualitative constraints if the original texture is structured and busy; and

applying, in a prescribed order, coefficient magnitude correlations, cross scale statistics, overall color, and color saturation qualitative constraints if the original texture is unstructured and smooth.

5. A computer program product for implementing a method for replacing original texture in a set of frames included in a video sequence with synthesized texture such that a bit rate of the video sequence that includes the synthesized texture is less than a bit rate of the original video sequence, the computer program product comprising:

a computer-readable medium having computer-executable instructions for performing the method, the method comprising;

selecting original texture from an initial frame included in the video sequence;

removing the selected original texture from a set of frames in the video sequence, wherein the set of frames includes the initial frame;

analyzing the removed texture to compute texture parameters;

synthesizing new texture using the texture parameters; and

inserting the synthesized texture into the set of frames, wherein;

synthesizing new texture using the texture parameters further comprises;

applying qualitative constraints to the new texture, wherein the qualitative constraints include at least one of;

marginal statistics,

coefficient correlations,

coefficient magnitude correlations,

cross-scale statistics,

overall color, and

color saturation; and

synthesizing new texture using the texture parameters further comprises one of;

applying, in a prescribed order, coefficient magnitude correlations, cross scale statistics, coefficient correlations, marginal statistics, overall color, and color saturation qualitative constraints if the original texture is structured and busy; and

applying, in a prescribed order, coefficient magnitude correlations, cross scale statistics, overall color, and color saturation qualitative constraints if the original texture is unstructured and smooth.