Texture replacement in video sequences and images

• US 7,995,072 B2
• Filed: 07/12/2005
• Issued: 08/09/2011
• Est. Priority Date: 10/11/2001
• Status: Expired due to Fees

• Current Assignee: AT&T Intellectual Property II LP (AT&T, Inc.)
• Original Assignee: AT&T Intellectual Property II LP (AT&T, Inc.)
• Inventors: Haskell, Barin Geoffry, Dumitras, Adriana
• Primary Examiner(s): Richer; Aaron M
• Application Number:
  US11/179,701
• Publication Number:
  US 20050243099A1
• Time in Patent Office: 2,219 Days
• Field of Search: 345582-588
• US Class Current: 345/582
• CPC Class Codes: G06T 11/001 Texturing; Colouring; Generation of texture or colour inpainting G06T5/77
• CPC Class Codes: G06T 7/40 Analysis of texture depth or shape recovery from texture G06T7/529
• CPC Class Codes: G06T 7/41 based on statistical description of texture
• CPC Class Codes: G06T 9/00 Image coding bandwidth or redundancy reduction for static pictures H04N1/41; coding or decoding of static colour picture signals H04N1/64; methods or arrangements for coding, decoding, compressing or decompressing digital video signals H04N19/00
• CPC Class Codes: G06T 9/001 Model-based coding, e.g. wire frame
• CPC Class Codes: H04N 19/14 Coding unit complexity, e.g. amount of activity or edge presence estimation H04N19/146 takes precedence
• CPC Class Codes: H04N 19/167 Position within a video image, e.g. region of interest [ROI]
• CPC Class Codes: H04N 19/17 the unit being an image region, e.g. an object
• CPC Class Codes: H04N 19/186 the unit being a colour or a chrominance component
• CPC Class Codes: H04N 19/23 with coding of regions that are present throughout a whole video segment, e.g. sprites, background or mosaic
• CPC Class Codes: H04N 19/27 involving both synthetic and natural picture components, e.g. synthetic natural hybrid coding [SNHC]

Abstract

Systems and methods for reducing bit rates by replacing original texture in a video sequence with synthesized texture. Reducing the bit rate of the video sequence begins by identifying and removing selected texture from frames in a video sequence. The removed texture is analyzed to generate texture parameters. New texture is synthesized using the texture parameters in combination with a set of constraints. Then, the newly synthesized texture is mapped back into the frames of the video sequence from which the original texture was removed. The resulting frames are then encoded. The bit rate of the video sequence with the synthesized texture is less than the bit rate of the video sequence with the original texture. Also, the ability of a decoder to decode the new video sequence is not compromised because no assumptions are made about the texture synthesis capabilities of the decoder.

Claims

1. A method for processing texture, the method comprising:

determining, via a processor, texture parameters from an analysis of an original texture associated with a set of frames of video data to yield determined texture parameters;

synthesizing, via the processor and based on the determined texture parameters, texture associated with the set of frames of video data by applying, in a first prescribed order, a first set of constraints for unstructured original texture and in a second prescribed order a second set of constraints for structured original texture, wherein the second set of constraints contains more members than the first set of constraints, and wherein the first set of constraints and the second set of constraints each comprises at least one constraint associated with dominant texture orientation and at least one constraint associated with color; and

inserting the texture into the set of frames based on the determined texture parameters.

2. The method of claim 1, further comprising synthesizing the texture according to whether the original texture is one of busy and smooth.

3. The method of claim 1, further comprising:

selecting the original texture from an initial frame within the set of frames to yield selected original texture;

removing the selected original texture from the set of frames in the video sequence to yield removed texture;

determining additional texture parameters from an analysis of the removed texture; and

inserting the texture based on the additional texture parameters into the set of frames.

4. The method of claim 1, wherein a bit rate of a video sequence that includes the texture is less than a bit rate of an original video sequence.

5. The method of claim 1, further comprising identifying a region of interest within texture of an initial frame.

6. The method of claim 5, further comprising computing a mean vector for the region of interest.

7. The method of claim 6, further comprising:

computing via the processor an angular map and a modulus map for the initial frame, wherein the angular map and the modulus map represent color characteristics;

comparing color characteristics of the region of interest with color characteristics of all pixels in the initial frame; and

including in the texture to be removed pixels whose color characteristics are similar to the color characteristics of the region of interest.

8. An encoder that processes texture, the encoder comprising:

a processor;

a first module configured to control the processor to determine texture parameters from an analysis of an original texture associated with a set of frames of video data to yield determined texture parameters;

a second module configured to control the processor to synthesize, based on the determined texture parameters, texture associated with the set of frames of video data by applying, in a first prescribed order, a first set of constraints for unstructured original texture and in a second prescribed order a second set of constraints for structured original texture, wherein the second set of constraints contains more members than the first set of constraints, and wherein the first set of constraints and the second set of constraints each comprises at least one constraint associated with dominant texture orientation and at least one constraint associated with color; and

a third module configured to control the processor to insert the texture into the set of frames based on the determined texture parameters.

9. The encoder of claim 8, further comprising a fourth module configured to control the processor to synthesize the texture according to whether the original texture is busy or smooth.

10. The encoder of claim 8, further comprising:

a fourth module configured to control the processor to select original texture from an initial frame within the set of frames to yield selected original texture;

a fifth module configured to control the processor to remove the selected original texture from the set of frames in the video sequence;

a sixth module configured to control the processor to determine additional texture parameters from an analysis of the removed texture; and

a seventh module configured to control the processor to insert the texture based on the additional texture parameters into the set of frames.

11. The encoder of claim 8, wherein a bit rate of a video sequence that includes the texture is less than a bit rate of an original video sequence.

12. The encoder of claim 8, further comprising a fourth module configured to control the processor to identify a region of interest within texture of an initial frame.

13. The encoder of claim 8, further comprising a fourth module configured to control the processor to compute a mean vector for the region of interest.

14. The encoder of claim 13, further comprising:

a fifth module configured to control the processor to compute an angular map and a modulus map for the initial frame, wherein the angular map and the modulus map represent color characteristics;

a sixth module configured to control the processor to compare color characteristics of the region of interest with color characteristics of all pixels in the initial frame; and

a seventh module configured to control the processor to include in the texture to be removed pixels whose color characteristics are similar to the color characteristics of the region of interest.

15. A non-transitory computer-readable medium storing instructions for controlling a computing device to process texture, the instructions comprising:

determining texture parameters from an analysis of an original texture associated with a set of frames of video data to yield determined texture parameters;

synthesizing, based on the determined texture parameters, texture associated with the set of frames of video data by applying, in a first prescribed order, a first set of constraints for unstructured original texture and in a second prescribed order a second set of constraints for structured original texture, wherein the second set of constraints contains more members than the first set of constraints, and wherein the first set of constraints and the second set of constraints each comprises at least one constraint associated with dominant texture orientation and at least one constraint associated with color; and

inserting the texture into the set of frames based on the determined texture parameters.

16. The non-transitory computer-readable medium of claim 15, wherein the instructions further comprise synthesizing the texture according to whether the original texture is busy or smooth.

17. The non-transitory computer-readable medium of claim 15, wherein the instructions further comprise:

selecting original texture from an initial frame within the set of frames to yield selected original texture;

removing the selected original texture from the set of frames in the video sequence;

determining additional texture parameters from an analysis of the removed texture; and

inserting the texture based on the additional texture parameters into the set of frames.

18. The non-transitory computer-readable medium of claim 15, wherein a bit rate of a video sequence that includes the texture is less than a bit rate of an original video sequence.

19. The non-transitory computer-readable medium of claim 18, the instructions further comprising:

computing an angular map and a modulus map for the initial frame, wherein the angular map and the modulus map represent color characteristics;

comparing color characteristics of the region of interest with color characteristics of all pixels in the initial frame; and

including in the texture to be removed pixels whose color characteristics are similar to the color characteristics of the region of interest.

20. The non-transitory computer-readable medium of claim 15, wherein the instructions further comprise identifying a region of interest within texture of an initial frame.

21. The non-transitory computer-readable medium of claim 20, wherein the instructions further comprise computing a mean vector for the region of interest.