Learning-based image compression

• US 8,223,837 B2
• Filed: 09/07/2007
• Issued: 07/17/2012
• Est. Priority Date: 09/07/2007
• Status: Expired due to Fees

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Sun, Xiaoyan, Wu, Feng, Li, Shipeng
• Primary Examiner(s): Shiferaw, Eleni
• Assistant Examiner(s): SIMS, JING F
• Application Number:
  US11/851,653
• Publication Number:
  US 20090067491A1
• Time in Patent Office: 1,775 Days
• Field of Search: 37524001-24006, 37524012-24016, 375/240.26, 37524008-2401, 348/700
• US Class Current: 375/240.1
• CPC Class Codes: H04N 19/132 Sampling, masking or truncation of coding units, e.g. adaptive resampling, frame skipping, frame interpolation or high-frequency transform coefficient masking
• CPC Class Codes: H04N 19/14 Coding unit complexity, e.g. amount of activity or edge presence estimation H04N19/146 takes precedence
• CPC Class Codes: H04N 19/587 involving temporal sub-sampling or interpolation, e.g. decimation or subsequent interpolation of pictures in a video sequence
• CPC Class Codes: H04N 19/59 involving spatial sub-sampling or interpolation, e.g. alteration of picture size or resolution
• CPC Class Codes: H04N 19/85 using pre-processing or post-processing specially adapted for video compression
• CPC Class Codes: H04N 19/90 using coding techniques not provided for in groups H04N19/10-H04N19/85, e.g. fractals

Abstract

Learning-based image compression is described. In one implementation, an encoder possessing a first set of learned visual knowledge primitives excludes visual information from an image prior to compression. A decoder possessing an independently learned set of visual knowledge primitives synthesizes the excluded visual information into the image after decompression. The encoder and decoder are decoupled with respect to the information excluded at the encoder and the information synthesized at the decoder. This results in superior data compression since the information excluded at the encoder is dropped completely and not transferred to the decoder. Primitive visual elements synthesized at the decoder may be different than primitive visual elements dropped at the encoder, but the resulting reconstituted image is perceptually equivalent to the original image.

Claims

1. A method, comprising:

under control of one or more computing systems comprising one or more processors,

detecting a similarity between visual information in an image and information in a first set of primitive visual elements for creating images;

determining primal sketch regions in the image, the primal sketch regions including multiple associated primitive patches;

filtering the primal sketch regions to obtain primitive patches of the multiple associated primitive patches;

removing the primitive patches from the image when the primitive patches are associated with the first set of primitive visual elements;

compressing the image

receiving the compressed image;

decompressing the compressed image; and

synthesizing the visual information that was removed, wherein the synthesizing uses a second set of primitive visual elements for creating images, wherein;

the first set of primitive visual elements is learned from first training images and the second set of primitive visual elements is learned from second training images that are different from the first training images, and

learning the first set of primitive visual elements and learning the second set of primitive visual elements use same learning strategy.

2. The method as recited in claim 1, further comprising one of storing the compressed image or transmitting the compressed image as a bitstream.

3. The method as recited in claim 1, wherein learning a set of primitive visual elements further comprises:

applying distortion to each original training image to obtain a distorted image simulating a compressed version of the original training image;

subtracting the distorted image from the original training image to obtain a differential image;

detecting visual edges in the distorted image;

based on the visual edges, determining primal sketch regions of the distorted image and of the differential image;

filtering the primal sketch regions of the distorted image to obtain distorted primitive patches; and

filtering the primal sketch regions of the differential image to obtain differential primitive patches, wherein the set of primitive visual elements includes primitive patch pairs, each primitive patch pair including one distorted primitive patch and one differential primitive patch, each pair corresponding to a primal sketch region.

4. The method as recited in claim 3, wherein synthesizing the visual information includes:

finding a primal sketch region in the decompressed and interpolated image;

extracting a distorted primitive patch from the primal sketch region;

finding a distorted primitive patch in the second set of primitive visual elements;

retrieving the differential primitive patch from the pair associated with the found distorted primitive patch; and

blending the differential primitive patch at the corresponding primal sketch region in the interpolated image to obtain a reconstructed version of part of the image.

5. The method as recited in claim 3, wherein the distorted primitive patch comprises a low-resolution patch from low-pass filtering and the differential primitive patch comprises information to reconstruct a part of the image corresponding to the distorted primitive patch.

6. The method as recited in claim 1, wherein removing visual information and compressing the image are performed by an encoder;

and

wherein decompressing the compressed image and synthesizing the visual information that was removed are performed by a decoder.

7. The method as recited in claim 1, further comprising:

down-sampling the image minus the removed visual information prior to compressing the image; and

interpolating the image by up-sampling after decompressing the image.

8. The method as recited in claim 1, wherein the visual information removed before compression of the image is decoupled from the visual information synthesized after decompression of the image, the decoupling comprising excluding from one or more decoders the visual information removed at an encoder.

9. A method, comprising:

under control of one or more computing systems comprising one or more processors,

learning a first set of primitive visual element pairs from a first group of training images via a learning strategy, wherein the first set is used to exclude information from an image prior to data compression; and

learning a second set of primitive visual element pairs from a second group of training images via the learning strategy, wherein;

the second set is used to synthesize the excluded information for reconstituting the image after data decompression,

a decoder possessing the second set does not use knowledge of the primitive visual element pairs in the first set that are used to exclude image information at an encoder, and

the image reconstituted via the second set is equivalent to the image from which the information was excluded via the first set.

10. The method as recited in claim 9, wherein the primitive visual element pairs of the first set are different than the primitive visual element pairs of the second set.

11. The method as recited in claim 9, wherein the image and the reconstituted image are constructed at least in part from different primitive visual elements and are equivalent to each other.

12. A system, comprising:

memory;

one or processors coupled to the memory;

an encoder possessing a first set of primitive visual knowledge elements for excluding information from an image prior to compression; and

a decoder possessing a second set of primitive visual knowledge elements for synthesizing excluded information into the image after decompression, wherein;

the decoder does not use knowledge of the primitive knowledge elements in the first set at the encoder; and

the encoder does not use knowledge of the primitive visual knowledge elements in the second set at the decoder, wherein the encoder strips high-pass image information from primal sketch regions of the image prior to compression based on the first set; and

wherein the decoder senses missing high-pass information at the primal sketch regions after decompression and provides the missing information from the second set.

13. The system as recited in claim 12, wherein the information supplied from the second set is different on a pixel-wise basis than the high-pass information stripped by the encoder;

and

wherein the information from the second set is equivalent to the high-pass information stripped by the encoder.