Learning-Based Image Compression

• US 20090067491A1
• Filed: 09/07/2007
• Published: 03/12/2009
• Est. Priority Date: 09/07/2007
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Li, Shipeng, Wu, Feng, Sun, Xiaoyan
• Granted Patent: US 8,223,837 B2
• Time in Patent Office: Days
• US Class Current: 375/240.30
• CPC Class Codes: H04N 19/132 Sampling, masking or truncation of coding units, e.g. adaptive resampling, frame skipping, frame interpolation or high-frequency transform coefficient masking
• CPC Class Codes: H04N 19/14 Coding unit complexity, e.g. amount of activity or edge presence estimation H04N19/146 takes precedence
• CPC Class Codes: H04N 19/587 involving temporal sub-sampling or interpolation, e.g. decimation or subsequent interpolation of pictures in a video sequence
• CPC Class Codes: H04N 19/59 involving spatial sub-sampling or interpolation, e.g. alteration of picture size or resolution
• CPC Class Codes: H04N 19/85 using pre-processing or post-processing specially adapted for video compression
• CPC Class Codes: H04N 19/90 using coding techniques not provided for in groups H04N19/10-H04N19/85, e.g. fractals

Abstract

Learning-based image compression is described. In one implementation, an encoder possessing a first set of learned visual knowledge primitives excludes visual information from an image prior to compression. A decoder possessing an independently learned set of visual knowledge primitives synthesizes the excluded visual information into the image after decompression. The encoder and decoder are decoupled with respect to the information excluded at the encoder and the information synthesized at the decoder. This results in superior data compression since the information excluded at the encoder is dropped completely and not transferred to the decoder. Primitive visual elements synthesized at the decoder may be different than primitive visual elements dropped at the encoder, but the resulting reconstituted image is perceptually equivalent to the original image.

Claims

1. A method, comprising:

detecting a similarity between visual information in an image and information in a first set of primitive visual elements for creating images;

removing the visual information from the image; and

compressing the image.

2. The method as recited in claim 1, wherein removing visual information includes:

determining primal sketch regions in the image;

filtering the primal sketch regions to obtain primitive patches; and

removing the primitive patches from the image when the primitive patches are associated with the first set of primitive visual elements.

3. The method as recited in claim 1, further comprising one of storing the compressed image or transmitting the compressed image as a bitstream.

4. The method as recited in claim 1, further comprising:

receiving the compressed image;

decompressing the compressed image; and

synthesizing the visual information that was removed, wherein the synthesizing uses a second set of primitive visual elements for creating images.

5. The method as recited in claim 4, wherein the first set of primitive visual elements is learned from first training images and the second set of primitive visual elements is learned from second training images that are different from the first training images;

and

wherein learning the first set of primitive visual elements and learning the second set of primitive visual elements use a substantially similar learning strategy.

6. The method as recited in claim 4, wherein removing visual information and compressing the image are performed by an encoder;

and

wherein decompressing the compressed image and synthesizing the visual information that was removed are performed by an encoder.

7. The method as recited in claim 4, further comprising:

down-sampling the image minus the removed visual information prior to compressing the image; and

interpolating the image by up-sampling after decompressing the image.

8. The method as recited in claim 5, wherein learning a set of primitive visual elements further comprises:

applying distortion to each original training image to obtain a distorted image simulating a compressed version of the original training image;

subtracting the distorted image from the original training image to obtain a differential image;

detecting visual edges in the distorted image;

based on the visual edges, determining primal sketch regions of the distorted image and of the differential image;

filtering the primal sketch regions of the distorted image to obtain distorted primitive patches;

filtering the primal sketch regions of the differential image to obtain differential primitive patches; and

wherein the set of primitive visual elements includes primitive patch pairs, each primitive patch pair including one distorted primitive patch and one differential primitive patch, each pair corresponding to the same primal sketch region.

9. The method as recited in claim 8, wherein synthesizing the visual information includes:

finding a primal sketch region in the decompressed and interpolated image;

extracting a distorted primitive patch from the primal sketch region;

finding a similar distorted primitive patch in the second set of primitive visual elements;

retrieving the differential primitive patch from the pair associated with the found distorted primitive patch; and

blending the differential primitive patch at the corresponding primal sketch region in the interpolated image to obtain a reconstructed version of part of the image.

10. The method as recited in claim 8, wherein the distorted primitive patch comprises a relatively low-resolution patch from low-pass filtering and the differential primitive patch comprises information to reconstruct a part of the image corresponding to the distorted primitive patch.

11. The method as recited in claim 4, wherein the visual information removed before compression of the image is decoupled from the visual information synthesized after decompression of the image, the decoupling comprising excluding from one or more decoders the visual information removed at an encoder.

12. A method, comprising:

learning a first set of primitive visual element pairs from a first set of training images via a learning strategy, wherein the first set is used to exclude information from an image prior to data compression; and

learning a second set of primitive visual element pairs from a second set of training images via the learning strategy, wherein the second set is used to synthesize the excluded information for reconstituting the image after data decompression.

13. The method as recited in claim 12, wherein the primitive visual element pairs of the first set are different than the primitive visual element pairs of the second set.

14. The method as recited in claim 12, wherein a decoder possessing the second set has no knowledge of the primitive visual element pairs in the first set that were used to exclude image information at an encoder.

15. The method as recited in claim 14, wherein the image reconstituted via the second set is perceptually equivalent to the image from which the information was excluded via the first set.

16. The method as recited in claim 15, wherein the image and the reconstituted image are constructed at least in part from different primitive visual elements and are perceptually equivalent to each other.

17. A system, comprising:

an encoder possessing a first set of primitive visual knowledge elements for excluding information from an image prior to compression; and

a decoder possessing a second set of primitive visual knowledge elements for synthesizing excluded information into the image after decompression.

18. The system as recited in claim 17, wherein the decoder has no knowledge of the primitive knowledge elements in the first set at the encoder;

and

the encoder has no knowledge of the primitive visual knowledge elements in the second set at the decoder.

19. The system as recited in claim 17, wherein the encoder strips high-pass image information from primal sketch regions of the image prior to compression based on the first set;

and

wherein the decoder senses missing high-pass information at the primal sketch regions after decompression and provides the missing information from the second set.

20. The system as recited in claim 19, wherein the information supplied from the second set is different on a pixel-wise basis than the high-pass information stripped by the encoder;

and

wherein the information from the second set is perceptually equivalent to the high-pass information stripped by the encoder.