Image resizing for web-based image search

• US 8,081,842 B2
• Filed: 04/30/2008
• Issued: 12/20/2011
• Est. Priority Date: 09/07/2007
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Lu, Yan, Wu, Feng, Qian, Richard, Shum, Heung-Yeung
• Primary Examiner(s): PATEL, KANJIBHAI B
• Application Number:
  US12/113,007
• Publication Number:
  US 20090252431A1
• Time in Patent Office: 1,329 Days
• Field of Search: 382/232, 382/239, 382/240, 382/255, 382/284, 382/298, 382/299, 382/300, 382/305, 382/312, 358/1.2, 358/539, 358/540, 345/606, 345/629, 345/660, 345/670, 345/671, 348/14.13, 375/240.19, 375/240.21
• US Class Current: 382/288
• CPC Class Codes: H04N 19/132 Sampling, masking or truncation of coding units, e.g. adaptive resampling, frame skipping, frame interpolation or high-frequency transform coefficient masking
• CPC Class Codes: H04N 19/14 Coding unit complexity, e.g. amount of activity or edge presence estimation H04N19/146 takes precedence
• CPC Class Codes: H04N 19/172 the region being a picture, frame or field
• CPC Class Codes: H04N 19/587 involving temporal sub-sampling or interpolation, e.g. decimation or subsequent interpolation of pictures in a video sequence
• CPC Class Codes: H04N 19/59 involving spatial sub-sampling or interpolation, e.g. alteration of picture size or resolution
• CPC Class Codes: H04N 19/625 using discrete cosine transform [DCT]
• CPC Class Codes: H04N 19/85 using pre-processing or post-processing specially adapted for video compression
• CPC Class Codes: H04N 19/86 involving reduction of coding artifacts, e.g. of blockiness
• CPC Class Codes: H04N 19/90 using coding techniques not provided for in groups H04N19/10-H04N19/85, e.g. fractals

Abstract

Image resizing for web-based searching is described. In one implementation, a system resizes a user-selected thumbnail image into a larger version of the image that emulates the quality of a large, original image, but without downloading the original image. First, the system extracts resizing parameters when each thumbnail image is created. Then, the system creates a codebook of primitive visual elements extracted from a collection of training images. The primitive visual elements in the codebook provide universal visual parts for reconstructing images. The codebook and a resizing plug-in can be sent once to the user over a background channel. When the user selects a thumbnail image for enlargement, the system resizes the thumbnail image via interpolation and then refines the enlarged image with primitive visual elements from the codebook. The refinement creates an enlarged image that emulates the quality of the large, original image, without downloading the original image.

Claims

1. A method of resizing a thumbnail image to a larger image without downloading the larger image, comprising:

creating a codebook of primitive visual elements extracted from a collection of training images;

downsampling an original image to generate a thumbnail image, including extracting resizing parameters;

transferring the thumbnail image and the resizing parameters to a remote client;

enlarging the thumbnail image to a low-frequency enlarged image at the remote client; and

applying the codebook of primitive visual elements to parts of the low-frequency enlarged image to hallucinate a high-frequency version of the enlarged image for emulating the quality of the original image.

2. The method as recited in claim 1, further comprising blending the low-frequency enlarged image with the high-frequency enlarged image to emulate an image quality level of the original image.

3. The method as recited in claim 1, wherein the resizing parameters only apply to a region of the image.

4. The method as recited in claim 1, wherein the codebook is either created at the remote client or transferred once from a server to the remote client.

5. The method as recited in claim 4, wherein the codebook and an image resizing plug-in are transferred to a browser at the remote client via a background channel.

6. The method as recited in claim 1, wherein the codebook of primitive visual elements is trained from a collection of images associated with statistically popular image search queries on the Internet.

7. The method as recited in claim 1, further comprising compressing the codebook for transfer.

8. The method as recited in claim 1, further comprising maintaining the codebook by transferring incremental updates based on new training images from newly popular image search queries on the Internet.

9. The method as recited in claim 1, wherein enlarging the thumbnail image includes up-sampling the thumbnail image via interpolation, the interpolation consisting of bi-cubic or bilinear interpolation.

10. The method as recited in claim 1, wherein the hallucinating comprises constructing high-frequency regions for the low-frequency enlarged image using high-frequency primitive visual elements in the codebook;

wherein finding the high-frequency primitive visual elements in the codebook includes;

matching low-frequency primitive visual regions of the low-frequency enlarged image with corresponding low-frequency primitive visual elements in the codebook;

for each low-frequency primitive visual element found as a match in the codebook, utilizing a corresponding high-frequency primitive visual element paired with the low-frequency primitive visual element in the codebook.

11. The method as recited in claim 10, further comprising:

applying edge detection on the low-frequency enlarged image to find primitive visual regions of the image;

extracting primitive visual elements at the primitive visual regions;

applying an artificial neural network (ANN) search method for matching the low-frequency primitive visual regions of the low-frequency enlarged image with corresponding low-frequency primitive visual elements in the codebook; and

for non-primal primitive regions in the low-frequency enlarged image, directly using up-sampled image values from the thumbnail image.

12. The method as recited in claim 11, further comprising matching a depth of the ANN search method to available computing resources for complexity control or quality control.

13. The method as recited in claim 1, further comprising applying an image quality enhancement to the enlarged image before the hallucinating.

14. The method as recited in claim 13, wherein the image quality enhancement includes de-blocking filtering to remove JPEG compression artifacts.

15. The method as recited in claim 1, further comprising applying quality control scalability and complexity control scalability to the thumbnail image resizing.

16. A system for enlarging an image without downloading a larger version of the image, comprising:

a thumbnail builder to create a thumbnail image from an original image while extracting resizing parameters;

a codebook trainer to create a codebook of primitive visual elements extracted from a collection of training images, the primitive visual elements providing universal visual parts for reconstructing images;

a resizer for enlarging the thumbnail image via interpolation into an enlarged image as directed by the resizing parameters;

a hallucinator to synthesize high-fidelity parts of the enlarged image by applying the codebook of primitive visual elements; and

a blender to combine the enlarged image with the high-fidelity parts to emulate the original image.

17. The system as recited in claim 16, wherein the hallucinator:

constructs high-frequency regions of the image using high-frequency primitive visual elements in the codebook;

wherein finding the high-frequency primitive visual elements in the codebook includes;

matching low-frequency primitive visual regions of the enlarged image with corresponding low-frequency primitive visual elements in the codebook;

for each low-frequency primitive visual element found as a match in the codebook, utilizing a high-frequency primitive visual element paired in the codebook with the low-frequency primitive visual element.

18. The system as recited in claim 17, further comprising:

an edge detector to find primitive visual regions of the enlarged image associated with visual edges;

an extractor to derive primitive visual elements at the primitive visual regions; and

a mapper to apply an artificial neural network (ANN) search method for matching the derived primitive visual elements of the enlarged image with corresponding primitive visual elements in the codebook, each primitive visual element in the codebook being paired with a corresponding high-fidelity primitive visual element for reconstructing a high quality version of the original image.

19. The system as recited in claim 16, further comprising an updater to refresh the codebook with new pairs of primitive visual elements from new training images associated with new popular image search queries on the Internet.

20. A system for resizing thumbnail images into a large image without downloading the large image, comprising:

means for creating thumbnail images from original images while extracting resizing parameters;

means for creating a codebook of primitive visual elements extracted from a collection of training images, wherein the primitive visual elements provide universal visual parts for reconstructing images;

means for resizing the thumbnail image to an enlarged image via interpolation as directed by the resizing parameters;

means for hallucinating a high-quality version of the enlarged image using the codebook of primitive visual elements; and

means for blending the enlarged image with the high-quality version of the image to emulate the original image.