ROBUST RECOVERY OF TRANSFORM INVARIANT LOW-RANK TEXTURES

• US 20120133779A1
• Filed: 11/29/2010
• Published: 05/31/2012
• Est. Priority Date: 11/29/2010
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Ma, Yi, Zhang, Zhengdong, Liang, Xiao
• Granted Patent: US 8,463,073 B2
• Time in Patent Office: Days
• US Class Current: 348/175
• CPC Class Codes: G06F 18/2136 based on sparsity criteria, e.g. with an overcomplete basis
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/20012 Locally adaptive
• CPC Class Codes: G06T 2207/20016 Hierarchical, coarse-to-fine, multiscale or multiresolution image processing; Pyramid transform
• CPC Class Codes: G06T 2207/20021 Dividing image into blocks, subimages or windows
• CPC Class Codes: G06T 7/40 Analysis of texture depth or shape recovery from texture G06T7/529
• CPC Class Codes: G06V 10/247 by affine transforms, e.g. correction due to perspective effects; Quadrilaterals, e.g. trapezoids
• CPC Class Codes: G06V 10/462 Salient features, e.g. scale invariant feature transforms [SIFT]
• CPC Class Codes: G06V 10/7715 Feature extraction, e.g. by transforming the feature space, e.g. multi-dimensional scaling [MDS]; Mappings, e.g. subspace methods

Abstract

A “Transform Invariant Low-Rank Texture” (TILT) Extractor, referred to as a “TILT Extractor” accurately extracts both textural and geometric information defining regions of low-rank planar patterns from 2D images of a scene, thereby enabling a large range of image processing applications. Unlike conventional feature extraction techniques that rely on point-based features, the TILT Extractor extracts texture regions from an image and derives global correlations or transformations of those regions in 3D (e.g., transformations including translation, rotation, reflection, skew, scale, etc.). These image domain transformations inherently provide information relative to an automatically determinable camera viewing direction. In other words, the TILT Extractor extracts low-rank regions and geometric correlations describing domain transforms of those regions relative to arbitrary camera viewpoints. The TILT Extractor also identifies sparse error in image intensity or other color channels resulting from noise, occlusions or other artifacts, thereby allowing elimination or reduction of such errors in images.

Claims

1. A method for extracting textural and geometric information from an image, comprising steps for:

receiving an input image;

selecting an image channel from available channels of a color space of the input image;

windowing the input image;

for each window of the input image, iterating steps for;

normalizing the selected image channel,

computing a Jacobian matrix from the normalized image channel with respect to a current geometric transformation,

recovering a current low-rank texture from the normalized image channel by using an iterative convex optimization process to update the current geometric transformation and find the low-rank texture having a fewest number of nonzero entries that is equivalent to the normalized image channel relative to the current geometric transformation until local convergence of the geometric transformation is achieved, and

repeating the iterated steps for each window until global convergence of the geometric transformation is achieved; and

outputting the current low-rank texture and the current geometric transformation.

2. The method of claim 1 wherein the iterative convex optimization process further uses a sparse error to update the current geometric transformation and find the low-rank texture having a fewest number of nonzero entries that is equivalent to the normalized image channel relative to the current geometric transformation and a sparse error component extracted from the normalized image channel until local convergence of the geometric transformation is achieved.

3. The method of claim 2 further comprising eliminating the sparse error component from the outputted current low-rank texture.

4. The method of claim 2 wherein the iterative convex optimization process is performed by iteratively solving:

5. The method of claim 2 wherein the sparse error component represents corruptions in the selected image channel of the input image.

6. The method of claim 2 further comprising weighting the sparse error component for use in the convex optimization process.

7. The method of claim 6 wherein the iterative convex optimization process is performed by iteratively solving:

8. The method of claim 1 wherein the color space of the input image includes a brightness channel, and wherein the brightness channel is the selected image channel.

9. A system for deriving low-rank textures from an input image, comprising:

a device for receiving an input image;

a device for windowing the input image;

for each window of the input image, a device for iterating steps for;

normalizing an intensity channel of the input image,

computing a Jacobian matrix from the normalized intensity channel with respect to a current geometric transformation,

recovering a current low-rank texture from the normalized intensity channel by using an iterative convex optimization process to update the current geometric transformation and a current sparse error component of the input image by finding the current low-rank texture having a fewest number of nonzero entries that is equivalent to the normalized intensity channel relative to the current geometric transformation and the current sparse error component of the input image until local convergence of the geometric transformation is achieved, and

repeating the iterated steps for each window until global convergence of the geometric transformation is achieved; and

a device for outputting the current low-rank texture, the current geometric transformation, and the current sparse error component.

10. The system of claim 9 further comprising a device for eliminating the sparse error component from the outputted current low-rank texture.

11. The system of claim 9 wherein the iterative convex optimization process is performed by iteratively solving:

12. The system of claim 9 wherein the sparse error component represents corruptions in the intensity channel of the input image.

13. The system of claim 9 further comprising a device for weighting the sparse error component for use in the iterative convex optimization process.

14. The system of claim 13 wherein the iterative convex optimization process is performed by iteratively solving:

15. A computer-readable storage device having computer executable instructions stored therein for extracting low-rank textural and geometric information and a sparse error component from an image, said instructions comprising:

a device for receiving an input image;

a device for windowing the input image;

for each window of the input image, a device for iterating steps for;

normalizing an intensity channel of the input image,

computing a Jacobian matrix from the normalized intensity channel with respect to a current geometric transformation,

recovering a current low-rank texture from the normalized intensity channel by using an iterative convex optimization process to update the current geometric transformation and a current sparse error component of the input image by iteratively solving;

16. The computer-readable storage device of claim 15 wherein the sparse error component represents corruptions in the input image.

17. The computer-readable storage device of claim 15 further comprising removing the sparse error component from the outputted current low-rank texture.

18. The computer-readable storage device of claim 15 further comprising extracting corresponding low-rank textures from a plurality of input images of a same scene captured from a plurality of different viewpoints.

19. The computer-readable storage device of claim 16 further comprising instruction for performing a super-resolution process by combining the corresponding low-rank textures to create a single enhanced resolution version of the corresponding low-rank textures.

20. The computer-readable storage device of claim 15 wherein the input image is a known pattern having known deformations, and wherein the current geometric transformation for each window of the input image is used in combination with the known deformations of the known pattern to calibrate a camera used to capture the input image.