Image analysis and synthesis networks using shape and texture information

• US 5,774,129 A
• Filed: 06/07/1995
• Issued: 06/30/1998
• Est. Priority Date: 06/07/1995
• Status: Expired due to Term

• Current Assignee: Massachusetts Institute of Technology
• Original Assignee: Massachusetts Institute of Technology
• Inventors: Jones, Michael, Vetter, Thomas, Beymer, David, Poggio, Tomaso
• Primary Examiner(s): Nguyen, Phu K.
• Application Number:
  US08/486,637
• Time in Patent Office: 1,119 Days
• Field of Search: 395/141, 395/118, 395/100, 395/142, 395/119, 395/143, 395/135
• US Class Current: 345/441
• CPC Class Codes: G06T 11/00 2D [Two Dimensional] image generation
• CPC Class Codes: G06T 9/001 Model-based coding, e.g. wire frame

Abstract

An image processing system (100) analyzes novel images in one of three modes. In a first mode, the system learns a model based on prototype images, and uses the model to characterize the shape of a novel image as a pixel-by-pixel linear combination of the prototypes. In a second mode, both shape and texture are used to characterize the novel image as a linear combination of the prototypes. In a third mode, shape information is modeled by a linear combination of prototypes and texture information is modeled by an optical flow, and processing of shape and texture information is interleaved. Novel images are synthesized by analysis of input images using transformed prototypes.

Claims

1. A method of synthesizing and operating a network to characterize a new physical image relative to a plurality of prototypes, the prototypes having shapes but no texture, the process comprising:

(a) synthesizing the network by;

(i) determining pointwise prototype correspondences among the plurality of prototypes;

(ii) creating a model responsive to the pointwise prototype correspondences, the model being a parameterized pointwise weighted linear combination of the shapes of the prototypes;

(b) operating the network by;

(i) filtering and normalizing the new image;

(ii) determining model parameters to iteratively minimize a pointwise distance metric from the new image to the model, the pointwise distance metric being responsive to shape, thereby establishing new image pointwise correspondences between the new image and each said prototype; and

(iii) characterizing the new image by the model parameters and the new image pointwise correspondences.

2. A method of synthesizing and operating a network to characterize a new physical image relative to a plurality of prototypes, the prototypes having shapes and textures, the process comprising:

(a) synthesizing the network by;

(i) determining pointwise prototype correspondences among the plurality of prototypes;

(ii) creating a model responsive to the pointwise prototype correspondences, the model being a parameterized pointwise weighted linear combination of the shapes and textures of the prototypes;

(b) operating the network by;

(i) filtering and normalizing the new image;

(ii) determining model parameters to iteratively minimize a pointwise distance metric from the new image to the model, the pointwise distance metric being responsive to shape and texture, thereby establishing new image pointwise correspondences between the new image and each said prototype; and

(iii) characterizing the new image by the model parameters and the new image pointwise correspondences.

3. A method of synthesizing and operating a network to characterize a new physical image having shape and texture relative to a plurality of prototypes, the prototypes also having shapes and textures, the process comprising:

(a) synthesizing the network by;

(i) determining pointwise prototype correspondences among the plurality of prototypes;

(ii) creating a model as a linear combination of geometrically normalized textures derived from the prototypes, the model having a model shape;

(b) operating the network by;

(i) filtering and normalizing the new image;

(ii) parameterizing the new image shape responsive to optical flow transformation of the new image shape to the model shape;

(iii) parameterizing the new image texture responsive to pointwise combinations of the textures of the prototypes; and

(iv) iterating (b)(ii) and (b)(iii).

4. A method of synthesizing and operating a network to create a novel physical image from an applied physical image having shape and texture relative to a plurality of prototypes, the prototypes also having shapes and textures, the applied physical image and the prototypes each corresponding to one of a plurality of poses, the process comprising:

(a) synthesizing the network by;

(i) determining pointwise prototype correspondences among the plurality of prototypes separately for each of said plurality of poses;

(ii) creating a model as a linear combination of geometrically normalized textures derived from the prototypes for a first one of said plurality of poses;

(b) operating the network by;

(i) filtering and normalizing the applied physical image;

(ii) parameterizing the applied physical image shape responsive to optical flow transformation of the applied image shape to the model shape;

(iii) parameterizing the applied image texture responsive to pointwise combinations of the textures of the prototypes;

(iv) iterating (b)(ii) and (b)(iii); and

(v) implementing a desired transformation of the applied physical image into the novel physical image by using parameters from the result of b(iv) applied to a subset of the prototypes associated with a desired second one of said plurality of poses.

5. An image processing system for characterizing an applied image of a physical object relative to a plurality of prototypes, the system comprising:

an imaging device for producing a signal responsive to the applied image;

a prototype storage device, the prototype storage device containing the plurality of prototypes;

a shape engine operatively connected to the imaging device and to the prototype storage device, the shape engine having a synthesis mode of determining pointwise prototype correspondence among the plurality of prototypes and creating a model responsive to the pointwise prototype correspondences, the shape engine further having an operating mode of filtering and normalizing the signal to produce a preprocessed signal, determining model parameters to minimize a pointwise distance metric from the applied image to the model responsive to the preprocessed signal, establishing pointwise correspondences responsive to the plurality of prototypes and the preprocessed signal; and

an output engine operatively connected to the shape engine, the output engine producing output responsive to the model parameters and the pointwise correspondences.

6. A system as in claim 5, further comprising a texture engine, the texture engine cooperating with the shape engine to produce the model parameters.

7. A system as in claim 5, wherein the output is a novel representation of the physical object.

8. A system as in claim 5, wherein the output engine disengages access controls to a secured area responsive to predetermined characteristics of the model parameters and pointwise correspondences.

9. An image processing system for characterizing an applied image of a physical object relative to a plurality of prototypes, the system comprising:

an imaging device for producing a signal responsive to the applied image;

a prototype storage device, the prototype storage device containing the plurality of prototypes;

texture engine for processing texture information responsive to the signal;

a shape engine operatively connected to the imaging device, to the prototype storage device, and to the texture engine, the shape engine and the texture engine cooperating to provide a synthesis mode of determining pointwise prototype correspondences among the plurality of prototypes and creating a model responsive to linear combinations of geometrically normalized textures derived from the prototypes, the shape engine and the texture engine further cooperating to provide an operating mode of filtering and normalizing the signal to produce a preprocessed signal, and iteratively parameterizing image shape and texture responsive to the plurality of prototypes and the preprocessed signal; and

an output engine operatively connected to the shape engine, the output engine producing output responsive to the model parameters and the pointwise correspondences.

10. A system as in claim 9, wherein the output is a novel representation of the physical object.