Example-based image analysis and synthesis using pixelwise correspondence

• US 5,745,668 A
• Filed: 02/12/1997
• Issued: 04/28/1998
• Est. Priority Date: 08/27/1993
• Status: Expired due to Term

• Current Assignee: Massachusetts Institute of Technology
• Original Assignee: Massachusetts Institute of Technology
• Inventors: Beymer, David J., Shashua, Amnon, Poggio, Tomaso
• Primary Examiner(s): Feild, Joseph H.
• Assistant Examiner(s): HONG, STEPHEN S
• Application Number:
  US08/797,993
• Time in Patent Office: 440 Days
• Field of Search: 395/118, 395/127, 395/141, 395/173, 395/175, 395/21, 395/23, 382/156, 382/157, 382/159, 382/201, 382/236, 348/15, 348/17, 348/19
• US Class Current: 345/475
• CPC Class Codes: G06T 9/001 Model-based coding, e.g. wire frame

Abstract

Synthesis of novel images from example images is achieved by determining a pixelwise optical flow among example images, computing a parameter set for a new image, and synthesizing the new image based on the parameter vector and the example images. The parameter set may describe characteristics of the image, in which case the characteristics are applied to a neural network trained with the example images in order to synthesize the novel image. The parameter set may also be an estimate of relative contributions of each of the example images to the new image, in which case the new image may be synthesized by taking a linear combination of the example images, weighted by the parameter set. In one embodiment, both the example and the new images are of the same object. In another embodiment, the example image represents a first object while the new image represents a second object. In yet another embodiment, a set of target images is synthesized from a single target image and a set of source images. Analysis of existing images to determine image parameters is achieved by determining parameters for the existing image based on comparison of the existing image with an image set for which parameters are known.

Claims

1. A method of synthesizing a new image from a plurality of example images, the example images and the new image each being comprised of a plurality of pixels and characterized by a parameter set, the method comprising:

analyzing said example images to determine pixelwise vector representations of said example images, the pixelwise vector representations providing pixel-by-pixel positions of said sample images with respect to a reference;

determining a desired parameter set for said new image; and

synthesizing said new image from said example images using said pixelwise vector representations and said desired parameter set.

2. A method as in claim 1, wherein analyzing said example images to determine pixelwise vector representations includes determining pixelwise optical flows among each of the example images and the reference, and wherein the reference is derived from at least one of said example images.

3. A method as in claim 1, the method further comprising training a neural network with deformation information derived from optical flow applied to each of the example images such that subsequent application of the desired parameter set to the neural network results in said synthesizing.

4. A method of synthesizing a new image from a first example image and a second example image, the example images and the new image each being comprised of a plurality of pixels, the method comprising:

analyzing said first example image and said second example image to determine a pixelwise optical flow from said first example image to said second example image, the pixelwise optical flow providing a pixel-by-pixel relative position of said second example image with respect to said first example image;

determining a desired parameter set for said new image; and

synthesizing said new image from said example images using said optical flow and said parameter set.

5. A method of transmitting images from a transmitting location to a receiving location, comprising:

transmitting a subset of said images as training images from said transmitting location to said receiving location;

determining a pixelwise optical flow for a subsequent image, the pixelwise optical flow providing a pixel-by-pixel position of said subsequent image with respect to a reference;

determining a parameter set for said subsequent image using the pixel-by-pixel position;

transmitting said parameter set from said transmitting location to said receiving location; and

synthesizing a replica of said subsequent image at said receiving location using said training images and said transmitted parameter set.

6. A method as in claim 5, wherein said parameter set defines characteristics of said subsequent image.

7. A method as in claim 5, wherein said parameter set defines relative contributions of said training images, and wherein said replica of said subsequent images is synthesized by taking a linear combination of said training images weighted by said parameter set.

8. A method of synthesizing a new target image by mapping image parameters from a source image set to a target image set, comprising:

defining a first set of image parameters for said source image set by using a subset of said source image set as a source training subset;

mapping said first set of image parameters to a second set of image parameters by using a subset of said target image set as a target training subset;

determining image parameters of a first image in said source image set based on pixelwise optical flow among ones of said source image set, said pixelwise optical flow providing a pixel-by-pixel position of said first image with respect to a reference; and

synthesizing said new target image from said target image set by applying said determined image parameters of said first image to said target image set.

9. A method of synthesizing a novel image from a single example image, comprising:

determining a mapping for pixelwise image changes by comparing prototype images in a prototype image set;

determining a pixelwise correspondence between said example image and one of said prototype images using pixelwise optical flow; and

synthesizing said novel image responsive to the results of said determining a pixelwise correspondence and responsive to said determining a mapping.

10. An image processing system, comprising:

an optical flow processor adapted to determine pixelwise optical flows among a plurality of applied images, the pixelwise optical flows providing pixel-by-pixel positions of said applied images with respect to a reference; and

an image synthesizer coupled to said optical flow processor and adapted to synthesize novel images using said pixel-by-pixel positions.

11. An image processing system as in claim 10, wherein said image synthesizer is implemented as a neural network.

12. An image processing system as in claim 10, wherein said novel images are defined by parameter vectors and wherein said image synthesizer uses said parameter vectors.

13. An image processing system, comprising:

an optical flow processor adapted to determine pixelwise optical flows among a plurality of applied images; and

an image analyzer coupled to said optical flow processor and adapted to estimate parameters of applied novel images responsive to said pixelwise optical flows.

14. An image processing system as in claim 13, wherein said image analyzer is implemented as a neural network.

15. An image processing system, comprising:

a first optical flow processor adapted to determine first pixelwise optical flows among a plurality of applied source images, the first pixelwise optical flows providing pixel-by-pixel positions of said applied source images with respect to a reference;

an image analyzer coupled to said first optical flow processor and adapted to estimate parameters of applied novel images using said pixel-by-pixel positions; and

an image synthesizer coupled to said second image analyzer and adapted to synthesize novel images using said estimated parameters.

16. An image processing system as in claim 15, wherein said source images represent a first object and said novel images represent a second object.

17. An image processing system as in claim 15, wherein said image synthesizer is remotely located from said image analyzer.

18. An image processing system as in claim 15, further comprising a second optical flow processor adapted to determine second pixelwise optical flows among a plurality of applied target images, wherein said image synthesizer is also adapted to synthesize novel images responsive to said second pixelwise optical flows.

19. A system as in claim 18, wherein said source images and said target images represent a single object.

20. A system as in claim 18, wherein said source images represent a first object and said target images represent a second object.