Computing a higher resolution image from multiple lower resolution images using model-based, robust bayesian estimation

• US 20060002635A1
• Filed: 06/30/2004
• Published: 01/05/2006
• Est. Priority Date: 06/30/2004
• Status: Active Grant

• Current Assignee: Matsushita Electric Industrial Company Limited (Panasonic Holdings Corporation), Tahoe Research Limited (f/k/a Learndale Limited) (Vector Capital Corporation)
• Original Assignee: Intel Corporation
• Inventors: Ettinger, Scott M., Nestares, Oscar, Haussecker, Horst W.
• Granted Patent: US 7,447,382 B2
• Time in Patent Office: Days
• US Class Current: 382/299
• CPC Class Codes: G06T 3/4053 based on super-resolution, i.e. the output image resolution being higher than the sensor resolution

Abstract

A result higher resolution (HR) image of a scene given multiple, observed lower resolution (LR) images of the scene is computed using a Bayesian estimation image reconstruction methodology. The methodology yields the result HR image based on a Likelihood probability function that implements a model for the formation of LR images in the presence of noise. This noise is modeled by a probabilistic, non-Gaussian, robust function. Other embodiments are also described and claimed.

Claims

1. A method for image processing, comprising:

computing a result higher resolution (HR) image of a scene given a plurality of observed lower resolution (LR) images of the scene using a Bayesian estimation image reconstruction methodology,

wherein the methodology yields the result HR image based on a Likelihood probability function that implements a model for the formation of LR images in the presence of noise,

and wherein the methodology models the noise by a probabilistic, non-Gaussian, robust function.

2. The method of claim 1 wherein the methodology yields the result HR image based on maximizing posteriori probability, that is the conditional probability of an unknown HR image given the observed LR images,

wherein the methodology yields the result HR image based on combining the Likelihood function with a Prior probability function,

the Prior function indicating which HR images are probable.

3. The method of claim 2 wherein the modeling of the noise by the robust function causes the role of a statistical outlier pixel in an observed LR image to be downplayed when computing a trial HR image based on the Likelihood function, so that a computed Likelihood probability for said observed LR image is higher than if the noise were modeled by a Gaussian function.

4. The method of claim 3 wherein:

the robust function is a Huber function.

5. The method of claim 4 wherein:

the robust function is a Tukey function.

6. The method of claim 2 wherein the Prior function used in the methodology implements one of a Gaussian Markov Random Field (MRF), a Huber MRF, and Tukey MRF to indicate the probability of which pixels in an image take on which values.

7. The method of claim 3 further comprising:

setting the robust function to a different function depending on whether the observed LR images have relatively few and relatively many outliers,

wherein the methodology estimates a shape of the robust function according to the availability of training data.

8. The method of claim 3 wherein the methodology estimates a shape of the robust function by selecting a scale factor,

wherein if there is sufficient training data in the form of one or more ground truth HR images and their corresponding LR images, the scale factor is estimated from samples obtained in computing an error between the observed LR images of the scene and their projections from the ground truth HR images.

9. The method of claim 8 wherein if there is insufficient training data, the scale factor is estimated by (1) taking a current, trial HR image of an iterative, maximum a posteriori estimation process as a ground truth HR image, and (2) a robust estimator for the scale factor.

10. The method of claim 3 further comprising defining the Prior function based upon computer aided design models for structures captured in the observed LR images.

11. The method of claim 3 wherein the Prior function is based on a non-Gaussian, robust Markov Random Field.

12. A system comprising:

a processor; and

memory having instructions that, when executed by the processor, generate a result higher resolution (HR) image of a scene based on a plurality of lower resolution (LR) images of the scene, using a Bayesian image reconstruction methodology based on a Likelihood probability function that implements a model for LR image formation that includes additive noise, and wherein the methodology models the additive noise by a probabilistic, non-Gaussian, robust function.

13. The system of claim 12 wherein the processor and memory are part of one of a desktop and notebook personal computer,

and wherein the memory stores further instructions that when executed by the processor obtain the plurality of LR images based on images downloaded into the personal computer from a digital camera.

14. The system of claim 13 wherein the instructions are to obtain the plurality of LR images as video, based on video downloaded into the personal computer from the digital camera, and wherein a plurality of result HR images are to be generated as HR video of the scene.

15. The system of claim 14 wherein the instructions are to generate the HR video in a high definition television, HDTV, format.

16. The system of claim 12 wherein the instructions are to yield the result HR image based on maximizing posteriori probability which is a combination of the Likelihood function and a Prior probability function, the Prior function indicating which HR images are probable.

17. An article of manufacture comprising:

a machine accessible medium containing instructions that, when executed, cause a machine to compute a result higher resolution (HR) image of a scene given a plurality of observed lower resolution (LR) images of the scene using a Bayesian image reconstruction methodology, wherein the methodology yields the result HR image based on a Likelihood probability function that implements a model for LR image formation in the presence of noise,

and wherein the methodology models the noise by a weighting function that causes the role of a statistical outlier pixel in an observed LR image to be downplayed when computing a trial HR image based on the Likelihood function, so that a computed Likelihood probability for said observed LR image given the trial HR image is higher than if the noise were modeled by a Gaussian function.

18. The article of manufacture of claim 17 wherein the instructions are such that the methodology yields the result HR image based on maximizing posteriori probability, that is the conditional probability of an unknown HR image given observations about LR images,

wherein the methodology yields the result HR image based on combining the Likelihood function with a Prior probability function,

the Prior function indicating which HR images are probable.

19. The article of manufacture of claim 18 wherein the medium includes further instructions that set the weighting function to a different function depending on whether the plurality of observed LR images have relatively few and relatively many outliers,

and wherein the methodology estimates a shape of the weighting function according to the availability of training data.

20. The article of manufacture of claim 18 wherein the instructions are such that the methodology estimates a shape of the weighting function by selecting a scale factor,

wherein if there is sufficient training data in the form of one or more ground truth HR images and their corresponding LR images, the scale factor is estimated from samples obtained in computing an error between the plurality of observed LR images of the scene and their projections from the ground truth HR images.

21. The article of manufacture of claim 20 wherein the instructions are such that if there is insufficient training data, the scale factor is estimated by (1) taking a current, trial HR image of an iterative, maximum a posteriori estimation process as a ground truth HR image, and (2) a robust estimator for the scale factor.