Generating super resolution digital images

• US 7,218,751 B2
• Filed: 06/29/2001
• Issued: 05/15/2007
• Est. Priority Date: 06/29/2001
• Status: Active Grant

• Current Assignee: Digimarc Corporation
• Original Assignee: Digimarc Corporation
• Inventors: Reed, Alastair M., Hannigan, Brett T.
• Primary Examiner(s): Wu; Jingge
• Assistant Examiner(s): EDWARDS, PATRICK L
• Application Number:
  US09/895,063
• Publication Number:
  US 20030002707A1
• Time in Patent Office: 2,146 Days
• Field of Search: 382/100, 382/284, 382/287, 382/294, 382/299, 348/36, 348/38
• US Class Current: 382/100
• CPC Class Codes: G06F 18/00 Pattern recognition
• CPC Class Codes: G06T 3/4007 based on interpolation, e.g. bilinear interpolation image demosaicing G06T3/4015; edge-driven or edge-based scaling G06T3/403
• CPC Class Codes: G06T 3/4015 Image demosaicing, e.g. colour filter arrays [CFA] or Bayer patterns

Abstract

Slight camera movement between capture of successive images is advantageously utilized to minimize or eliminate the need to interpolate in order to fill in the “holes” in a Bayer pattern. The captured color values from multiple appropriately positioned images are used to fill these holes. For example, instead of interpolating the value of red for the second pixel position on the first row of a Bayer pattern, an image is selected which is positioned one pixel to the right of the first image, and the red vales from this image are used for the red values of the second pixel on the first line. Values of the pixels in multiple images which are appropriately aligned to each pixel position are averaged to generate a better value for each pixel position. Information carried by a digital watermark (either alone or together with other techniques) is used to determine the alignment of the images. Images are selected which are positioned so that corresponding pixels fall within a specified tolerance from a location in a Bayer pattern. The pixel values of the images which fall within the specified tolerance of each pixel position in a Bayer pattern are selected and used for the alignment.

Claims

1. A method of generating a high resolution image from a set of nearly identical low resolution images, each of which contains a reference signal, the method comprising:

identifying said reference signal in each of said low resolution images to relate the locations of said images,

identifying a first group of said low resolution images having red image information at a location most closely aligned with a first position in said high resolution image;

identifying a second, different, group of said low resolution images having red image information at a location most closely aligned with a second, different, position in said high resolution image;

processing the red image information from said first group of images to generate red data for the first position in said high resolution image; and

processing the red image information from said second group of images to generate red data for the second position in said high resolution image.

2. The method recited in claim 1 wherein the first and second positions correspond to positions in a Bayer square.

3. The method in claim 1 wherein pixels in said low resolution images have colors in accordance with a Bayer square.

4. The method recited in claim 1 wherein said reference signal comprises a watermark signal.

5. The method of claim 1 wherein none of the low resolution images in the first group of images is included in the second group of images.

6. A method of generating a high resolution image from a series of nearly identical low resolution images, each of which contains a reference signal, the method comprising:

reading said reference signal from each of said low resolution images to determine alignment of pixels in the image,

selecting images whose pixels are within a specified tolerance from specified positions, and

combining the selected images to generate a high resolution image,

wherein said reference signal comprises a watermark grid signal.

7. A system for generating a high resolution image from a series of nearly identical relatively low resolution images, the system including:

a watermark reading program for reading a watermark grid signal from each of said low resolution images to determine alignment of the pixels in the low resolution images relative to positions in a Bayer square,

an image selection program for selecting the low resolution images whose pixels are within a specified tolerance from each position in the Bayer square, and

an image combination program for combining the selected low resolution images to generate a high resolution image.

8. A method of generating a high resolution image from a plurality of low resolution images, the method comprising,

capturing a plurality of low resolution electronic images of a subject, the subject defining a hidden reference signal,

using said reference signal to determine alignment of a plurality of said low resolution images, and

combining at least some of said low resolution images into a high resolution image,

wherein said reference signal comprises a watermark grid signal.

9. A method of generating a high resolution image from a plurality of low resolution images, the method comprising:

capturing a series of low resolution images, each of which contains a reference signal,

reading said reference signal from each of said low resolution images,

aligning said low resolution images in accordance with said reference signal, and

combining at least some of said aligned low resolution images into a high resolution image,

wherein said reference signal comprises a watermark grid signal.

10. A method comprising:

capturing first and second frames of similar image data using a common 2D image sensor that is subject to slight movement between frames, said sensor comprising a geometrical pattern of sensor elements, each element sensing light of one of at least first or second colors, wherein certain locations in said 2D sensor do not sense light of the first color, resulting in unknown first color information at various locations in the frames of image data captured by said sensor;

determining position information relating position of the first image frame to the second image frame, said determining including decoding a signal steganographically encoded data in the first image frame, and decoding a signal steganographically encoded in the second image frame; and

generating a composite image frame in which said unknown first color information at one of said various locations in the first frame of image data is mitigated by reference to first color information from the second image frame that is determined to correspond to said one location.

11. The method of claim 10 in which said determining includes correlating pattern data in said first and second image frames.

12. The method of claim 11 in which said determining makes use of signals steganographically encoded in the first and second image frames, and also makes use of correlating visible image content.

13. The method of claim 12 wherein said visible image content comprises fiducial marks provided on a subject being imaged.

14. A method comprising:

capturing a first frame of image data using a 2D sensor, and capturing a second frame of image data using said same 2D sensor;

sensing a hidden watermark signal, useful in determining image rotation and scaling, in each of said frames of image data; and

by reference to said sensed hidden watermark signal, combining data from said first and second frames to produce an enhanced frame of image data.

15. A method of generating a high resolution image from a set of nearly identical low resolution images, the method comprising:

capturing low resolution images from a subject, the subject having previously been deliberately marked with a marker signal to facilitate machine processing of images captured therefrom;

identifying said marker signal in each of said low resolution images to relate locations of said images;

identifying a first group of said low resolution images having image information of a first primary color at a location most closely aligned with a first position in said high resolution image;

identifying a second group of said low resolution images having image information of a second, different, primary color at a location most closely aligned with said first position in said high resolution image;

processing the image information of the first primary color from said first group of images to generate data of said first primary color for the first position in said high resolution image; and

processing the image information of the second primary color from said second group of images to generate data of said second primary color for said first position in said high resolution image.

16. The method of claim 15 wherein none of the low resolution images in the first group of images is included in the second group of images.

17. The method of claim 15 wherein at least one of said low resolution images is included in neither said first nor second groups.

18. A method of generating a high resolution image from a plurality of low resolution images, the method comprising,

capturing a plurality of low resolution electronic images of a subject, the subject defining a hidden reference signal,

using said reference signal to determine alignment of a plurality of said low resolution images, and

combining data from at least some of said low resolution images into a high resolution image;

wherein the subject comprises a printed substrate having the hidden reference signal formed thereon.

19. The method recited in claim 18 wherein said low resolution images are aligned in accordance with the holes in a Bayer square.

20. The method recited in claim 18 wherein a plurality of low resolution images are captured and only those low resolution images which align to within a specified tolerance with holes in a Bayer square are used to form said composite image.

21. The method of claim 18 wherein the subject comprises a printed article, and said print defines the hidden reference signal.

22. The method of claim 21 wherein the article comprises printed artwork encoded with a digital watermark, the hidden reference signal comprising said watermark.

23. The method of claim 18 that includes combining data from some, but not all, of said low resolution images into a high resolution image.