Compressed domain image summation apparatus, systems, and methods

• US 7,920,746 B2
• Filed: 04/23/2007
• Issued: 04/05/2011
• Est. Priority Date: 04/23/2007
• Status: Expired due to Fees

• Current Assignee: Aptina Imaging Corporation (ON Semiconductor Corporation)
• Original Assignee: Aptina Imaging Corporation (ON Semiconductor Corporation)
• Inventors: Silverstein, Amnon, Rodricks, Brian, Lin, Sheng
• Primary Examiner(s): Alavi; Amir
• Application Number:
  US11/738,897
• Publication Number:
  US 20080260265A1
• Time in Patent Office: 1,443 Days
• Field of Search: 382218-220, 382/232, 382/233, 382/236, 382/238, 382/276, 382/284, 382/305, 382/312, 382/315, 348/208.4, 348/222.1, 348/231.99, 348/296, 348/333.11, 348/362, 348/E3.002, 348/E3.018, 348/E5.037, 375/240.06, 375/240.12, 375/240.23, 375/240.24, 375/240.27, 375/240.29, 375/E7.081, 711/170, 358/474, 358/497
• US Class Current: 382/232
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/20201 Motion blur correction
• CPC Class Codes: G06T 2207/20216 Image averaging
• CPC Class Codes: G06T 5/10 using non-spatial domain filtering
• CPC Class Codes: G06T 5/50 using two or more images, e.g. averaging or subtraction
• CPC Class Codes: G06T 5/70 Denoising; Smoothing
• CPC Class Codes: G06T 5/73 Deblurring; Sharpening
• CPC Class Codes: G06T 7/262 using transform domain methods, e.g. Fourier domain methods
• CPC Class Codes: G06T 7/37 using transform domain methods
• CPC Class Codes: H04N 19/48 using compressed domain processing techniques other than decoding, e.g. modification of transform coefficients, variable length coding [VLC] data or run-length data motion estimation in a transform domain H04N19/547; processing of decoded motion vectors H04N19/513
• CPC Class Codes: H04N 19/547 Motion estimation performed in a transform domain
• CPC Class Codes: H04N 19/56 Motion estimation with initialisation of the vector search, e.g. estimating a good candidate to initiate a search
• CPC Class Codes: H04N 23/70 Circuitry for compensating brightness variation in the scene
• CPC Class Codes: H04N 23/741 by increasing the dynamic range of the image compared to the dynamic range of the electronic image sensors
• CPC Class Codes: H04N 23/76 by influencing the image signals
• CPC Class Codes: H04N 23/81 for suppressing or minimising disturbance in the image signal generation

Abstract

Apparatus, systems, and methods disclosed herein may transpose image blocks from successively-captured versions of an image according to relative movement between an image capture device and the scene being captured. The transposition may provide for alignment of the successively-captured images notwithstanding the movement. The transposed image blocks from the successive images are composited in the frequency domain by integrating frequency domain coefficients from each into a composite final image. Additional embodiments are disclosed.

Claims

1. An apparatus, comprising:

a strip buffer to temporarily store an image data stream associated with a new version of an image as the image data stream is received from an image sensor array (ISA);

an image comparator to compare a selected set of low-frequency coefficients of a new frequency domain (FD) image block associated with the new version of the image to a corresponding set of low-frequency coefficients of an earlier-stored composite FD image block associated with at least one previously received version of the image; and

combinational logic to combine the new FD image block with the earlier-stored composite FD image block to yield a new composite FD image block if the selected set of low-frequency coefficients of the new FD image block match the corresponding set of low-frequency coefficients of the earlier-stored composite FD image block within a match threshold.

2. The apparatus of claim 1, further comprising:

a first domain translator coupled to the strip buffer to convert an earlier-received image block associated with a first-received image from a spatial domain to a frequency domain to yield an earlier-received FD image block.

3. The apparatus of claim 2, further comprising:

a first entropy encoder operatively coupled to the first domain translator to entropy-encode the earlier-received FD image block to yield an entropy-encoded earlier-received FD image block and to store the entropy-encoded earlier-received FD image block in a compressed image buffer as an entropy-encoded earlier-stored composite FD image block.

4. The apparatus of claim 1, further comprising:

a second domain translator coupled to the strip buffer to convert a block of pixel illuminance values selected from the image data stream from a spatial domain (SD) to a frequency domain using an SD to FD transform operation to yield the new FD image block.

5. The apparatus of claim 1, further comprising:

averaging logic coupled to the combinational logic to compute a weighted average of an FD coefficient from the new FD image block and a corresponding FD coefficient from the earlier-stored composite FD image block, wherein a weight associated with the FD coefficient from the new FD image block comprises a value of one and wherein a weight associated with the corresponding FD coefficient from the earlier-stored composite FD image block comprises a number equal to a number of image frames composited into the earlier-stored composite FD image block.

6. The apparatus of claim 1, further comprising:

a second entropy encoder coupled to the combinational logic to entropy-encode the new composite FD image block to yield an entropy-encoded new composite FD image block and to store the entropy-encoded new composite FD image block in a compressed image buffer.

7. The apparatus of claim 1, further comprising:

a first entropy decoder coupled to the image comparator to perform an entropy decoding operation on an entropy-encoded earlier-stored composite FD image block to yield the earlier-stored composite FD image block.

8. The apparatus of claim 1, further comprising:

transpositional logic operatively coupled to the strip buffer to perform a spatial transposition operation on a set of coordinates associated with a pixel within the earlier-stored composite FD image block to find a set of transposed coordinates associated with a corresponding pixel from the new version of the image.

9. The apparatus of claim 8, further comprising:

an intra-block coordinate interpolator coupled to the transpositional logic to read a set of block positional coordinates associated with the earlier-stored composite FD image block and to determine the set of coordinates associated with the pixel within the earlier-stored composite FD image block, wherein each coordinate of the set of coordinates is calculated as an offset from a corresponding block positional coordinate.

10. The apparatus of claim 8, further comprising:

an affine matrix calculator coupled to the transpositional logic to generate an affine matrix such that a set of coordinates of a set of starting points associated with at least one vector multiplied by the affine matrix results in a set of ending points associated with the at least one vector, wherein the at least one vector represents a path between a position in a frame of a previously-received image block associated with the at least one previously received version of the image and a position in the frame of a corresponding subsequently-received image block associated with the new version of the image.

11. The apparatus of claim 1, further comprising:

motion matching logic operatively coupled to the strip buffer to calculate a vector between a position in a frame of a spatial domain (SD) version of the earlier-stored composite FD image block and a position in the frame of a matching image block found in a search window associated with the new version of the image and to store the vector in a vector table if a strength of a match between the SD version of the earlier-stored composite FD image block and the matching image block is above a selected match threshold.

12. The apparatus of claim 11, further comprising:

a search window buffer coupled to the strip buffer to store a set of pixels associated with the search window.

13. The apparatus of claim 11, further comprising:

image feature strength logic operatively coupled to the motion matching logic to scan a selected area of a compressed image buffer to find a highest strength earlier-stored composite FD image block with an image feature strength metric value higher than any other FD image block from the selected area by calculating an image feature strength metric value associated with the earlier-stored composite FD image block using a set of low-frequency coefficients from the earlier-stored composite FD image block; and

a region buffer associated with the selected area to store the highest strength earlier-stored composite FD image block.

14. The apparatus of claim 13, further comprising:

a second entropy decoder coupled to the image feature strength logic to load an entropy-encoded earlier-stored composite FD image block from the compressed image buffer and to perform an entropy decoding operation on at least a high-order portion of the entropy-encoded earlier-stored composite FD image block to reveal the set of low-frequency coefficients of the earlier-stored composite FD image block.

15. A method, comprising:

using hardware-based circuitry to perform the steps of;

comparing a selected set of low-frequency coefficients of a new frequency domain (FD) image block from an image data stream associated with a new version of an image to a corresponding set of low-frequency coefficients of an earlier-stored composite FD image block associated with at least one previously received version of the image; and

combining the new FD image block with the earlier-stored composite FD image block to yield a new composite FD image block if the set of low-frequency coefficients of the new FD image block match the set of low-frequency coefficients of the earlier-stored composite FD image block within a match threshold.

16. The method of claim 15, further comprising:

receiving sequential portions of an image data stream from an image sensor array (ISA); and

storing the sequential portions of the image data stream in a strip buffer as the portions are received from the ISA.

17. The method of claim 15, wherein the entropy-encoded earlier-stored composite FD image block comprises a joint photographic experts group (JPEG) image block.

18. The method of claim 15, further comprising:

calculating a weighted average of an FD coefficient from the new FD image block and a corresponding FD coefficient from the earlier-stored composite FD image block, wherein a weight associated with the FD coefficient from the new FD image block comprises a value of one and wherein a weight associated with the corresponding FD coefficient from the earlier-stored composite FD image block comprises a number equal to a number of image frames composited into the earlier-stored composite FD image block; and

using the weighted average to combine the new FD image block with the earlier-stored composite FD image block.

19. The method of claim 15, wherein the earlier-stored composite FD image block is associated with at least one previously-captured frame of an image and wherein the new FD image block is associated with a next-captured frame of the image.

20. The method of claim 15, further comprising:

performing an entropy decoding operation on an entropy-encoded earlier-stored composite FD image block to yield the earlier-stored composite FD image block.

21. The method of claim 15, further comprising:

reading a set of block positional coordinates associated with the earlier-stored composite FD image block; and

determining a set of pixel coordinates associated with at least one pixel within the earlier-stored composite FD image block, the set of pixel coordinates calculated as offsets from the set of block positional coordinates.

22. The method of claim 21, further comprising:

performing a spatial transposition operation on the set of coordinates associated with the at least one pixel from the earlier-stored composite FD image block to find a set of coordinates associated with a set of pixels from the image data stream such that the set of pixels from the image data stream are in approximate spatial alignment with pixels associated with a spatial domain version of the earlier-stored composite FD image block; and

selecting the set of pixels from the image data stream to form a new image block.

23. The method of claim 15, further comprising:

calculating an affine matrix such that a set of coordinates of a set of starting points associated with at least one vector multiplied by the affine matrix results in a set of ending points associated with the at least one vector, wherein the at least one vector represents a path between a position in a frame of a previously-received image block and a corresponding position in the frame of a subsequently-received image block.

24. The method of claim 15, further comprising:

entropy-encoding the new composite FD image block to yield an entropy-encoded new composite FD image block; and

storing the entropy-encoded new composite FD image block in a compressed image buffer.

25. The method of claim 15, further comprising:

discarding the new FD image block if the set of low-frequency coefficients of the new FD image block do not match the set of low-frequency coefficients of the earlier-stored composite FD image block within the match threshold, leaving the earlier-stored composite FD image block as the new composite FD image block.

26. The method of claim 15, further comprising:

loading an entropy-encoded earlier-stored composite FD image block from a compressed image buffer;

performing an entropy decoding operation on at least a high-order portion of the entropy-encoded earlier-stored composite FD image block to reveal a set of low-frequency coefficients of the earlier-stored composite FD image block; and

calculating an image feature strength metric value associated with the earlier-stored composite FD image block using the set of low-frequency coefficients.

27. The method of claim 15, further comprising:

calculating a vector between a position in a frame of a spatial domain (SD) version of the selected composite FD image block and a position in the frame of a matching image block found in a search window of an image represented by the image data stream; and

storing the vector in a vector table if a strength of a match between the SD version of the selected composite FD image block and the matching image block is above a selected match threshold.

28. The method of claim 27, further comprising:

performing an FD to SD transform operation on the selected composite FD image block to obtain the SD version of the selected composite FD image block;

conducting a search of the search window for a candidate matching image block;

measuring a strength of a match between the SD version of the selected composite FD image block and the candidate matching image block; and

selecting the candidate matching image block as the matching image block if the strength of the match between the SD version of the selected composite FD image block and the candidate matching image block is greater than a strength of a match associated with any other candidate matching image block found in the search window.

29. The method of claim 28, further comprising:

utilizing at least one of an inverse of a sum of absolute differences operation or an inverse of a sum of squares of an absolute difference operation to measure a match strength characteristic.

30. The method of claim 28, further comprising:

establishing a set of origination coordinates of the vector as a set of coordinates associated with a pixel in the SD version of the selected composite FD image block; and

establishing a set of destination coordinates of the vector as a set of coordinates associated with a pixel in the matching image block, wherein the pixel in the matching image block is in spatial correspondence with the pixel in the selected composite FD image block.

31. The method of claim 28, further comprising:

loading a set of high-quality vectors from a table of vectors; and

initiating the search within the search window at an image block closest to a location corresponding to a median value of a set of destination coordinates associated with the set of high-quality vectors.