System and method to selectively combine video frame image data

• US 8,111,300 B2
• Filed: 04/22/2009
• Issued: 02/07/2012
• Est. Priority Date: 04/22/2009
• Status: Active Grant

• Current Assignee: Qualcomm, Inc.
• Original Assignee: Qualcomm, Inc.
• Inventors: Hwang, Hau, Li, Hsiang-Tsun, Atanassov, Kalin M.
• Primary Examiner(s): Ho, Tuan
• Assistant Examiner(s): Wang, Kent
• Application Number:
  US12/428,354
• Publication Number:
  US 20100271498A1
• Time in Patent Office: 1,021 Days
• Field of Search: 348/222.1, 348/413.1, 348/422.1, 348/208.14, 348/208.1, 348/208.6, 348/218.1, 348/47, 348/208.4, 348/221.1, 348/362, 382/103, 382/190, 382/260, 382/284, 382/294, 382/299, 375/240.01, 375/240.16
• US Class Current: 348/222.1
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/20016 Hierarchical, coarse-to-fine, multiscale or multiresolution image processing; Pyramid transform
• CPC Class Codes: G06T 2207/20021 Dividing image into blocks, subimages or windows
• CPC Class Codes: G06T 5/50 using two or more images, e.g. averaging or subtraction
• CPC Class Codes: G06T 7/32 using correlation-based methods
• CPC Class Codes: H04N 23/68 for stable pick-up of the scene, e.g. compensating for camera body vibrations
• CPC Class Codes: H04N 23/70 Circuitry for compensating brightness variation in the scene
• CPC Class Codes: H04N 23/741 by increasing the dynamic range of the image compared to the dynamic range of the electronic image sensors
• CPC Class Codes: H04N 23/951 by using two or more images to influence resolution, frame rate or aspect ratio

Abstract

Systems and methods to selectively combine video frame image data are disclosed. First image data corresponding to a first video frame and second image data corresponding to a second video frame are received from an image sensor. The second image data is adjusted by at least partially compensating for offsets between portions of the first image data with respect to corresponding portions of the second image data to produce adjusted second image data. Combined image data corresponding to a combined video frame is generated by performing a hierarchical combining operation on the first image data and the adjusted second image data.

Claims

1. A method of storing combined image data to a storage device, comprising:

receiving first image data corresponding to a first video frame and second image data corresponding to a second video frame, the first video frame and the second video frame received from an image sensor;

adjusting the second image data by at least partially compensating for offsets between portions of the first image data with respect to corresponding portions of the second image data to produce adjusted second image data;

generating combined image data corresponding to a combined video frame by filtering the first image data and the adjusted second image data to compare filtered image characteristics, and wherein first values of the first image data and corresponding second values of the second image data are selectively combined based on the filtered image characteristics; and

storing the combined image data to a storage device.

2. The method of claim 1, wherein the combined image data has at least one of a greater resolution than the first image data, a higher dynamic range than the first image data, a larger depth of field than the first image data, less noise than the first image data, or less blurring due to motion than the first image data.

3. The method of claim 1, further comprising compressing the combined image data prior to storing the data to the storage device.

4. The method of claim 1, wherein the hierarchical combining operation includes a coarse combining operation and a fine combining operation.

5. The method of claim 4, wherein the coarse combining operation includes selectively discarding one or more macroblocks of the adjusted second image data when a difference between the one or more macroblocks of the adjusted second image data and corresponding macroblocks of the first image data exceeds a selectable threshold value.

6. The method of claim 1, wherein adjusting the second image data includes applying a set of motion vector data corresponding to the offsets between the portions of the first image data with respect to the corresponding portions of the second image.

7. The method of claim 6, wherein the set of motion vector data is determined via a hierarchical registration process.

8. The method of claim 7, wherein the hierarchical registration process includes:

determining a first alignment offset between a first portion of the first image data and a first portion of-the second image data; and

determining a second alignment offset between a second portion of the first image data and a second portion of the second image data based on the first alignment offset, wherein the second portion of the first image data is within the first portion of the first image data.

9. The method of claim 1, wherein the image storage device includes one or more display buffers, registers, caches, flash memory elements, or hard disks.

10. The method of claim 1, wherein generating the combined image data includes receiving a weight to be applied to a pixel value of the first image data and receiving a second weight to be applied to a pixel value of the second image data.

11. An apparatus comprising:

a registration circuit configured to generate a set of motion vector data based on first image data corresponding to a first video frame and second image data corresponding to a second video frame, the first video frame and the second video frame received from an image sensor;

a combination circuit coupled to selectively combine the first image data and adjusted second image data by filtering the first image data and the adjusted second image data to compare filtered image characteristics, and wherein first values of the first image data and corresponding second values of the second image data are selectively combined based on the filtered image characteristics;

wherein the adjusted second image data corresponds to the second image data after being adjusted according to the motion vector data; and

a control circuit to control the combination circuit to generate combined image data that corresponds to a combined video frame.

12. The apparatus of claim 11, wherein the combined image data has at least one of a greater resolution than the first image data, a higher dynamic range than the first image data, a larger depth of field than the first image data, less noise than the first image data, or less blurring due to motion than the first image data.

13. The apparatus of claim 11, wherein the combination circuit includes a coarse combination circuit and a fine combination circuit.

14. The apparatus of claim 11, further comprising a compress and store module configured to compress the combined image data and to store the compressed combined image data at a memory device.

15. The apparatus of claim 11, further comprising:

a camera lens; and

video capture circuitry coupled to the camera lens to generate the first image data and to generate the second image data.

16. The apparatus of claim 15, further comprising a processor coupled to a wireless transceiver to send and receive data via an antenna.

17. An apparatus comprising:

a registration circuit configured to generate a set of motion vector data based on first image data corresponding to a first video frame and second image data corresponding to a second video frame, the first video frame and the second video frame received from an image sensor;

a combination circuit configured to combine the first image data and adjusted second image data, wherein the adjusted second image data corresponds to the second image data after being adjusted according to the motion vector data;

the combination circuit also including;

a macro block image alignment circuit having a first input coupled to receive the first image data and a second input coupled to receive the second image data, a third input coupled to receive a macro block motion vector signal, an aligned first image output and an aligned second image output;

a block difference discriminator circuit having a first input coupled to receive the aligned first image output, a second input coupled to receive the aligned second image output, a third input coupled to receive a macro block difference signal, and having a first image difference output and a second image difference output;

a mean pixel difference circuit having a first input coupled to the block difference discriminator circuit and configured to provide a pixel difference output signal to indicate a mean pixel difference result; and

a control circuit to control the combination circuit to generate combined image data that corresponds to a combined video frame.

18. The apparatus of claim 17, further comprising:

a first filter coupled between the block difference discriminator circuit and the mean pixel difference circuit having a first input coupled to receive the first image difference output and an output providing a filtered first image difference output; and

a second filter coupled between the block difference discriminator circuit and the mean pixel difference circuit having a first input coupled to receive the second image difference output and having an output to provide a filtered second image difference output.

19. An apparatus comprising:

a registration means for generating a set of motion vector data based on first image data corresponding to a first video frame and second image data corresponding to a second video frame, the first video frame and the second video frame received from an image sensor;

a combination means for selectively combining the first image data and adjusted second image data by filtering the first image data and the adjusted second image data to compare filtered image characteristics, and wherein first values of the first image data and corresponding second values of the second image data are selectively combined based on the filtered image characteristics;

wherein the adjusted second image data corresponds to the second image data after being adjusted according to the motion vector data; and

a control means for controlling the combination circuit to generate combined image data corresponding to a combined video frame.

20. The apparatus of claim 19, further comprising:

a camera lens; and

video capture means coupled to the camera lens for generating the first image data and the second image data.

21. The apparatus of claim 19, further comprising means for encoding the combined image data to generate encoded video data.

22. A non-transitory computer readable medium storing computer executable code comprising:

code for receiving first image data corresponding to a first video frame and second image data corresponding to a second video frame, the first video frame and the second video frame received from an image sensor;

code for adjusting the second image data by at least partially compensating for offsets between portions of the first image data with respect to corresponding portions of the second image data to produce adjusted second image data; and

code for generating combined image data corresponding to a combined video frame by filtering the first image data and the adjusted second image data to compare filtered image characteristics, and wherein first values of the first image data and corresponding second values of the second image data are selectively combined based on the filtered image characteristics.

23. The computer readable medium of claim 22, wherein the combined image data has at least one of a greater resolution than the first image data, a higher dynamic range than the first image data, a larger depth of field than the first image data, less noise than the first image data, or less blurring due to motion than the first image data.

24. The computer readable medium of claim 22, further comprising code for providing the combined image data to a video encoder to generate encoded video data.

25. The computer readable medium of claim 22, wherein the code for generating combined image data includes a coarse combining operation and a fine combining operation.