SYSTEM AND METHOD TO SELECTIVELY COMBINE VIDEO FRAME IMAGE DATA

• US 20100271498A1
• Filed: 04/22/2009
• Published: 10/28/2010
• Est. Priority Date: 04/22/2009
• Status: Active Grant

• Current Assignee: Qualcomm, Inc.
• Original Assignee: Qualcomm, Inc.
• Inventors: Li, Hsiang-Tsun, Hwang, Hau, Atanassov, Kalin M.
• Granted Patent: US 8,111,300 B2
• Time in Patent Office: Days
• US Class Current: 348/222.100
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/20016 Hierarchical, coarse-to-fine, multiscale or multiresolution image processing; Pyramid transform
• CPC Class Codes: G06T 2207/20021 Dividing image into blocks, subimages or windows
• CPC Class Codes: G06T 5/50 using two or more images, e.g. averaging or subtraction
• CPC Class Codes: G06T 7/32 using correlation-based methods
• CPC Class Codes: H04N 23/68 for stable pick-up of the scene, e.g. compensating for camera body vibrations
• CPC Class Codes: H04N 23/70 Circuitry for compensating brightness variation in the scene
• CPC Class Codes: H04N 23/741 by increasing the dynamic range of the image compared to the dynamic range of the electronic image sensors
• CPC Class Codes: H04N 23/951 by using two or more images to influence resolution, frame rate or aspect ratio

Abstract

Systems and methods to selectively combine video frame image data are disclosed. First image data corresponding to a first video frame and second image data corresponding to a second video frame are received from an image sensor. The second image data is adjusted by at least partially compensating for offsets between portions of the first image data with respect to corresponding portions of the second image data to produce adjusted second image data. Combined image data corresponding to a combined video frame is generated by performing a hierarchical combining operation on the first image data and the adjusted second image data.

Claims

1. A method comprising:

receiving first image data corresponding to a first video frame and second image data corresponding to a second video frame, the first video frame and the second video frame received from an image sensor;

adjusting the second image data by at least partially compensating for offsets between portions of the first image data with respect to corresponding portions of the second image data to produce adjusted second image data; and

generating combined image data corresponding to a combined video frame by performing a hierarchical combining operation on the first image data and the adjusted second image data.

2. The method of claim 1, wherein the combined image data has at least one of a greater resolution than the first image data, a higher dynamic range than the first image data, a larger depth of field than the first image data, less noise than the first image data, or less blurring due to motion than the first image data.

3. The method of claim 1, further comprising compressing the combined image data and storing the compressed combined image data at a memory device.

4. The method of claim 1, wherein the hierarchical combining operation includes a coarse combining operation and a fine combining operation.

5. The method of claim 4, wherein the coarse combining operation includes selectively discarding one or more macroblocks of the adjusted second image data when a difference between the one or more macroblocks of the adjusted second image data and corresponding macroblocks of the first image data exceeds a selectable threshold value.

6. The method of claim 4, wherein the fine combining operation includes filtering the first image data and the adjusted second image data to compare filtered image characteristics, and wherein first values of the first image data and corresponding second values of the second image data are selectively combined based on the filtered image characteristics.

7. The method of claim 1, wherein adjusting the second image data includes applying a set of motion vector data corresponding to the offsets between the portions of the first image data with respect to the corresponding portions of the second image.

8. The method of claim 7, wherein the set of motion vector data is determined via a hierarchical registration process.

9. The method of claim 8, wherein the hierarchical registration process includes:

determining a first alignment offset between a first portion of the first image and a first portion of the second image; and

determining a second alignment offset between a second portion of the first image and a second portion of the second image based on the first alignment offset, wherein the second portion of the first image is within the first portion of the first image.

10. An apparatus comprising:

a registration circuit configured to generate a set of motion vector data based on first image data corresponding to a first video frame and second image data corresponding to a second video frame, the first video frame and the second video frame received from an image sensor;

a combination circuit coupled to perform a hierarchical combination operation to selectively combine the first image data and adjusted second image data, wherein the adjusted second image data corresponds to the second image data after being adjusted according to the motion vector data; and

a control circuit to control the combination circuit to generate combined image data that corresponds to a combined video frame.

11. The apparatus of claim 10, wherein the combined image data has at least one of a greater resolution than the first image data, a higher dynamic range than the first image data, a larger depth of field than the first image data, less noise than the first image data, or less blurring due to motion than the first image data.

12. The apparatus of claim 10, wherein the combination circuit includes a coarse combination circuit and a fine combination circuit.

13. The apparatus of claim 10, wherein the combination circuit comprises:

a macro block image alignment circuit having a first input coupled to receive the first image data and a second input coupled to receive the second image data, a third input coupled to receive a macro block motion vector signal, an aligned first image output and an aligned second image output;

a block difference discriminator circuit having a first input coupled to receive the aligned first image output, a second input coupled to receive the aligned second image output, a third input coupled to receive a macro block difference signal, and having a first image difference output and a second image difference output; and

a mean pixel difference circuit having a first input coupled to the block difference discriminator circuit and configured to provide a pixel difference output signal to indicate a mean pixel difference result.

14. The apparatus of claim 13, further comprising:

a first filter coupled between the block difference discriminator circuit and the mean pixel difference circuit having a first input coupled to receive the first image difference output and an output providing a filtered first image difference output; and

a second filter coupled between the block difference discriminator circuit and the mean pixel difference circuit having a first input coupled to receive the second image difference output and having an output to provide a filtered second image difference output.

15. The apparatus of claim 10, further comprising a compress and store module configured to compress the combined image data and to store the compressed combined image data at a memory device.

16. The apparatus of claim 10, further comprising:

a camera lens; and

video capture circuitry coupled to the camera lens to generate the first image data and to generate the second image data.

17. The apparatus of claim 16, further comprising a processor coupled to a wireless transceiver to send and receive data via an antenna.

18. An apparatus comprising:

a registration means for generating a set of motion vector data based on first image data corresponding to a first video frame and second image data corresponding to a second video frame, the first video frame and the second video frame received from an image sensor;

a combination means for performing a hierarchical combination operation to selectively combine the first image data and adjusted second image data, wherein the adjusted second image data corresponds to the second image data after being adjusted according to the motion vector data; and

a control means for controlling the combination circuit to generate combined image data corresponding to a combined video frame.

19. The apparatus of claim 18, further comprising:

a camera lens; and

video capture means coupled to the camera lens for generating the first image data and the second image data.

20. The apparatus of claim 18, further comprising means for encoding the combined image data to generate encoded video data.

21. A computer readable medium storing computer executable code comprising:

code for receiving first image data corresponding to a first video frame and second image data corresponding to a second video frame, the first video frame and the second video frame received from an image sensor;

code for adjusting the second image data by at least partially compensating for offsets between portions of the first image data with respect to corresponding portions of the second image data to produce adjusted second image data; and

code for generating combined image data corresponding to a combined video frame by performing a hierarchical combining operation on the first image data and the adjusted second image data.

22. The computer readable medium of claim 21, wherein the combined image data has at least one of a greater resolution than the first image data, a higher dynamic range than the first image data, a larger depth of field than the first image data, less noise than the first image data, or less blurring due to motion than the first image data.

23. The computer readable medium of claim 21, further comprising code for providing the combined image data to a video encoder to generate encoded video data.

24. The computer readable medium of claim 21, wherein the hierarchical combining operation includes a coarse combining operation and a fine combining operation.