Video acquisition with integrated GPU processing

• US 7,596,280 B2
• Filed: 09/29/2005
• Issued: 09/29/2009
• Est. Priority Date: 09/29/2005
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: Bilbrey, Brett, Ouzilevski, Alexei V., Zipnick, Jay
• Primary Examiner(s): Werner; Brian P
• Assistant Examiner(s): CHU, RANDOLPH I
• Application Number:
  US11/238,269
• Publication Number:
  US 20070071342A1
• Time in Patent Office: 1,461 Days
• Field of Search: 382/274, 382/254, 382/103, 382/294, 348/251, 348/254, 345/20, 345/63, 345/77, 345581-618, 345596-599, 345690-697
• US Class Current: 382/274
• CPC Class Codes: G06T 5/20 using local operators
• CPC Class Codes: G06T 5/50 using two or more images, e.g. averaging or subtraction

Abstract

Systems and techniques for processing sequences of video images involve receiving, on a computer, data corresponding to a sequence of video images detected by an image sensor. The received data is processed using a graphics processor to adjust one or more visual characteristics of the video images corresponding to the received data. The received data can include video data defining pixel values and ancillary data relating to settings on the image sensor. The video data can be processed in accordance with ancillary data to adjust the visual characteristics, which can include filtering the images, blending images, and/or other processing operations.

Claims

1. A method comprising:

using one or more processors to perform

identifying a first gain level associated with an autoexposure phase of a video detection process;

receiving a sequence of frames detected by an image sensor in a high dynamic range phase of the video detection process, the frames detected using at least two different gain levels, each of the at least two different gain levels differing from the first gain level, and each frame comprising a plurality of pixel values;

identifying a frame from the sequence of frames to blend with at least one other frame detected using a different gain level to produce a blended frame for replacing the frame in the sequence of frames;

normalizing and adjusting pixel values for the frame and the at least one other frame, wherein the pixel values are normalized to approximate the first gain level and the pixel values are adjusted using blending factors; and

combining the adjusted, normalized pixel values to generate a blended pixel value.

2. The method of claim 1 wherein the sequence of frames comprises a portion of a video sequence.

3. The method of claim 1 wherein normalizing pixel value comprises compensating for the gain level used to detect the pixel value.

4. The method of claim 1 further comprising:

estimating a degree of motion between frames in the sequence of frames for each of a plurality of pixels in a particular frame;

determining if the degree of motion for each pixel exceeds a threshold; and

if the degree of motion exceeds the threshold, compensating for the gain level used to detect the pixel to produce a replacement pixel for replacing the pixel in the particular frame.

5. The method of claim 4 wherein estimating the degree of motion between frames comprises comparing corresponding pixels in successive frames detected using a particular gain level.

6. The method of claim 4 further comprising compensating for the gain level of successive frames in the sequence of frames, wherein estimating the degree of motion between frames comprises comparing corresponding pixels in gain-compensated, successive frames.

7. The method of claim 1 further comprising temporally filtering the blended frames.

8. An article of manufacture comprising a computer-readable storage device storing instructions for causing data processing apparatus to:

identify a first gain level associated with an autoexposure phase of a video detection process;

receive pixel data for each of a plurality of frames in a high dynamic range phase of the video detection process, the plurality of frames detected by an image sensor using at least two different gain levels, each of the at least two different gain levels differing from the first gain level;

generate a modified video sequence for the high dynamic range phase of the video detection process by;

detecting a degree of motion between frames based on the pixel data;

blending the pixel data for the plurality of frames to generate a blended frame for the video sequence for pixels having a degree of motion below a selected threshold; and

compensating for gain levels associated with pixels having a degree of motion above the selected threshold to produce compensated pixel data having a gain level approximating the first gain level, wherein the modified video sequence includes a first set of pixels resulting from blending the pixel data and a different, second set of pixels resulting from compensating for gain levels.

9. The article of claim 8 wherein the instructions are further operable to cause data processing apparatus to:

receive pixel data for a subsequent frame in the video sequence; and

blend the pixel data fur the subsequent frame with the pixel data from a subset of the plurality of frames to generate a subsequent blended frame for the video sequence.

10. The article of claim 8 wherein the degree of motion is detected between one of the plurality of frames and a preceding frame with a same gain level.

11. The article of claim 8 wherein the degree of motion is detected between one of the plurality of frames and a preceding frame with a different gain level.

12. The article of claim 8 wherein pixel data is blended using a weighted combination of the pixel data.

13. The article of claim 12 wherein each pixel of the blended frame comprises a combination of weighted pixel data terms, each pixel data term weighted using a blending factor corresponding to a luminance value of the pixel.

14. The article of claim 8 wherein the instructions are further operable to cause data processing apparatus to filter a plurality of temporally spaced blended frames to reduce noise.

15. The article of claim 14 wherein the instructions are further operable to cause data processing apparatus to detect a degree of motion for use in determining a strength of temporal filtering.

16. A system for expanding the dynamic range of a video sequence, the system comprising:

a memory for storing a sequence of frames detected by an image sensor, the frames detected using at least two different gain levels associated with a high dynamic range phase of a video detection process, each of the at least two different gain levels differing from a selected gain level associated with an autoexposure phase of the video detection process, and wherein each frame comprises a plurality of pixel values;

a module configured to generate a modified video sequence for the high dynamic range phase of the video detecting process by;

detecting a degree of motion between frames in the sequence of frames for each of a plurality of pixels in a particular frame;

blending frames in the sequence to produce processed frames with an expanded dynamic range for pixels having a degree of motion below a selected threshold; and

compensating for gain levels associated with pixels having a degree of motion above the selected threshold to produce compensated pixel data having a gain level approximating the selected gain level, wherein the modified video sequence includes a first set of pixels resulting from blending the pixel data and a different, second set of pixels resulting from compensating for gain levels.

17. The system of claim 16 wherein the sequence of frames includes at least one frame detected using a first gain level and at least two frames detected using a second gain level.

18. The system of claim 17 wherein the module is configured to detect a degree of motion based on a comparison between-frames detected using the second gain level and to blend a frame detected using the first gain level and a frame detected using the second gain level.

19. The system of claim 18 a filter configured to filter the blended frames to remove noise based at least in part on the gain level of the blended frames.

20. The system of claim 19 wherein the memory includes a circular buffer for storing the sequence of frames.

21. A system for expanding the dynamic range of a video sequence, the system comprising:

a memory for storing a sequence of frames detected by an image sensor, the frames detected using at least two different gain levels associated with a high dynamic range phase of a video detection process, each of the at least two different gain levels differing from a selected gain level associated with an autoexposure the video detection process, and wherein each frame comprises a plurality of pixel values;

a module configured to generate a modified video sequence for the high dynamic range phase of the video detection process by;

identifying a frame to blend with at least one other frame detected using a different gain level to produce a blended frame for replacing the frame in the sequence of frames;

normalizing and adjusting each pixel value for the frame and the at least one other frame, wherein the pixel values are normalized to approximate the selected gain level and the pixel values are adjusted using blending factors; and

combining the adjusted, normalized pixel values to generate a blended pixel value.

22. The system of claim 21 wherein the sequence of frames includes at least one frame detected using a first gain level and at least two frames detected using a second gain level.

23. The system of claim 21 wherein the module is configured to normalize pixel values by compensating for the gain level used to detect the pixel value.

24. Thu system of claim 21 a filter configured to filter the blended frames to remove nose based at least in part on the gain level of the blended frames.

25. The system of claim 21 wherein the memory includes a circular buffer for storing the sequence of frames.