Account for clipped pixels in auto-focus statistics collection

• US 9,648,261 B2
• Filed: 08/26/2015
• Issued: 05/09/2017
• Est. Priority Date: 08/26/2015
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: Lim, Suk Hwan, Pope, David R., Tajbakhsh, Touraj
• Primary Examiner(s): OSINSKI, MICHAEL S
• Application Number:
  US14/836,918
• Publication Number:
  US 20170064232A1
• Time in Patent Office: 622 Days
• CPC Class Codes: G06T 5/20 using local operators
• CPC Class Codes: H04N 23/67 Focus control based on electronic image sensor signals
• CPC Class Codes: H04N 25/134 based on three different wavelength filter elements
• CPC Class Codes: H04N 25/60 Noise processing, e.g. detecting, correcting, reducing or removing noise
• CPC Class Codes: H04N 25/61 the noise originating only from the lens unit, e.g. flare, shading, vignetting or "cos4"
• CPC Class Codes: H04N 25/683 by defect estimation performed on the scene signal, e.g. real time or on the fly detection

Abstract

An image processing pipeline may account for clipped pixels in auto focus statistics. Generating auto focus statistics may include evaluating a neighborhood of pixels with respect to a given pixel in a stream of pixels for an image. If a clipped pixel is identified within the neighborhood of pixels then the evaluation of the given pixel may be excluded from an auto focus statistic. The image processing pipeline may also provide auto focus statistics that do not exclude clipped pixels. A luminance edge detection value may, in some embodiments, be generated by applying an IIR filter to the given pixel in a stream of pixels to band-pass filter the given pixel before including the band-pass filtered pixel in the generation of the luminance edge detection value.

Claims

1. An apparatus, comprising:

an image sensor that collects image data;

an image signal processor, configured to;

receive a stream of pixel data collected at the image sensor for an image frame;

determine an edge detection value for the image frame, wherein to determine the edge detection value the image signal processor is configured to at least;

evaluate respective neighborhoods of pixels for individual pixels in the stream of pixel data to determine the edge detection value; and

for at least one of the individual pixels;

detect a clipped pixel within the respective neighborhood; and

exclude the evaluation of the at least one individual pixel from the edge detection value; and

store the edge detection value in a memory; and

an automated controller, configured to;

access the memory to obtain the edge detection value; and

direct a focal adjustment at the image sensor based, at least in part, on the edge detection value.

2. The apparatus of claim 1,

wherein the stream of pixel data is received in a raw format;

and

wherein the image signal processor is further configured to;

down-sample the stream of pixel data;

perform the determination of the edge detection value to determine another edge detection value according to the down-sampled stream of pixel data; and

output the other edge detection value.

3. The apparatus of claim 1, wherein the edge detection value is determined according to a luminance channel determined for the stream of pixel data, wherein the image signal processor is configured to perform the determination for different color channels of the stream of pixel data to determine other edge detection values corresponding to the different color channels.

4. The apparatus of claim 1, wherein to detect the clipped pixel, the image signal processor is configured to at least:

compare the pixels of the respective neighborhoods of pixels to tracking data maintained for the stream of pixel data, wherein the tracking data indicates those pixels determined to be clipped at one or more image processing operations that process the stream of pixel data prior to the determination of the edge detection value;

or

compare the pixel of the respective neighborhoods of pixels to a clipped pixel threshold, wherein the clipped pixel exceeds the clipped pixel threshold.

5. The apparatus of claim 1, wherein the image signal processor is further configured to:

determine another edge detection value, wherein the to determine the other edge detection value the image signal processor is configured to at least;

apply an infinite impulse response (IIR) filter to generate respective band-pass filter values for pixels in the stream of pixel data;

calculate the other edge detection value based on the band-pass filter values, wherein the calculation includes the band-pass filter values for clipped pixels in the stream of pixel data; and

store the other edge detection value in the memory.

6. The apparatus of claim 5, wherein the IIR filter is a horizontal IIR filter.

7. The apparatus of claim 1, wherein the apparatus comprises a mobile computing device.

8. A method, comprising:

receiving, at an image signal processor, a stream of pixel data collected at an image sensor for an image frame;

evaluating, by the image signal processor, the stream of pixel data to generate an auto focus statistic according to the image frame, comprising;

identifying a clipped pixel in the stream of pixel data within a neighborhood of pixels evaluated for a given pixel of the stream of pixel data; and

excluding the evaluation of the neighborhood of pixels for the given pixel from the generation of the auto focus statistic; and

performing, at the image sensor, a focal adjustment determined based on the auto focus statistic.

9. The method of claim 8, wherein excluding the evaluation of the neighborhood of pixels for the given pixel value comprises excluding output of a two-dimensional spatial filter applied to a luminance channel of the clipped pixel, wherein the neighborhood of the pixels corresponds to an area identified by a filter kernel applied by the two-dimensional spatial filter.

10. The method of claim 9, further comprising:

prior to evaluating the stream of pixel data, down-sampling the stream of pixel data such that the evaluated stream of pixel data is a down-sampled stream of pixel data.

11. The method of claim 8, wherein identifying the clipped pixel in the stream of pixel data comprises individually comparing the pixels of the neighborhood of pixels to a clipped pixel threshold, wherein the clipped pixel exceeds the clipped pixel threshold.

12. The method of claim 8, further comprising:

maintaining tracking data for pixels in the stream of pixel data that indicates pixels determined to be clipped pixels at one or more image processing operations that process the stream of pixels prior to generating the auto focus statistic; and

wherein identifying the clipped pixel in the stream of pixel data comprises accessing the tracking data for the pixels of the neighborhood of pixels.

13. The method of claim 8, further comprising:

generating, by the image signal processor, another auto focus statistic for the image frame, wherein the other auto focus statistic is a luminance channel edge detection value, wherein the generating comprises;

applying an infinite impulse response (IIR) filter to generate respective band-pass filter values for pixels in the stream of pixel data; and

calculating the luminance channel edge detection value based on the band-pass filter values, wherein the calculating includes the band-pass filter values for clipped pixels in the stream of pixel data; and

providing the luminance channel edge detection value to an automated controller for the image sensor, wherein the auto focus statistic is also provided to the automated controller, wherein the automated controller performs the focal adjustment at the image sensor.

14. The method of claim 13, wherein the IIR filter is a vertical IIR filter.

15. A system, comprising:

a device configured to perform image processing, the device comprising;

a sensor interface configured to receive image data from an image sensor; and

an image processing pipeline comprising an auto focus statistics collection unit, wherein the image processing pipeline is configured to;

receive via the sensor interface a stream of pixel data for an image frame collected at the image sensor;

generate, by the auto focus statistics collection unit, an auto focus statistic for the image frame, wherein to generate the auto focus statistic the auto focus statistics collection unit is configured to at least;

evaluate respective neighborhoods of pixels for individual pixels in the stream of pixel data;

for at least one of the individual pixels;

 

detect a clipped pixel within the respective neighborhood;

 

exclude the evaluation of the at least one individual pixel from the auto focus statistic; and

output the auto focus statistic.

16. The system of claim 15, wherein the stream of pixel data is in Bayer raw format comprising red, blue and green color components, wherein to evaluate the respective neighborhoods of pixels for the individual pixels in the stream of pixel data, the image signal processor is configured to apply a horizontal filter to the stream of pixel data to generate a respective version of the auto focus statistics for the red, blue and green color components.

17. The system of claim 15, wherein the image signal processor is further configured to:

down-sample the stream of pixel data;

perform the generation of the auto focus statistic to generate another auto focus statistic according to the down-sampled stream of pixel data;

output the other auto focus statistic.

18. The system of claim 15,

wherein the image signal processor further comprises a plurality of other statistic collection units;

wherein the image signal processor is further configured to track those pixels of the stream of pixel data identified as clipped at one or more of the plurality of other statistic collection units; and

wherein to detect the clipped pixel within the respective neighborhood, the image signal processor is configured to access tracking data for the pixels of the respective neighborhood.

19. The system of claim 15, wherein the auto focus statistics collection unit is further configured to:

generate, by the auto focus statistics collection unit, another auto focus statistic for the image frame, wherein to generate the other auto focus statistic the auto focus statistics collection unit is configured to at least;

apply an infinite impulse response (IIR) filter to generate respective band-pass filter values for pixels in the stream of pixel data;

calculate the other auto focus statistic based on the band-pass filter values; and

output the other auto focus statistic.

20. The system of claim 19, wherein the system further comprises the image sensor and an automated controller for the image sensor, wherein the automated controller is configured to:

obtain the auto focus statistic and the other auto focus statistic; and

based on the auto focus statistic or the other auto focus statistic, perform a focal adjustment at the image sensor.