Scaling of image data in sensor interface based on detection of defective pixels

• US 9,860,429 B1
• Filed: 06/30/2016
• Issued: 01/02/2018
• Est. Priority Date: 06/30/2016
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: Silverstein, D. Amnon, Shin, Jaewon
• Primary Examiner(s): Haskins, Twyler
• Assistant Examiner(s): Tejano, Dwight C
• Application Number:
  US15/198,363
• Publication Number:
  US 20180007241A1
• Time in Patent Office: 551 Days
• Field of Search: 348241-251
• CPC Class Codes: G06T 1/20 Processor architectures; Processor configuration, e.g. pipelining
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/20024 Filtering details
• CPC Class Codes: G06T 3/4015 Image demosaicing, e.g. colour filter arrays [CFA] or Bayer patterns
• CPC Class Codes: G06T 5/20 using local operators
• CPC Class Codes: G06T 5/70 Denoising; Smoothing
• CPC Class Codes: G06T 7/0002 Inspection of images, e.g. flaw detection
• CPC Class Codes: H04N 2209/045 using mosaic colour filter
• CPC Class Codes: H04N 2209/046 Colour interpolation to calculate the missing colour values
• CPC Class Codes: H04N 23/10 for generating image signals from different wavelengths
• CPC Class Codes: H04N 25/63 applied to dark current
• CPC Class Codes: H04N 25/683 by defect estimation performed on the scene signal, e.g. real time or on the fly detection

Abstract

Embodiments of the present disclosure relate to a sensor interface circuit that performs scaling of image data in a Bayer pattern without spreading defective pixels across multiple pixels. The sensor interface circuit may include a register circuit storing operating parameters of the sensor interface circuit. The sensor interface circuit includes a scaling circuit with a first defect pixel detection circuit to detect a first defective pixel in an input image by analyzing pixels in a line of an input image data along a first direction. A first scaling circuit is coupled to the first defect pixel detection circuit and generates a scaled line of pixels representing the line of the input image scaled along the first direction according to the operating parameters.

Claims

1. A sensor interface circuit, comprising:

a register circuit configured to store operating parameters of the sensor interface circuit; and

a scaling circuit comprising;

a first defect pixel detection circuit configured to detect a first defective pixel in an input image by analyzing pixels in a line of an input image data along a first direction; and

a first scaling circuit coupled to the first defect pixel detection circuit, the first scaling circuit configured to generate a scaled line of pixels representing the line of the input image scaled along the first direction according to the operating parameters by;

(i) defining first subsets of pixels in the line of the input image so that the first defective pixel is included in only one of the first subsets of pixels; and

(ii) filtering each of the defined first subsets of pixels through a first multi-tap polyphase filter.

2. The sensor interface circuit of claim 1, wherein the scaling circuit further comprises:

a second scaling circuit configured to generate a scaled image data according to the operating parameters by filtering a plurality of lines of pixels scaled in the first direction by the first scaling circuit through a second multi-tap polyphase filter along a second direction perpendicular to the first direction.

3. The sensor interface circuit of claim 2, wherein the scaling circuit further comprises at least one line buffer between the first scaling circuit and the second scaling circuit to store at least one line of scaled pixels.

4. The sensor interface circuit of claim 2, wherein the scaling circuit further comprises:

a line buffer, configured to store one or more lines of pixels scaled in the first direction; and

a second defect detection circuit coupled to the line buffer and the first scaling circuit, the second defect detection circuit configured to detect a second defective pixel in the input image by analyzing, along the second direction, the plurality of lines of pixels scaled in the first direction, wherein

the second scaling circuit is further configured to define second subsets of pixels along the second direction in the plurality of lines of pixels scaled in the first direction for filtering by the second multi-tap polyphase filter so that the second defective pixel is included in only one of the second subsets of pixels.

5. The sensor interface circuit of claim 2, wherein the input image and the scaled image comprise Bayer pattern encoded data.

6. The sensor interface circuit of claim 1, wherein the first defect pixel detection circuit is configured to determine a pixel in the line as the first defective pixel based on a ratio between a pixel value of the first defective pixel relative to an average of pixel values of a predetermined number of pixels neighboring the first defective pixel.

7. The sensor interface circuit of claim 6, wherein the first defect pixel detection circuit is configured to determine a pixel in the line as the first defective pixel further based on a first ratio between (i) a largest pixel value difference between the neighboring pixels and (ii) a difference between a pixel value of the first defective pixel and a largest pixel value of the neighboring pixels or a second ratio between (i) the largest pixel value difference and (ii) a difference between a pixel value of the first defective pixel and a smallest pixel value of the neighboring pixels.

8. The sensor interface circuit of claim 1, wherein the first multi-tap polyphase filter comprises a two-tap polyphase filter.

9. The sensor interface circuit of claim 1, wherein the register circuit is further configured to receive the operating parameters for storing from a central control circuit during a configuration cycle.

10. A method of interfacing with an image sensor, comprising:

storing, by a register circuit, operating parameters for configuring operation of a sensor interface circuit;

detecting, by a first defect pixel detection circuit of the sensor interface circuit, a first defective pixel in an input image by analyzing pixels in a line of an input image along a first direction; and

generating, at a first scaling circuit of the sensor interface circuit, a scaled line of pixels representing the line of input image scaled along the first direction according to the operating parameters, the scaled line of pixels generated by;

defining first subsets of pixels in the line of the input image so that the first defective pixel is included in only one of the first subsets of pixels; and

filtering each of the defined first subsets of pixels through a first multi-tap polyphase filter.

11. The method of claim 10, further comprising:

detecting, at a second defect detection circuit, a second defective pixel in the input image by at least analyzing a line of pixels along a second direction in a plurality lines of pixels that are scaled by the first scaling circuit, the second direction perpendicular to the first direction;

defining second subsets of pixels along the second direction in the plurality of scaled lines of pixels so that the second defective pixel is included in only one of the second subsets of pixels; and

generating, by a second scaling circuit of the sensor interface circuit, a scaled image according to the operating parameters by filtering each of the second subset of pixels through a multi-tap polyphase filter along the second direction.

12. The method of claim 11, further comprising storing a part of the plurality lines of pixels in a line buffer, wherein the stored lines of pixels are read by the second defect detection circuit and the second scaling circuit.

13. The method of claim 11, wherein the input image and the scaled image comprise Bayer pattern encoded data.

14. The method of claim 10, wherein a pixel in the line is determined as the first defective pixel based on a ratio between a pixel value of the first defective pixel relative to an average of pixel values of a predetermined number of pixels neighboring the first defective pixel in the line.

15. The method of claim 14, wherein the pixel in the line is further determined as the defective pixel based on a first ratio between (i) a largest pixel value difference between the neighboring pixels and (ii) a difference between a pixel value of the first defective pixel and a largest pixel value of the neighboring pixels or a second ratio between (i) the largest pixel value difference and (ii) a difference between a pixel value of the first defective pixel and a smallest pixel value of the neighboring pixels.

16. The method of claim 10, wherein the first multi-tap polyphase filter comprises a two-tap polyphase filter.

17. An image signal processor, comprising:

a scaling circuit comprising;

a first defect pixel detection circuit configured to detect a first defective pixel in an input image by analyzing pixels in a line of an input image along a first direction;

a first scaling circuit coupled to the first defect pixel detection circuit, the first scaling circuit configured to generate a scaled line of pixels representing the line of the input image scaled along the first direction by;

(i) defining first subsets of pixels in the line of the input image so that the first defective pixel is included in only one of the first subsets of pixels; and

(ii) filtering each of the defined first subsets of pixels through a first multi-tap polyphase filter; and

a second scaling circuit configured to generate a scaled image by filtering a plurality of lines of pixels scaled in the first direction by the first scaling circuit through a second multi-tap polyphase filter along a second direction perpendicular to the first direction; and

a pipeline coupled to the scaling circuit and configured to perform processing on the scaled image.

18. The image signal processor of claim 17, wherein the input image and the scaled image comprise Bayer pattern encoded data.

19. The image signal processor of claim 17, wherein the scaling circuit further comprises:

a line buffer, configured to store one line of pixels scaled in the first direction; and

a second defect detection circuit coupled to the line buffer and the first scaling circuit, the second defect detection circuit configured to detect a second defective pixel in the input image by analyzing the plurality lines of pixels scaled in the first direction along the second direction,

wherein the second scaling circuit is further configured to define second subsets of pixels along the second direction in the plurality of lines of pixels scaled in the first direction for filtering by the second multi-tap polyphase filter so that the second defective pixel is included in only one of the second subsets of pixels.

20. The image signal processor of claim 17, wherein the first defect pixel detection circuit is configured to determine a pixel in the line as the first defective pixel based on:

a first ratio between a pixel value of the first defective pixel relative to an average of pixel values of a predetermined number of pixels neighboring the first defective pixel;

a second ratio between (i) a largest pixel value difference between the neighboring pixels and (ii) a difference between a pixel value of the first defective pixel and a largest pixel value of the neighboring pixels;

or

a third ratio between (i) the largest pixel value difference and (ii) a difference between a pixel value of the first defective pixel and a smallest pixel value of the neighboring pixels.