Method of processing noise in image data, noise reduction unit, and imaging apparatus

• US 7,825,964 B2
• Filed: 08/24/2007
• Issued: 11/02/2010
• Est. Priority Date: 09/28/2006
• Status: Active Grant

• Current Assignee: Sony Semiconductor Solutions Corporation (Sony Group Corp.)
• Original Assignee: Sony Corporation (Sony Group Corp.)
• Inventors: Hoshino, Kazuhiro, Sumi, Hirofumi
• Primary Examiner(s): Ye; Lin
• Assistant Examiner(s): CHEN, CHIA WEI A
• Application Number:
  US11/895,528
• Publication Number:
  US 20080079827A1
• Time in Patent Office: 1,166 Days
• Field of Search: 348/241, 348/248, 348/222.1, 382/275, 382/278, 382/279
• US Class Current: 348/241
• CPC Class Codes: H04N 9/646 for image enhancement, e.g. vertical detail restoration, cross-colour elimination, contour correction, chrominance trapping filters

Abstract

A method of processing noise in image data by an image processor having a signal-processing portion converting an image signal from an image sensor into a digital signal and outputting the converted signal as image data for each frame, the image data indicating sets of pixel values each having a brightness at a corresponding one of coordinate points arranged in directions of rows and columns is disclosed. The method includes the steps of: extracting pixel values; deciding pixel value; finding autocorrelation coefficients of pixel values which are less than a first threshold value; and deciding random noise in the image.

Claims

1. A method of processing noise in image data by an image processor having a signal-processing portion converting an image signal from an image sensor into a digital signal and outputting the converted signal as image data for each frame, the image data indicating sets of pixel values each having a brightness at a corresponding one of coordinate points arranged in directions of rows and columns, the method comprising the steps of:

extracting pixel values by accepting each frame of the image data delivered from the signal-processing portion in turn while shifting the image data at regular intervals of time in a direction of a time axis and extracting, in each frame, pixel values represented by the image data accepted for each frame;

deciding pixel value by comparing each pixel value extracted by the pixel values extracting step against a previously set first threshold value and making a decision as to whether the pixel value is less than the first threshold value;

finding autocorrelation coefficients of pixel values which are less than the first threshold value and which are possessed by pixels located at the same coordinate point defined by the sets of image data over plural ones of the frames accepted while being shifted at the regular intervals of time; and

deciding random noise in the image by making a decision as to whether the autocorrelation coefficients found in the autocorrelation coefficients finding step are less than a previously set second threshold value and determining that the image data has suffered from deterioration of image quality due to random noise when the autocorrelation coefficients are less than the second threshold value.

2. The method of processing noise in image data as set forth in claim 1, further comprising the step of averaging pixel values by selecting only pixels having pixel values less than the first threshold value, when the decision made at the random noise deciding step is that image quality has been deteriorated due to random noise, arithmetically averaging the selected pixel values, and modifying the pixel values to a pixel value equal to or larger than the first threshold value.

3. The method of processing noise in image data as set forth in claim 1, further comprising the step of determining the pixel values equal to or larger than the first threshold value as the pixel values of the pixels without modification when the decision made at the random noise deciding step is that the pixels of the pixels are equal to or larger the first threshold value.

4. The method of processing noise in image data as set forth in claim 3, wherein in the pixel values determining step, when the decision at the random noise deciding step is that the autocorrelation coefficients found in the autocorrelation coefficients finding step are equal to or larger than the second threshold value, pixel values corresponding to the autocorrelation coefficients are judged to be possessed by dark pixels but have low levels of random noise and are determined as the pixel values represented by the pixel data without modification.

5. The method of processing noise in image data as set forth in claim 1, wherein the pixel values extracting step, the pixel values deciding step, the autocorrelation coefficients finding step, and the random noise deciding step are repeated the same number of times as there are pixels having pixel values less than the first threshold value.

6. The method of processing noise in image data as set forth in claim 1, wherein in the pixel values deciding step, pixel values represented by the image data which are accepted for each frame and which are less than the first threshold value are plural pixel values in an identical coordinate area on an image in the frames.

7. A method of processing noise in image data by an image processor having a signal-processing portion for converting an image signal from an image sensor into a digital signal and outputting the converted signal as image data for each frame, the image data indicating sets of pixel values each having a brightness at a corresponding one of coordinate points arranged in directions of rows and columns, the method comprising the steps of:

extracting pixel values by accepting the sets of image data delivered from the signal-processing portion in turn for each frame while shifting the sets of image data at regular intervals of time in a direction of a time axis and extracting, in each frame, the pixel values represented by the image data accepted in each frame;

deciding pixel values by comparing each pixel value extracted by the pixel values extracting step against a previously set first threshold value and making a decision as to whether the pixel value is less than the first threshold value;

finding first autocorrelation coefficients of pixel values which are possessed by pixels located at the same coordinate point defined by the image data over plural ones of the frames accepted while being shifted at the regular intervals of time in the direction of the time axis;

deciding random noise in the image by making a decision as to whether the autocorrelation coefficients found in the autocorrelation coefficients finding step are less than a previously set second threshold value and determining that the image data has suffered from deterioration of image quality due to random noise when the autocorrelation coefficients are less than the second threshold value;

setting pixel values equal to or larger than the first threshold value as a second pixel value when the decision at the pixel values deciding step is that the pixel values of the pixels are equal to or larger than the first threshold value;

finding second autocorrelation coefficients by extracting pixel values which are equal to or larger than of the second pixel value and which are possessed by pixels located at the same coordinate point over plural frames accepted while being shifted at the regular intervals of time in the direction of the time axis and finding the autocorrelation coefficients of the extracted pixel values;

deciding autocorrelation coefficients by making a decision as to whether the autocorrelation coefficients found in the second autocorrelation coefficients finding step are less than a previously set third threshold value;

selecting pixels possessing pixel values having autocorrelation coefficients less than the third threshold value as pixels of a short-distance view of the image when the decision made at the autocorrelation coefficients deciding step is that the autocorrelation coefficients are less than the third threshold value; and

making a correction for camera shake regarding the pixels of the short-distance view selected in the short-distance view selecting step.

8. The method of processing noise in image data as set forth in claim 7, further comprising the step of averaging pixel values by selecting only pixels having pixel values less than the first threshold value, when the decision made at the random noise deciding step is that image quality has been deteriorated due to random noise, arithmetically averaging the selected pixel values, and modifying the pixel values to a pixel value equal to or larger than the first threshold value.

9. The method of processing noise in image data as set forth in claim 7, further comprising the step of determining the pixel values equal to or larger than the first threshold value as the pixel values of the pixels without modification when the decision made at the random noise deciding step is that the pixel values of the pixels are equal to or larger than the first threshold value.

10. The method of processing noise in image data as set forth in claim 9, wherein when the decision made in the random noise deciding step is that the autocorrelation coefficients found in the autocorrelation coefficients deciding step are equal to or larger than the second threshold value, pixel values corresponding to the autocorrelation coefficients are judged to be possessed by dark pixels but have low levels of random noise and are determined as the pixel values represented by the pixel data without modification.

11. The method of processing noise in image data as set forth in claim 7, wherein the pixel values extracting step, the pixel values deciding step, the first autocorrelation coefficients finding step, and the random noise deciding step are repeated the same number of times as there are pixels which are extracted by the pixel values extracting step and which have pixel values less than the first threshold value.

12. The method of processing noise in image data as set forth in claim 7, wherein in the pixel values deciding step, pixel values represented by the image data which are accepted for each frame and which are less than the first threshold value are plural pixel values in an identical coordinate area on an image in the frames.

13. A noise reduction unit for processing noise in image data, comprising:

an image sensor;

a signal-processing portion for converting an image signal from the image sensor into a digital signal and outputting the converted signal as sets of image data for each frame, each set of image data indicating a pixel value having a brightness at a corresponding one of coordinate points arranged in directions of rows and columns;

pixel value extraction means for accepting the sets of image data from the signal-processing portion in turn for each frame while shifting the sets of image data at regular intervals of time in a direction of a time axis and extracting, in each frame, the pixel values represented by the image data accepted in each frame;

pixel value decision means for comparing each pixel value extracted by the pixel value extraction means against a previously set first threshold value and making a decision as to whether the pixel value is less than the first threshold value;

autocorrelation coefficient calculation means for finding autocorrelation coefficients of the pixel values which are less than the first threshold value and which are possessed by the pixels located at the same coordinate point defined by the image data over plural ones of the frames accepted while being shifted at the regular intervals of time in the direction of the time axis; and

random noise decision means for making a decision as to whether each autocorrelation coefficient found by the autocorrelation coefficient calculation means is less than a previously set second threshold value and determining that the image data has suffered from deterioration of image quality due to random noise when the decision is that the autocorrelation coefficient is less than the second threshold value.

14. The noise reduction unit for processing noise in image data as set forth in claim 13, further comprising averaging means for selecting only pixels having pixel values less than the first threshold value when the random noise decision means has determined that image quality has been deteriorated due to random noise, arithmetically averaging the selected pixel values, and modifying the pixel values to a pixel value equal to or larger than the first threshold value.

15. The noise reduction unit for processing noise in image data as set forth in claim 13, further comprising pixel value determination means for determining pixel values equal to or larger than the first threshold value as the pixel values of the pixels without modification when the pixel value decision means has determined that the pixel values of the pixels are equal to or larger than the first threshold value.

16. The noise reduction unit for processing noise in image data as set forth in claim 13, wherein when the random noise decision means has determined that the autocorrelation coefficients found by the autocorrelation coefficient calculation means are equal to or larger than the second threshold value, the pixel value determination means judges pixel values corresponding to the autocorrelation coefficients to be possessed by dark pixels but have low levels of random noise and determines the pixel values possessed by the pixels as the pixel values indicated by the pixel data without modification.

17. The noise reduction unit for processing noise in image data as set forth in claim 13, wherein operations respectively performed by the pixel value extraction means, the pixel value decision means, the autocorrelation coefficient calculation means, and the random noise decision means are repeated the same number of times as there are pixels which are extracted by the pixel value extraction means and which have pixel values less than the first threshold value.

18. The noise reduction unit for processing image in image data as set forth in claim 13, wherein in the pixel value decision means, pixel values represented by the image data which are accepted for each frame and which are less than the first threshold value are plural pixel values in an identical coordinate area on an image in the frames.

19. A noise reduction unit for processing noise in image data, comprising:

an image sensor;

a signal-processing portion for converting an image signal from the image sensor into a digital signal and outputting the converted signal as sets of image data for each frame, each set of image data indicating a pixel value having a brightness at a corresponding one of coordinate points arranged in directions of rows and columns;

pixel value extraction means for accepting the sets of image data delivered from the signal-processing portion in turn for each frame while shifting the image data at regular intervals of time in a direction of a time axis and extracting, in each frame, the pixel values represented by the image data accepted in each frame;

pixel value decision means for comparing each pixel value extracted by the pixel value extraction means against a previously set first threshold value and making a decision as to whether the pixel value is less than the first threshold value;

first autocorrelation coefficient calculation means for finding autocorrelation coefficients of pixel values which are less than the first threshold value and which are possessed by pixels located at the same coordinate point defined by the image data over plural ones of the frames accepted while being shifted at the regular intervals of time in the direction of the time axis;

random noise decision means for making a decision as to whether the autocorrelation coefficients found by the first autocorrelation coefficient calculation means are less than a previously set second threshold value and, when the autocorrelation coefficients are judged to be less than the second threshold value, determining that the image data has suffered from deterioration of image quality due to random noise;

pixel value setting means which, when the pixel value decision means has determined that the pixel values of the pixels are equal to or larger than the first threshold value, sets the pixel values equal to or larger than the first threshold value as a second threshold value;

second autocorrelation coefficient calculation means for extracting pixel values which are equal to or larger than the second threshold value and which are possessed by pixels located at the same coordinate point over plural ones of the frames accepted while being shifted at the regular intervals of time in the direction of the time axis and finding autocorrelation coefficients of the extracted pixel values;

autocorrelation coefficient decision means for making a decision as to whether the autocorrelation coefficients found by the second autocorrelation coefficient calculation means are less than a previously set third threshold value;

short-distance view selection means for selecting pixels possessing pixel values having autocorrelation coefficients less than the third threshold value as pixels of a short-distance view of the image when the autocorrelation coefficients are judged to be less than the third threshold value by the autocorrelation coefficient decision means; and

camera shake correction means for making a correction for camera shake of the pixels of the short-distance view of the image selected by the short-distance view selection means.

20. The noise reduction unit for processing noise in image data as set forth in claim 19, further comprising averaging means for selecting only pixels having pixel values less than the first threshold value when the random noise decision means has determined that image quality has been deteriorated due to random noise, arithmetically averaging the selected pixel values, and modifying the pixel values to a pixel value equal to or larger than the first threshold value.

21. The noise reduction unit for processing image in image data as set forth in claim 19, further comprising pixel value determination means for determining pixel values equal to or larger than the first threshold value as the pixel values of the pixels without modification when the pixel value decision means has determined that the pixel values of the pixels are equal to or larger than the first threshold value.

22. The noise reduction unit for processing noise in image data as set forth in claim 19, wherein when the random noise decision means has determined that the autocorrelation coefficients found by the first autocorrelation coefficient calculation means are equal to or larger than the second threshold value, the pixel value determination means judges pixel values corresponding to the autocorrelation coefficients to be possessed by dark pixels but have low levels of random noise and determines the pixel values possessed by the pixels as the pixel values represented by the pixel data without modification.

23. The noise reduction unit for processing noise in image data as set forth in claim 19, wherein operations respectively performed by the pixel value extraction means, the pixel value decision means, the autocorrelation coefficient calculation means, and random noise decision means are repeated the same number of times as there are pixels which are extracted by the pixel value extraction means and which have pixel values less than the first threshold value.

24. The noise reduction unit for processing noise in image data as set forth in claim 19, wherein pixel values which are accepted for each of the pixels and which are judged to be less than the first threshold value by the pixel value decision means are plural pixel values in an identical coordinate area on an image in the frames.

25. An imaging apparatus comprising:

an image sensor;

an optical system for guiding incident light from a subject into the image sensor;

a signal-processing portion for converting an image signal from the image sensor into a digital signal and outputting the converted signal as image data for each frame, the image data including pixel values each having a brightness at a corresponding one of coordinate points arranged in directions of rows and columns; and

a random noise decision/reduction processing portion which judges and/or reduces random noise generated by the image sensor;

wherein the random noise decision/reduction processing portion has;

pixel value extraction means for accepting the image data delivered from the signal-processing portion in turn for each frame while shifting the image data at regular intervals of time in a direction of a time axis and extracting, in each frame, the pixel value represented by the image data accepted in each frame;

pixel value decision means for comparing each pixel value extracted by the pixel value extraction means against a previously set first threshold value and making a decision as to whether the pixel value is less than the first threshold value;

first autocorrelation coefficient calculation means for finding autocorrelation coefficients of the pixel values which are less than the first threshold value and which are possessed by the pixels located at the same coordinate point defined by the image data over plural ones of the frames accepted while being shifted at the regular intervals of time in the direction of the time axis; and

random noise decision means for making a decision as to whether the autocorrelation coefficients found by the first autocorrelation coefficient calculation means are less than a previously set second threshold value and, when the autocorrelation coefficients are judged to be less than the second threshold value, determining that the image data has suffered from deterioration of image quality due to random noise.

26. The imaging apparatus as set forth in claim 25, wherein the random noise decision/reduction processing portion further includes:

pixel value setting means for setting the pixel values equal to or larger than the first threshold value as a second pixel value when the pixel value decision means has determined that the pixel values of the pixels are equal to or larger than the first threshold value;

second autocorrelation coefficient calculation means for extracting pixel values which are equal to or larger than the second pixel value and which are possessed by pixels located at the same coordinate point over plural ones of the plural frames accepted while being shifted at the regular intervals of time in the direction of the time axis and finding autocorrelation coefficients of the extracted pixel values;

autocorrelation coefficient decision means for making a decision as to whether the autocorrelation coefficients found by the second autocorrelation coefficient calculation means are less than a previously set third threshold value;

short-distance view selection means for selecting pixels possessing pixel values having autocorrelation coefficients less than the third threshold value as pixels of a short-distance view of the image when the autocorrelation coefficients are judged to be less than the third threshold value by the autocorrelation coefficient decision means; and

camera shake correction means for making a correction for camera shake of the pixels of the short-distance view of the image selected by the short-distance view selection means.

27. The imaging apparatus as set forth in claim 25, wherein the random noise decision/reduction processing portion further includes averaging means for selecting only pixels having pixel values less than the first threshold value when the random noise decision means has determined that image quality has been deteriorated due to random noise, arithmetically averaging the selected pixel values, and modifying the pixel values to a pixel value equal to or larger than the first threshold value.

28. The imaging apparatus as set forth in claim 25, further comprising pixel value determination means for determining pixel values equal to or larger than the first threshold value as the pixel values of the pixels without modification when the pixel value decision means has determined that the pixel values of the pixels are equal to or larger than the first threshold value.

29. The imaging apparatus as set forth in claim 25, wherein when the random noise decision means has determined that the autocorrelation coefficients found by the first autocorrelation coefficient calculation means are equal to or larger than the second threshold value, the pixel value determination means judges pixel values corresponding to the autocorrelation coefficients to be possessed by dark pixels but have low levels of random noise and determines the pixel values possessed by the pixels as the pixel values represented by the pixel data without modification.

30. The imaging apparatus as set forth in claim 25, wherein operations respectively performed by the pixel value extraction means, the pixel value decision means, the autocorrelation coefficient calculation means, and the random noise decision means are repeated the same number of times as there are pixels which are extracted by the pixel value extraction means and which are possessed by pixel values less than the first threshold value.

31. The imaging apparatus as set forth in claim 25, wherein pixel values which are accepted for each of the pixels and which are judged to be less than the first threshold value by the pixel value decision means are plural pixel values in an identical coordinate area on an image in the frames.

32. A noise reduction unit for processing noise in image data, comprising:

an image sensor;

a signal-processing portion for converting an image signal from the image sensor into a digital signal and outputting the converted signal as sets of image data for each frame, each set of image data indicating a pixel value having a brightness at a corresponding one of coordinate points arranged in directions of rows and columns;

a pixel value extraction section which accepts the sets of image data from the signal-processing portion in turn for each frame while shifting the sets of image data at regular intervals of time in a direction of a time axis and extracts, in each frame, the pixel values represented by the image data accepted in each frame;

a pixel value decision section which compares each pixel value extracted by the pixel value extraction section against a previously set first threshold value and makes a decision as to whether the pixel value is less than the first threshold value;

an autocorrelation coefficient calculation section which finds autocorrelation coefficients of the pixel values which are less than the first threshold value and which are possessed by the pixels located at the same coordinate point defined by the image data over plural ones of the frames accepted while being shifted at the regular intervals of time in the direction of the time axis; and

a random noise decision section which makes a decision as to whether each autocorrelation coefficient found by the autocorrelation coefficient calculation section is less than a previously set second threshold value and determines that the image data has suffered from deterioration of image quality due to random noise when the decision is that the autocorrelation coefficient is less than the second threshold value.

33. A noise reduction unit for processing noise in image data, comprising:

an image sensor;

a signal-processing portion for converting an image signal from the image sensor into a digital signal and outputting the converted signal as sets of image data for each frame, each set of image data indicating a pixel value having a brightness at a corresponding one of coordinate points arranged in directions of rows and columns;

a pixel value extraction section which accepts the sets of image data delivered from the signal-processing portion in turn for each frame while shifting the image data at regular intervals of time in a direction of a time axis and extracts, in each frame, the pixel values represented by the image data accepted in each frame;

a pixel value decision section which compares each pixel value extracted by the pixel value extraction section against a previously set first threshold value and makes a decision as to whether the pixel value is less than the first threshold value;

a first autocorrelation coefficient calculation section which finds autocorrelation coefficients of pixel values which are less than the first threshold value and which are possessed by pixels located at the same coordinate point defined by the image data over plural ones of the frames accepted while being shifted at the regular intervals of time in the direction of the time axis;

a random noise decision section which makes a decision as to whether the autocorrelation coefficients found by the first autocorrelation coefficient calculation section are less than a previously set second threshold value and, when the autocorrelation coefficients are judged to be less than the second threshold value, determines that the image data has suffered from deterioration of image quality due to random noise;

a pixel value setting section which, when the pixel value decision section has determined that the pixel values of the pixels are equal to or larger than the first threshold value, sets the pixel values equal to or larger than the first threshold value as a second threshold value;

a second autocorrelation coefficient calculation section which extracts pixel values which are equal to or larger than the second threshold value and which are possessed by pixels located at the same coordinate point over plural ones of the frames accepted while being shifted at the regular intervals of time in the direction of the time axis and finding autocorrelation coefficients of the extracted pixel values;

an autocorrelation coefficient decision section which makes a decision as to whether the autocorrelation coefficients found by the second autocorrelation coefficient calculation section are less than a previously set third threshold value;

a short-distance view selection section which selects pixels possessing pixel values having autocorrelation coefficients less than the third threshold value as pixels of a short-distance view of the image when the autocorrelation coefficients are judged to be less than the third threshold value by the autocorrelation coefficient decision section; and

a camera shake correction section which makes a correction for camera shake of the pixels of the short-distance view of the image selected by the short-distance view selection section.

34. An imaging apparatus comprising:

an image sensor;

an optical system for guiding incident light from a subject into the image sensor;

a signal-processing portion for converting an image signal from the image sensor into a digital signal and outputting the converted signal as image data for each frame, the image data including pixel values each having a brightness at a corresponding one of coordinate points arranged in directions of rows and columns; and

a random noise decision/reduction processing portion which judges and/or reduces random noise generated by the image sensor;

wherein the random noise decision/reduction processing portion has;

a pixel value extraction section which accepts the image data delivered from the signal-processing portion in turn for each frame while shifting the image data at regular intervals of time in a direction of a time axis and extracts, in each frame, the pixel value represented by the image data accepted in each frame;

a pixel value decision section which compares each pixel value extracted by the pixel value extraction section against a previously set first threshold value and makes a decision as to whether the pixel value is less than the first threshold value;

a first autocorrelation coefficient calculation section which finds autocorrelation coefficients of the pixel values which are less than the first threshold value and which are possessed by the pixels located at the same coordinate point defined by the image data over plural ones of the frames accepted while being shifted at the regular intervals of time in the direction of the time axis; and

a random noise decision section which makes a decision as to whether the autocorrelation coefficients found by the first autocorrelation coefficient calculation section are less than a previously set second threshold value and, when the autocorrelation coefficients are judged to be less than the second threshold value, determines that the image data has suffered from deterioration of image quality due to random noise.