Method for mismatch detection between the frequency of illumination source and the duration of optical integration time for imager with rolling shutter

• US 7,142,234 B2
• Filed: 12/10/2002
• Issued: 11/28/2006
• Est. Priority Date: 12/10/2002
• Status: Active Grant

• Current Assignee: Aptina Imaging Corporation (ON Semiconductor Corporation)
• Original Assignee: Micron Technology, Inc.
• Inventors: Subbotin, Igor, Kaplinsky, Michael
• Primary Examiner(s): Vu, Ngoc-Yen
• Assistant Examiner(s): DURNFORD GESZVAIN, DILLON
• Application Number:
  US10/315,213
• Publication Number:
  US 20040109069A1
• Time in Patent Office: 1,449 Days
• Field of Search: 348/226.1, 348/228.1
• US Class Current: 348/226.1
• CPC Class Codes: H04N 23/70 Circuitry for compensating brightness variation in the scene
• CPC Class Codes: H04N 23/73 by influencing the exposure time
• CPC Class Codes: H04N 23/745 Detection of flicker frequency or suppression of flicker wherein the flicker is caused by illumination, e.g. due to fluorescent tube illumination or pulsed LED illumination

Abstract

A method for achieving flickerless operation of imagers using a rolling shutter, including the steps of detecting flicker in an image frame and adjusting an integration time such that the integration time is an integer multiple of a light intensity period, is described. The method reduces the task of detecting flicker to detecting a spatial sine wave in the image frame by subtracting two image frames from each other. The subtraction reduces or eliminates image content from the image frames and makes the detection process independent of image content. This method can be practiced based on a very short sequence of image frames (as few as two image frames) and is virtually insusceptible to motion of the image content. Adjustments to the integration time are made by changing the horizontal blanking time of the imager and, thereby extending the duration time of each row of said image frame.

Claims

1. A method for achieving flickerless operation of imagers using a rolling shutter comprising the steps of:

first detecting flicker in an image frame caused by an illumination source having a non-zero light intensity period; and

adjusting an integration time such that said integration time is an integer multiple of said non-zero light intensity period,

wherein said first detecting step further comprises the steps of;

calculating an expected frequency of a sine wave, said sine wave representing a spatial period of said flicker in said image frame;

second detecting a sine wave intensity fluctuation in a difference of two image frames; and

third detecting a sine intensity pattern in a direction of a scan.

2. The method according to claim 1, wherein said second detecting step further comprises the step of subtracting two image frames from each other to create a difference array.

3. The method according to claim 2, further comprising the step of smoothing said difference array.

4. The method according to claim 3, wherein said smoothing step further comprises the step of substituting each element of said difference array with an average of some of said each element'"'"'s nearest neighbors.

5. The method according to claim 3, wherein said third detecting step is accomplished by locating positions of first three zero-crossing points in said smoothed difference array.

6. The method according to claim 5, further comprising the steps of:

scanning said smoothed difference array and comparing signs of all neighboring elements to each other, until said first three zero-crossing points are located;

computing a distance between a first zero-crossing point and a third zero-crossing point, said distance between said first zero-crossing point and said third zero-crossing point representing a period of a spatial flicker-induced wave;

comparing said period of said spatial flicker-induced wave with said calculated expected frequency of said sine wave; and

determining if flicker has been detected based on a preset tolerance for said comparison between said period of said spatial flicker-induced wave with said calculated expected frequency of said sine wave.

7. The method according to claim 2, wherein said subtracting step further comprises the step of selecting said two image frames to be subtracted from each other to be “

n”

image frames apart, where “

n”

is a small integer number of image frames.

8. The method according to claim 1, wherein said third detecting step is accomplished by locating positions of first three zero-crossing points in said difference array.

9. The method according to claim 8, further comprising the steps of:

scanning said difference array and comparing signs of all neighboring elements to each other, until said first three zero-crossing points are located;

computing a distance between a first zero-crossing point and a third zero-crossing point, said distance between said first zero-crossing point and said third zero-crossing point representing a period of a spatial flicker-induced wave;

comparing said period of said spatial flicker-induced wave with said calculated expected frequency of said sine wave; and

determining if flicker has been detected based on a preset tolerance for said comparison between said period of said spatial flicker-induced wave with said calculated expected frequency of said sine wave.

10. The method according to claim 1, wherein said adjusting step is based on changing a horizontal blanking time of said imager.

11. The method according to claim 1, wherein said imager is a CMOS imager.

12. The method according to claim 1, wherein said second detecting step further comprises the steps of:

averaging all pixels in each image frame row;

calculating an average pixel value per image frame row;

storing said calculated average pixel value per image frame row; and

subtracting average pixel values per image frame row of a subsequent image frame from corresponding stored average pixel values per image frame row.

13. The method according to claim 1, wherein said third detecting step is accomplished by locating positions of a plurality of zero-crossing points in said difference array.

14. The method according to claim 13, further comprising the steps of:

scanning said difference array and comparing signs of all neighboring elements, until said all of said plurality of zero-crossing points are located;

computing a distance between every other zero-crossing point, said distance between said every other zero-crossing point representing a period of a spatial flicker-induced wave;

comparing each said period of said spatial flicker-induced wave with said calculated expected frequency of said sine wave; and

determining if flicker has been detected based on a preset tolerance for said comparison between each said period of said spatial flicker-induced wave with said calculated expected frequency of said sine wave.

15. The method according to claim 13, further comprising the step of smoothing said difference array.

16. The method according to claim 15, further comprising the steps of:

scanning said smoothed difference array and comparing signs of all neighboring elements to each other, until said all of said plurality of zero-crossing points are located;

computing a distance between every other zero-crossing point, said distance between said every other zero-crossing point representing a period of a spatial flicker-induced wave;

comparing each said period of said spatial flicker-induced wave with said calculated expected frequency of said sine wave; and

determining if flicker has been detected based on a preset tolerance for said comparison between each said period of said spatial flicker-induced wave with said calculated expected frequency of said sine wave.

17. The method according to claim 1, wherein said first detecting step is repeated for a preset period of time.

18. An image processor for achieving flickerless operation of images comprising:

an imager using a rolling shutter; and

a flicker avoidance circuit, wherein said flicker avoidance circuit performs the following functions;

first detecting flicker in an image frame caused by an illumination source having a non-zero light intensity period; and

adjusting an integration time such that said integration time is an integer multiple of said non-zero light intensity period,

wherein said first detecting step further comprises the steps of;

calculating an expected frequency of a sine wave, said sine wave representing a spatial period of said flicker in said image frame;

second detecting a sine wave intensity fluctuation in a difference of two image frames; and

third detecting a sine intensity pattern in a direction of a scan.

19. The image processor according to claim 18, wherein said second detecting step further comprises the step of subtracting two image frames from each other to create a difference array.

20. The image processor according to claim 19, further comprising the step of smoothing said difference array.

21. The image processor according to claim 20, wherein said smoothing step further comprises the step of substituting each element of said difference array with an average of some of said each element'"'"'s nearest neighbors.

22. The image processor according to claim 20, wherein said third detecting step is accomplished by locating positions of first three zero-crossing points in said smoothed difference array.

23. The image processor according to claim 22, further comprising the steps of:

scanning said smoothed difference array and comparing signs of all neighboring elements to each other, until said first three zero-crossing points are located;

computing a distance between a first zero-crossing point and a third zero-crossing point, said distance between said first zero-crossing point and said third zero-crossing point representing a period of a spatial flicker-induced wave;

comparing said period of said spatial flicker-induced wave with said calculated expected frequency of said sine wave; and

determining if flicker has been detected based on a preset tolerance for said comparison between said period of said spatial flicker-induced wave with said calculated expected frequency of said sine wave.

24. The image processor according to claim 19, wherein said subtracting step further comprises the step of selecting said two image frames to be subtracted from each other to be “

n”

image frames apart, where “

n”

is a small integer number of image frames.

25. The image processor according to claim 18, wherein said third detecting step is accomplished by locating positions of first three zero-crossing points in said difference array.

26. The image processor according to claim 25, further comprising the steps of:

scanning said difference array and comparing signs of all neighboring elements to each other, until said first three zero-crossing points are located;

computing a distance between a first zero-crossing point and a third zero-crossing point, said distance between said first zero-crossing point and said third zero-crossing point representing a period of a spatial flicker-induced wave;

comparing said period of said spatial flicker-induced wave with said calculated expected frequency of said sine wave; and

determining if flicker has been detected based on a preset tolerance for said comparison between said period of said spatial flicker-induced wave with said calculated expected frequency of said sine wave.

27. The image processor according to claim 18, wherein said adjusting step is based on changing a horizontal blanking time of said imager.

28. The image processor according to claim 18, wherein said imager is a CMOS imager.

29. The image processor according to claim 18, wherein said second detecting step further comprises the steps of:

averaging all pixels in each image frame row;

calculating an average pixel value per image frame row;

storing said calculated average pixel value per image frame row; and

subtracting average pixel values per image frame row of a subsequent image frame from corresponding stored average pixel values per image frame row.

30. The image processor according to claim 18, wherein said third detecting step is accomplished by locating positions of a plurality of zero-crossing points in said difference array.

31. The image processor according to claim 30, further comprising the steps of:

scanning said difference array and comparing signs of all neighboring elements, until said all of said plurality of zero-crossing points are located;

computing a distance between every other zero-crossing point, said distance between said every other zero-crossing point representing a period of a spatial flicker-induced wave;

comparing each said period of said spatial flicker-induced wave with said calculated expected frequency of said sine wave; and

determining if flicker has been detected based on a preset tolerance for said comparison between each said period of said spatial flicker-induced wave with said calculated expected frequency of said sine wave.

32. The image processor according to claim 30, further comprising the step of smoothing said difference array.

33. The image processor according to claim 32, further comprising the steps of:

scanning said smoothed difference array and comparing signs of all neighboring elements to each other, until said all of said plurality of zero-crossing points are located;

computing a distance between every other zero-crossing point, said distance between said every other zero-crossing point representing a period of a spatial flicker-induced wave;

comparing each said period of said spatial flicker-induced wave with said calculated expected frequency of said sine wave; and

determining if flicker has been detected based on a preset tolerance for said comparison between each said period of said spatial flicker-induced wave with said calculated expected frequency of said sine wave.

34. The image processor according to claim 18, wherein said first detecting step is repeated for a preset period of time.

35. An image processing system comprising:

a processor;

a memory device coupled to said processor; and

an image device coupled to said processor, said image device further comprising;

an image processor using a rolling shutter; and

a flicker avoidance circuit, wherein said flicker avoidance circuit performs the following functions;

first detecting flicker in an image frame caused by an illumination source having a non-zero light intensity period; and

adjusting an integration time such that said integration time is an integer multiple of said non-zero light intensity period,

wherein said first detecting step further comprises the steps of;

calculating an expected frequency of a sine wave, said sine wave representing a spatial period of said flicker in said image frame;

second detecting a sine wave intensity fluctuation in a difference of two image frames; and

third detecting a sine intensity pattern in a direction of a scan.

36. The image processing system according to claim 35, wherein said second detecting step further comprises the step of subtracting two image frames from each other to create a difference array.

37. The image processing system according to claim 36, further comprising the step of smoothing said difference array.

38. The image processing system according to claim 37, wherein said smoothing step further comprises the step of substituting each element of said difference array with an average of some of said each element'"'"'s nearest neighbors.

39. The image processing system according to claim 37, wherein said third detecting step is accomplished by locating positions of first three zero-crossing points in said smoothed difference array.

40. The image processing system according to claim 39, further comprising the steps of:

scanning said smoothed difference array and comparing signs of all neighboring elements to each other, until said first three zero-crossing points are located;

computing a distance between a first zero-crossing point and a third zero-crossing point, said distance between said first zero-crossing point and said third zero-crossing point representing a period of a spatial flicker-induced wave;

comparing said period of said spatial flicker-induced wave with said calculated expected frequency of said sine wave; and

determining if flicker has been detected based on a preset tolerance for said comparison between said period of said spatial flicker-induced wave with said calculated expected frequency of said sine wave.

41. The image processing system according to claim 36, wherein said subtracting step further comprises the step of selecting said two image frames to be subtracted from each other to be “

n”

image frames apart, where “

n”

is a small integer number of image frames.

42. The image processing system according to claim 35, wherein said third detecting step is accomplished by locating positions of first three zero-crossing points in said difference array.

43. The image processing system according to claim 42, further comprising the steps of:

scanning said difference array and comparing signs of all neighboring elements to each other, until said first three zero-crossing points are located;

computing a distance between a first zero-crossing point and a third zero-crossing point, said distance between said first zero-crossing point and said third zero-crossing point representing a period of a spatial flicker-induced wave;

comparing said period of said spatial flicker-induced wave with said calculated expected frequency of said sine wave; and

determining if flicker has been detected based on a preset tolerance for said comparison between said period of said spatial flicker-induced wave with said calculated expected frequency of said sine wave.

44. The image processing system according to claim 35, wherein said adjusting step is based on changing a horizontal blanking time of said imager.

45. The image processing system according to claim 35, wherein said imager is a CMOS imager.

46. The image processing system according to claim 35, wherein said second detecting step further comprises the steps of:

averaging all pixels in each image frame row;

calculating an average pixel value per image frame row;

storing said calculated average pixel value per image frame row; and

subtracting average pixel values per image frame row of a subsequent image frame from corresponding stored average pixel values per image frame row.

47. The image processing system according to claim 35, wherein said third detecting step is accomplished by locating positions of a plurality of zero-crossing points in said difference array.

48. The image processing system according to claim 47, further comprising the steps of:

scanning said difference array and comparing signs of all neighboring elements, until said all of said plurality of zero-crossing points are located;

computing a distance between every other zero-crossing point, said distance between said every other zero-crossing point representing a period of a spatial flicker-induced wave;

comparing each said period of said spatial flicker-induced wave with said calculated expected frequency of said sine wave; and

determining if flicker has been detected based on a preset tolerance for said comparison between each said period of said spatial flicker-induced wave with said calculated expected frequency of said sine wave.

49. The image processing system according to claim 47, further comprising the step of smoothing said difference array.

50. The image processing system according to claim 49, further comprising the steps of:

scanning said smoothed difference array and comparing signs of all neighboring elements to each other, until said all of said plurality of zero-crossing points are located;

computing a distance between every other zero-crossing point, said distance between said every other zero-crossing point representing a period of a spatial flicker-induced wave;

comparing each said period of said spatial flicker-induced wave with said calculated expected frequency of said sine wave; and

determining if flicker has been detected based on a preset tolerance for said comparison between each said period of said spatial flicker-induced wave with said calculated expected frequency of said sine wave.

51. The image processing system according to claim 35, wherein said first detecting step is repeated for a preset period of time.

52. An image processing system comprising:

a CPU;

an image system coupled to said CPU, said image system comprising an image processor, said image processor comprising;

a rolling shutter; and

a flicker avoidance circuit, wherein said flicker avoidance circuit performs the following functions;

first detecting flicker in an image frame caused by an illumination source having a non-zero light intensity period; and

adjusting an integration time such that said integration time is an integer multiple of said non-zero light intensity period,

wherein said first detecting step further comprises the steps of;

calculating an expected frequency of a sine wave, said sine wave representing a spatial period of said flicker in said image frame;

second detecting a sine wave intensity fluctuation in a difference of two image frames; and

third detecting a sine intensity pattern in a direction of a scan.

53. The image processing system according to claim 52, wherein said second detecting step further comprises the step of subtracting two image frames from each other to create a difference array.

54. The image processing system according to claim 52, further comprising the step of smoothing said difference array.

55. The image processing system according to claim 54, wherein said smoothing step further comprises the step of substituting each element of said difference array with an average of some of said each element'"'"'s nearest neighbors.

56. The image processing system according to claim 54, wherein said third detecting step is accomplished by locating positions of first three zero-crossing points in said smoothed difference array.

57. The image processing system according to claim 56, further comprising the steps of:

scanning said smoothed difference array and comparing signs of all neighboring elements to each other, until said first three zero-crossing points are located;

computing a distance between a first zero-crossing point and a third zero-crossing point, said distance between said first zero-crossing point and said third zero-crossing point representing a period of a spatial flicker-induced wave;

comparing said period of said spatial flicker-induced wave with said calculated expected frequency of said sine wave; and

determining if flicker has been detected based on a preset tolerance for said comparison between said period of said spatial flicker-induced wave with said calculated expected frequency of said sine wave.

58. The image processing system according to claim 52, wherein said subtracting step further comprises the step of selecting said two image frames to be subtracted from each other to be “

n”

image frames apart, where “

n”

is a small integer number of image frames.

59. The image processing system according to claim 52, wherein said third detecting step is accomplished by locating positions of first three zero-crossing points in said difference array.

60. The image processing system according to claim 59, further comprising the steps of:

scanning said difference array and comparing signs of all neighboring elements to each other, until said first three zero-crossing points are located;

computing a distance between a first zero-crossing point and a third zero-crossing point, said distance between said first zero-crossing point and said third zero-crossing point representing a period of a spatial flicker-induced wave;

comparing said period of said spatial flicker-induced wave with said calculated expected frequency of said sine wave; and

determining if flicker has been detected based on a preset tolerance for said comparison between said period of said spatial flicker-induced wave with said calculated expected frequency of said sine wave.

61. The image processing system according to claim 52, wherein said adjusting step is based on changing a horizontal blanking time of said imager.

62. The processing system according to claim 52, wherein said imager is a CMOS imager.

63. The image processing system according to claim 52, wherein said second detecting step further comprises the steps of:

averaging all pixels in each image frame row;

calculating an average pixel value per image frame row;

storing said calculated average pixel value per image frame row; and

subtracting average pixel values per image frame row of a subsequent image frame from corresponding stored average pixel values per image frame row.

64. The image processing system according to claim 52, wherein said third detecting step is accomplished by locating positions of a plurality of zero-crossing points in said difference array.

65. The image processing system according to claim 64, further comprising the steps of:

scanning said difference array and comparing signs of all neighboring elements, until said all of said plurality of zero-crossing points are located;

computing a distance between every other zero-crossing point, said distance between said every other zero-crossing point representing a period of a spatial flicker-induced wave;

comparing each said period of said spatial flicker-induced wave with said calculated expected frequency of said sine wave; and

determining if flicker has been detected based on a preset tolerance for said comparison between each said period of said spatial flicker-induced wave with said calculated expected frequency of said sine wave.

66. The image processing system according to claim 64, further comprising the step of smoothing said difference array.

67. The image processing system according to claim 66, further comprising the steps of:

scanning said smoothed difference array and comparing signs of all neighboring elements to each other, until said all of said plurality of zero-crossing points are located;

computing a distance between every other zero-crossing point, said distance between said every other zero-crossing point representing a period of a spatial flicker-induced wave;

comparing each said period of said spatial flicker-induced wave with said calculated expected frequency of said sine wave; and

determining if flicker has been detected based on a preset tolerance for said comparison between each said period of said spatial flicker-induced wave with said calculated expected frequency of said sine wave.

68. The image processing system according to claim 52, wherein said first detecting step is repeated for a preset period of time.