Automated statistical self-calibrating detection and removal of blemishes in digital images based on determining probabilities based on image analysis of single images

• US 7,424,170 B2
• Filed: 09/30/2003
• Issued: 09/09/2008
• Est. Priority Date: 09/30/2003
• Status: Expired due to Fees

• Current Assignee: FotoNation Limited (Adeia Inc.)
• Original Assignee: FotoNation Vision Limited (Adeia Inc.)
• Inventors: Steinberg, Eran, Prilutsky, Yury, Zamfir, Adrian, Bigioi, Petronel, Corcoran, Peter, Buzuloiu, Vasile, Zamfir, Marta, Ursu, Danutz
• Primary Examiner(s): Kassa, Yosef
• Application Number:
  US10/677,140
• Publication Number:
  US 20050068451A1
• Time in Patent Office: 1,806 Days
• Field of Search: 382/260, 382/264, 382/274, 382/275, 382/278, 358/3.26, 358/3.27, 358/520, 358/463
• US Class Current: 382/275
• CPC Class Codes: G06T 5/77 Retouching; Inpainting; Scratch removal
• CPC Class Codes: G06V 30/10 Character recognition
• CPC Class Codes: G06V 30/155 Removing patterns interfering with the pattern to be recognised, such as ruled lines or underlines
• CPC Class Codes: H04N 23/811 by dust removal, e.g. from surfaces of the image sensor or processing of the image signal output by the electronic image sensor
• CPC Class Codes: H04N 25/61 the noise originating only from the lens unit, e.g. flare, shading, vignetting or "cos4"
• CPC Class Codes: H04N 25/683 by defect estimation performed on the scene signal, e.g. real time or on the fly detection

Abstract

A method of automatically correcting dust artifact within images acquired by a system including a digital camera includes determining probabilities of dust artifact regions corresponding to pixels within a digitally-acquired image based at least in part on a pixel analysis of the region in view of predetermined characteristics indicative of the presence of a dust artifact region. The dust artifact regions are associated with one or more extracted parameters relating to the optical system when the image was acquired. A statistical dust map is formed including mapped dust regions based on the dust artifact determining and associating operations. Digital data is corrected corresponding to correlated dust artifact regions within the acquired image based on the associated statistical dust map.

Claims

1. A method of automatically correcting dust artifact within images acquired by a system including a digital camera, comprising:

(a) determining probability of dust artifact regions corresponding to pixels within a digitally-acquired image based at least in part on a pixel analysis of the region in view of predetermined characteristics indicative of the presence of a dust artifact region;

(b) associating the dust artifact regions with one or more extracted parameters relating to the optical system when the image was acquired;

(c) forming a statistical dust map including mapped dust regions based on the dust artifact determining and associating operations; and

(d) correcting digital data corresponding to correlated dust artifact regions within said digitally-acquired image based on the associated statistical dust map, and

(e) wherein the dust artifact determining includes;

(I) loading the statistical dust map;

(II) loading extracted parameter information of a present image;

(III) performing a calculation for relating the statistical dust may with the present image according to a selected value of at least one extracted parameter; and

(IV) comparing dust artifact detection data with the statistical dust map data.

2. The method of claim 1, further comprising eliminating certain suspected dust artifact regions as having a probability below a first threshold value.

3. The method of claim 2, further comprising judging certain further dust artifact regions as having a probability above said threshold value, such as to be subject to further probability determining including comparison with further acquired images prior to judging whether each said further dust artifact region will be subject to said eliminating operation.

4. The method of claim 2, further comprising judging certain probable dust artifact regions as having a probability above a second threshold value such as to be subject to said correcting operation.

5. The method of claim 4, wherein said first and second threshold values are different.

6. The method of claim 5, further comprising judging certain further dust artifact regions as having a probability between said first and said second threshold values, such as to be subject to further probability determining including comparison with further acquired images prior to judging whether each said further dust artifact region will be subject to said correcting operation.

7. The method of claim 1, further comprising judging certain probable dust artifact regions as having a probability above a threshold value such as to be subject to said correcting operation.

8. The method of claim 7, further comprising judging certain further dust artifact regions as having a probability below said threshold value, such as to be subject to further probability determining including comparison with further acquired images prior to judging whether each said further dust artifact region will be subject to said correcting operation.

9. The method of claim 1, wherein said probability determining includes weighting suspected dust artifact regions according to one or more predetermined probability weighting assessment conditions.

10. The method of claim 9, said one or more weighting assessment conditions comprising size, shape, brightness or opacity of said suspected dust artifact regions, or degree of similarity size, shape, brightness, opacity or location with one or more suspected dust artifact regions in one or more other images, or combinations thereof.

11. The method of claim 1, said one or more extracted parameters comprising aperture size, F-number, magnification, lens type or focal length of an optical system of the digital camera, or combinations thereof.

12. The method of claim 11, said one or more extracted parameters are calculated empirically from comparison of one or more said dust artifact regions within said multiple original digital images with said digital acquisition device.

13. The method of claim 1, said one or more extracted parameters comprising aperture size or focal length or both.

14. The method of claim 1, wherein said further digitally-acquired images comprising different images than said originally acquired images.

15. The method of claim 14, said different images having been acquired of different objects.

16. The method of claim 14, wherein said different images are acquired of different scenes.

17. The method of claim 14, said different images having been acquired with different values of said one or more extracted parameters.

18. The method of claim 1, further comprising correcting pixels corresponding to correlated dust artifact regions within each of said original images based on the associated statistical dust map.

19. The method of claim 17, said one or more extracted parameters comprising aperture size, F-number, magnification, lens type or focal length of an optical system of the digital camera, or combinations thereof.

20. The method of claim 17, said one or more extracted parameters comprising aperture size or focal length or both.

21. The method of claim 1, further comprising correcting pixels corresponding to correlated dust artifact regions within said original images based on the associated statistical dust map.

22. The method of claim 1, further comprising digitally-acquiring additional images with said digital camera, repeating said determining and associating, and updating said statistical dust map including updating said mapped dust regions based on the additional dust artifact determining and associating.

23. The method of claim 21, further comprising correcting pixels corresponding to correlated dust artifact regions within said additional images based on the updated, associated statistical dust map.

24. The method of claim 21, further comprising updating one or more of said further and original images based on said updating of said associated statistical dust map.

25. The method of claim 21, further comprising limiting updating one or more of said further and original images based on said updating of said associated statistical dust map to updates that do not include appearance of new dust or movement of existing dust.

26. The method of claim 21, further comprising limiting updating one or more of said further and original images based on said updating of said associated statistical dust map to updates that include previously determined dust artifact regions.

27. The method of claim 21, further comprising creating a version description of changes in said statistical dust map.

28. The method of claim 21, wherein said version is based on a chronological time stamp.

29. The method of claim 21, wherein said version is based on replacement of lens.

30. The method of claim 21, wherein said version information comprises change of said probabilities in said statistical dust map that certain pixels correspond to dust artifact regions.

31. The method of claim 21, wherein said version information includes one or more parameters comprising of change in dust location, change in dust position, appearance of new dust region, disappearance of existing dust region.

32. The method of claim 21, wherein further comprising determining whether dust map needs to be replaced based on determining that sufficient disparity exists based amount and quality of said changes in said statistical dust map.

33. The method of claim 31, further comprising updating one or more of said further and original images based on said updating of said associated statistical dust map based on said version description.

34. The method of claim 1, said image correction method being automatically performed within a digital camera that comprises said optical system, said sensor array, said processing electronics and said memory.

35. The method of claim 1, said image correction method being performed at least in part within an external processing device that couples with a digital camera that comprises said optical system and said sensor array to form a digital image acquisition and processing system that also comprises said processing electronics and said memory.

36. The method of claim 34, the programming instructions being stored on a memory within the external device which performs the image correction method.

37. The method of claim 1, wherein said determining includes dynamically updating said probabilities based on comparisons with suspected equivalent dust artifact regions within said further digitally-acquired images.

38. The method of claim 1, wherein said forming includes dynamically updating said statistical dust map based on further pixel analyses of suspected dust artifact regions in further images in view of predetermined characteristics indicative of the presence of a dust artifact region.

39. The method of claim 1, wherein said suspected dust artifact regions of said at least two images comprise inner regions and aura regions, and wherein said comparison comprises a first comparison of said inner regions and a second comparison of said aura regions.

40. The method of claim 38, wherein said determining further comprises statistically combining probabilities based on said each of said comparisons of said inner regions and of said aura regions.

41. The method of claim 1, wherein said determining includes dynamically updating said probabilities based on comparisons with suspected equivalent dust artifact regions within said further digitally-acquired images.

42. The method of claim 1, wherein said determining probabilities further comprises statistically combining an plurality of individual probabilities based on each said regions within two or more said images.

43. The method of claim 1, further comprising determining probabilities that certain pixels correspond to regions free of dust within said images based at least in part on a comparison of suspected dust artifact regions within one or more of said images.

44. The method of claim 42, further comprising eliminating certain suspected dust artifact based on probabilities that certain pixels correspond to regions free of dust.

45. The method of claim 1, further comprising validating whether said further digitally-acquired image has non contradicting data that said probability that certain pixels correspond to dust artifact regions within said further digitally-acquired image priori to correcting pixels corresponding to correlated dust artifact regions within further digitally-acquired images based on the associated statistical dust map.

46. The method of claim 44, further comprising instructing the user to create a new said dust map if said validating determines a noticeable change in the dust map.

47. The method of claim 1, wherein said dust artifact regions of said digitally-acquired images are determined to comprise shadow regions and aura regions.

48. The method of claim 46, said focal length extracted parameters are calculated empirically from comparison of the transposition of said shadow regions of said dust artifact regions within multiple digital images acquired with said digital acquisition device.

49. The method of claim 46, said aperture extracted parameters are calculated empirically from comparison of a fall off of said aura regions of said dust artifact regions within multiple digital images acquired with said digital acquisition device.

50. The method of claim 46, wherein said correcting comprises a first correcting of said aura regions arid a second correcting of said shadow regions.

51. The method of claim 46, said determining with respect to a shadow region being based on an extracted parameter-dependent shadow region analysis, wherein the shadow region analysis presumes that certain regions on a sensor of the digital image acquisition device are fully obscured by said dust.

52. The method of claim 50, wherein the shadow region analysis includes calculating effects of differences in values of the one or more extracted parameters m different images on dust artifact illumination, shape, position, reflection or transmission properties, distance of dust to the sensor, aperture, exit pupil, or focal length, or combinations thereof.

53. The method of claim 51, wherein said different images are acquired with different values of said one or more extracted parameters.

54. The method of claim 51, wherein said different images are acquired of different objects.

55. The method of claim 51, wherein said different images are acquired of different scenes.

56. The method of claim 46, said determining with respect to an aura region being based on an extracted parameter-dependent aura region analysis, wherein the aura region analysis presumes that certain regions on a of the digital image acquisition device are partially obscured by said dust.

57. The method of claim 55, wherein said aura region analysis includes calculating effects of differences in values of the one or more extracted parameters in different images on dust artifact illumination, shape, position, reflection or transmission properties, distance of dust to the sensor, aperture, exit pupil, or focal length, or combinations thereof.

58. The method of claim 56, wherein said different images are acquired with different values of said one or more extracted parameters.

59. The method of claim 56, wherein said different images are acquired of different objects.

60. The method of claim 46, the correcting operation comprising in-painting or restoration, or both.

61. The method of claim 59, said correcting including in-painting the shadow region.

62. The method of claim 60, said in-painting including determining and applying shadow region correction spectral information based on spectral information obtained from pixels outside said shadow region.

63. The method of claim 59, said correcting including restoration of the aura region.

64. The method of claim 62, said restoration including determining and applying aura region correction spectral information based on spectral information obtained from pixels within said aura region.

65. The method of claim 1, the method being performed on raw image data as captured by a camera sensor.

66. The method of claim 1, said image correction method being performed on a processed image after being converted from raw format to a known red, green, blue representation.

67. The method of claim 1, wherein said correcting includes replacing said pixels within said one or more digitally-acquired images with new pixels.

68. The method of claim 1, wherein said correcting includes enhancing said values of pixels within said one or more digitally-acquired images.

69. The method of claim 1, wherein correcting instructions are kept in an external location to the image data.

70. The method of claim 1, said external location comprising an image header.

71. The method of claim 1, the dust artifact determining operation including:

(I) loading the statistical dust map;

(II) loading extracted parameter information of a present image;

(III) performing calculations within the statistical dust map having extracted parameter variable-dependencies; and

(IV) comparing dust artifact detection data with the extracted parameter dependent statistical dust map data.

72. The method of claim 70, the extracted parameter information including values of aperture size and focal length.

73. The method of claim 71, the extracted parameter information further including lens type information.

74. The method of claim 1, wherein for the performing, the at least one extracted parameter is otherwise uncorrelated between the present image and the dust map.

75. The method of claim 1, said dust artifact regions including an aura region partially obscured by dust and a shadow region at least substantially obscured by dust inside said aura region.

76. The method of claim 74, said determining with respect to a shadow region being based on an extracted parameter-dependent shadow region analysis, wherein the shadow region analysis presumes that certain regions on a sensor of the digital image acquisition device are fully obscured by said dust.

77. The method of claim 75, wherein the shadow region analysis includes calculating effects of differences in values of the one or more extracted parameters in different images on dust artifact illumination, shape, position, reflection or transmission properties, distance of dust to the sensor, aperture, exit pupil, or focal length, or combinations thereof.

78. The method of claim 76, said different images having been acquired of different objects.

79. The method of claim 76, wherein said different images are acquired of different scene.

80. The method of claim 74, said determining with respect to an aura region being based on an extracted parameter-dependent aura region analysis, wherein the aura region analysis presumes that certain regions on a of the digital image acquisition device are partially obscured by said dust.

81. The method of claim 79, wherein said aura region analysis includes calculating effects of differences in values of the one or more extracted parameters in different images on dust artifact illumination, shape, position, reflection or transmission properties, distance of dust to the sensor, aperture, exit pupil, or focal length, or combinations thereof.

82. The method of claim 80, said different images having been acquired of different objects.

83. The method of claim 74, the correcting operation comprising in-painting or restoration, or both.

84. The method of claim 82, said correcting including in-painting the shadow region.

85. The method of claim 83, said in-painting including determining and applying shadow region correction spectral information based on spectral information obtained from pixels outside said shadow region.

86. The method of claim 82, said correcting including restoration of the aura region.

87. The method of claim 85, said restoration including determining and applying aura region correction spectral information based on spectral information obtained from pixels within said aura region.

88. The method of claim 73, wherein for the comparing, said dust artifact detection data is now correlated with the statistical dust map data.