Detection of airborne flash artifacts using preflash image

• US 7,551,800 B2
• Filed: 08/09/2007
• Issued: 06/23/2009
• Est. Priority Date: 08/09/2006
• Status: Expired due to Fees

• Current Assignee: FotoNation Limited (Adeia Inc.)
• Original Assignee: FotoNation Vision Limited (Adeia Inc.)
• Inventors: Steinberg, Eran, Zamfir, Adrian, Drimbarean, Alexandru, Corcoran, Peter, Vertan, Constantin
• Primary Examiner(s): PATEL, KANJIBHAI B
• Application Number:
  US11/836,773
• Publication Number:
  US 20080069433A1
• Time in Patent Office: 684 Days
• Field of Search: 382/162, 382/167, 382/254, 382/264, 382/274, 382/275, 348/222.1, 348/223.1, 348/224.1, 348/371, 358/3.26, 358/3.27, 358/515, 358/518
• US Class Current: 382/275
• CPC Class Codes: G06T 2207/20032 Median filtering
• CPC Class Codes: G06T 5/40 using histogram techniques
• CPC Class Codes: G06T 5/70 Denoising; Smoothing
• CPC Class Codes: G06T 5/94 based on local image properties, e.g. for local contrast enhancement
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06T 7/136 involving thresholding
• CPC Class Codes: G06V 10/30 Noise filtering
• CPC Class Codes: G06V 10/48 by mapping characteristic values of the pattern into a parameter space, e.g. Hough transformation

Abstract

Flash image orb artifacts arising from specular reflections from airborne particles are corrected. An image of a scene using flash illumination and a reference image of substantially the same scene are acquired. The reference and flash images are subtracted to generate a difference image. Edge and interior region features of an orb artifact within the difference image are determined. The edge feature is matched with the interior region, and the orb artifact is corrected from the flash image based on the matching.

Claims

1. A method of using a processor to correct a flash image orb artifact arising from specular reflections from airborne particles, comprising:

(a) acquiring an image of a scene using flash illumination;

(b) acquiring a reference image of substantially the same scene;

(c) subtracting the reference and flash images to generate a difference image;

(d) determining an edge feature of an orb artifact within the difference image;

(e) determining a feature of an interior region of an orb artifact within the difference image;

(f) matching the edge feature with the feature of the interior region;

(g) correcting the orb artifact from the flash image based on the matching, thereby generating a corrected image; and

(h) displaying, transmitting, communicating or digitally storing or otherwise outputting the corrected image or a further processed version, or combinations thereof.

2. The method of claim 1, wherein the flash image and reference image are normalized prior to subtracting.

3. The method of claim 2, further comprising:

(i) upsampling the reference image to match the size of the flash image and aligning the reference and flash images; and

(ii) normalizing luminance across the reference and flash images, and

(iii) wherein the subtracting comprises subtracting the upscaled, aligned and luminance normalized reference and flash images.

4. The method of claim 1, further comprising scaling the difference image.

5. The method of claim 1, further comprising thresholding the difference image.

6. The method of claim 5, further comprising scaling the difference image after the thresholding.

7. The method of claim 6, further comprising edge detection prior to said thresholding.

8. The method of claim 1, further comprising color calibrating the reference and flash images based on a transformation of RGB color channels, including matching of colors selected from the reference and flash images within regions at substantial a same location within said scene.

9. The method of claim 8, further comprising:

(i) matching background regions of the reference and flash images including global luminance matching; and

(ii) matching foreground regions of the reference and flash images including local luminance matching.

10. The method of claim 9, wherein the matching of foreground regions comprises matching multiple foreground regions including separating the multiple foreground regions based on distance.

11. The method of claim 10, wherein the separating based on distance comprises depth from defocus mapping of the foreground regions.

12. The method of claim 9, wherein the matching of foreground regions comprises color luminance matching.

13. The method of claim 12, wherein the color luminance matching comprises analyzing skin color variation due to luminance variation between the reference and flash images.

14. The method of claim 1, wherein the determining of an interior region includes the creation of a map of difference variations within the interior region and the correcting of the orb artifact includes applying the map to determine the correction to be applied to one or more image pixels.

15. The method of claim 14, wherein the correcting further includes applying in-painting technique to correct pixels.

16. The method of claim 15, wherein difference variations are deemed to be too large to allow accurate correction of said pixels without applying in-painting.

17. The method of claim 1 wherein the correcting of the orb artifact employs in-painting techniques to correct pixels within the orb edge feature.

18. The method of claim 1, further comprising:

(i) anisotropic filtering of the difference image;

(ii) Canny edge detecting and edge cleaning;

(iii) computing a distance transform or Vornoi diagram of edges, or both; and

(iv) detecting edge features.

19. The method of claim 18, further comprising eliminating overlapping candidates.

20. The method of claim 19, wherein the eliminating of overlapping candidates comprises determining a survival factor of the specular reflection, and wherein the eliminating of overlapping candidates is based on best survival factor.

21. The method of claim 20, further comprising recording a final list of candidate reflections.

22. The method of claim 1, further comprising:

(i) matching background regions of the reference and flash images including glocal luminance matching; and

(ii) matching foreground regions of the reference and flash images including local luminance matching.

23. The method of claim 1, further comprising generating a luminance difference map based on luminance matching at least of the interior region.

24. A method of using a processor to correct a flash image orb artifact arising from specular reflections from airborne particles, comprising:

(a) acquiring an image of a scene using flash illumination;

(b) acquiring a reference image of substantially the same scene;

(c) color calibrating the reference and flash images based on a transformation of RGB color channels, including matching of colors selected from the reference and flash images within regions at substantially a same location within said scene;

(d) removing an orb artifact from the flash image based on the color calibrating, thereby generating a corrected image; and

(e) displaying, transmitting, communicating or digitally storing or otherwise outputting the corrected image or a further processed version, or combinations thereof.

25. The method of claim 24, further comprising:

(i) upscaling and aligning the reference and flash images; and

(ii) normalizing luminance across the reference and flash images, and

(iii) wherein the subtracting comprises subtracting the upscaled, aligned and luminance normalized reference and flash images.

26. The method of claim 24, further comprising scaling the difference image.

27. The method of claim 24, further comprising thresholding the difference image.

28. The method of claim 26, further comprising scaling the difference image after the thresholding.

29. The method of claim 28, further comprising edge detection prior to said thresholding.

30. The method of claim 24, further comprising:

(i) matching background regions of the reference and flash images including glocal luminance matching; and

(ii) matching foreground regions of the reference and flash images including local luminance matching.

31. The method of claim 30, wherein the matching of foreground regions comprises matching multiple foreground regions including separating the multiple foreground regions based on distance.

32. The method of claim 31, wherein the separating based on distance comprises depth from defocus mapping of the foreground regions.

33. The method of claim 30, wherein the matching of foreground regions comprises color luminance matching.

34. The method of claim 33, wherein the color luminance matching comprises analyzing skin color variation due to luminance variation between the reference and flash images.

35. The method of claim 24, further comprising generating a color difference map based on the matching of colors.

36. The method of claim 24, further comprising:

(i) determining an edge feature of the airborne artifact within the difference image;

(ii) determining an interior region of the airborne artifact within the difference image; and

(iii) matching the edge feature with the interior region.

37. The method of claim 24, wherein the determining of an interior region includes the creation of a map of difference variations within the interior region and the correcting of the orb artifact includes applying the map to determine the correction to be applied to one or more image pixels.

38. The method of claim 37, wherein the correcting further includes applying in-painting technique to correct pixels.

39. The method of claim 38, wherein difference variations are deemed to be too large to allow accurate correction of said pixels without applying in-painting.

40. The method of claim 24, wherein the correcting of the orb artifact employs in-painting techniques to correct pixels within the orb edge feature.

41. A method of using a processor to correct a flash image orb artifact arising from specular reflections from airborne particles, comprising:

(a) acquiring an image of a scene using flash illumination;

(b) acquiring a reference image of substantially the same scene;

(c) matching background regions of the reference and flash images including global luminance matching, or matching foreground regions of the reference and flash images including local luminance matching, or matching both the foreground regions and the background regions of the reference and flash images including respectively global and local luminance matching; and

(d) removing an orb artifact from the flash image based on the foreground or background matching, or based on both foreground matching and background matching, thereby generating a corrected image; and

(e) displaying, transmitting, communicating or digitally storing or otherwise outputting the corrected image or a further processed version, or combinations thereof.

42. The method of claim 41, wherein the matching of foreground regions comprises matching multiple foreground regions including separating the multiple foreground regions based on distance.

43. The method of claim 42, wherein the separating based on distance comprises depth from defocus mapping of the foreground regions.

44. The method of claim 41, wherein the matching of foreground regions comprises color luminance matching.

45. The method of claim 44, wherein the color luminance matching comprises analyzing skin color variation due to luminance variation between the reference and flash images.

46. The method of claim 41, further comprising generating a luminance difference map based on the global or local luminance matching, or both.

47. The method of claim 41, further comprising subtracting normalized versions of the reference and flash images to generate a difference image.

48. The method of claim 41, further comprising:

(i) determining an edge feature of the airborne artifact within the difference image;

(ii) determining an interior region of the airborne artifact within the difference image; and

(iii) matching the edge feature with the interior region.

49. The method of claim 41, wherein the matching comprises both matching background regions of the reference and flash images including global luminance matching, and matching foreground regions of the reference and flash images including local luminance matching.

50. The method of claim 49, wherein the removing comprises removing the orb artifact from the flash image based on both the foreground and background matching.

51. The method of claim 41, wherein the determining of an interior region includes the creation of a map of difference variations within the interior region and the correcting of the orb artifact includes applying the map to determine the correction to be applied to one or more image pixels.

52. The method of claim 51, wherein the correcting further includes applying in-painting technique to correct pixels.

53. The method of claim 52, wherein difference variations are deemed to be too large to allow accurate correction of said pixels without applying in-painting.

54. The method of claim 41, wherein the correcting of the orb artifact employs in-painting techniques to correct pixels within the orb edge feature.

55. One or more processor-readable media having program code embodied therein for programming one or more processors to perform a method of correcting a flash image orb artifact arising from specular reflections from airborne particles, the method comprising:

(a) acquiring an image of a scene using flash illumination;

(b) acquiring a reference image of substantially the same scene;

(c) subtracting the reference and flash images to generate a difference image;

(d) determining an edge feature of an orb artifact within the difference image;

(e) determining a feature of an interior region of an orb artifact within the difference image;

(f) matching the edge feature with the feature of the interior region; and

(g) correcting the orb artifact from the flash image based on the matching.

56. The one or more media of claim 55, wherein the image and reference image are normalized prior to subtracting.

57. The one or more media of claim 55, the method further comprising scaling the difference image.

58. The one or more media of claim 55, the method further comprising thresholding the difference image.

59. The one or more media of claim 55, the method further comprising color calibrating the reference and flash images based on a transformation of RGB color channels, including matching of colors selected from the reference and flash images within regions at substantial a same location within said scene.

60. The one or more media of claim 55, wherein the determining of an interior region includes the creation of a map of difference variations within the interior region and the correcting of the orb artifact includes applying the map to determine the correction to be applied to one or more image pixels.

61. The one or more media of claim 55, wherein the correcting of the orb artifact employs in-painting techniques to correct pixels within the orb edge feature.

62. The one or more media of claim 55, the method further comprising:

(i) anisotropic filtering of the difference image;

(ii) Canny edge detecting and edge cleaning;

(iii) computing a distance transform or Vornoi diagram of edges, or both; and

(iv) detecting edge features.

63. The one or more media of claim 55, the method further comprising:

(i) matching background regions of the reference and flash images including glocal luminance matching; and

(ii) matching foreground regions of the reference and flash images including local luminance matching.

64. The one or more media of claim 55, the method further comprising generating a luminance difference map based on luminance matching at least of the interior region.

65. The one or more media of claim 55, wherein the correcting further includes applying in-painting technique to correct pixels.

66. The one or more media of claim 65, wherein the correcting of the orb artifact employs in-painting techniques to correct pixels within the orb edge feature.

67. One or more processor-readable media having program code embodied therein for programming one or more processors to perform a method of correcting a flash image orb artifact arising from specular reflections from airborne particles, the method comprising:

(a) acquiring an image of a scene using flash illumination;

(b) acquiring a reference image of substantially the same scene;

(c) color calibrating the reference and flash images based on a transformation of RGB color channels, including matching of colors selected from the reference and flash images within regions at substantially a same location within said scene; and

(d) removing an orb artifact from the flash image based on the color calibrating.

68. The one or more media of claim 67, the method further comprising:

(i) upscaling and aligning the reference and flash images; and

(ii) normalizing luminance across the reference and flash images, and

(iii) wherein the subtracting comprises subtracting the upscaled, aligned and luminance normalized reference and flash images.

69. The one or more media of claim 67, the method further comprising scaling the difference image.

70. The one or more media of claim 67, the method further comprising thresholding the difference image.

71. The one or more media of claim 67, the method further comprising generating a color difference map based on the matching of colors.

72. The one or more media of claim 67, the method further comprising:

(i) determining an edge feature of the airborne artifact within the difference image;

(ii) determining an interior region of the airborne artifact within the difference image; and

(iii) matching the edge feature with the interior region.

73. The one or more media of claim 67, wherein the correcting further includes applying in-painting technique to correct pixels.

74. The one or more media of claim 73, wherein the correcting of the orb artifact employs in-painting techniques to correct pixels within the orb edge feature.

75. One or more processor-readable media having program code embodied therein for programming one or more processors to perform a method of correcting a flash image orb artifact arising from specular reflections from airborne particles, the method comprising:

(a) acquiring an image of a scene using flash illumination;

(b) acquiring a reference image of substantially the same scene;

(c) matching background regions of the reference and flash images including global luminance matching, or matching foreground regions of the reference and flash images including local luminance matching, or matching both the foreground regions and the background regions of the reference and flash images including respectively global and local luminance matching; and

(d) removing an orb artifact from the flash image based on the foreground or background matching, or based on both foreground matching and background matching.

76. The one or more media of claim 75, wherein the matching of foreground regions comprises matching multiple foreground regions including separating the multiple foreground regions based on distance.

77. The one or more media of claim 75, wherein the matching of foreground regions comprises color luminance matching.

78. The one or more media of claim 75, the method further comprising generating a luminance difference map based on the global or local luminance matching, or both.

79. The one or more media of claim 75, the method further comprising subtracting normalized versions of the reference and flash images to generate a difference image.

80. The one or more media of claim 75, the method further comprising:

(i) determining an edge feature of the airborne artifact within the difference image;

(ii) determining an interior region of the airborne artifact within the difference image; and

(iii) matching the edge feature with the interior region.

81. The one or more media of claim 75, wherein the matching comprises both matching background regions of the reference and flash images including global luminance matching, and matching foreground regions of the reference and flash images including local luminance matching.

82. The one or more media of claim 75, wherein the correcting further includes applying in-painting technique to correct pixels.

83. The one or more media of claim 75, wherein the correcting of the orb artifact employs in-painting techniques to correct pixels within the orb edge feature.