Method and apparatus for the removal of flash artifacts

• US 20020150306A1
• Filed: 04/11/2001
• Published: 10/17/2002
• Est. Priority Date: 04/11/2001
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Hewlett-Packard Development Company, L.P. (HP Inc.)
• Inventors: Baron, John M.
• Granted Patent: US 6,859,565 B2
• Time in Patent Office: Days
• US Class Current: 382/275
• CPC Class Codes: G06T 5/50 using two or more images, e.g. averaging or subtraction
• CPC Class Codes: H04N 23/74 by influencing the scene brightness using illuminating means

Abstract

An image without use of a flash is taken, along with an image using a flash. A difference image is generated by subtracting the flash-less image from the flash image. A threshold is applied to the difference image such that only large differences in intensity remain in the difference image. This artifact image is then subtracted from the flash image, thereby removing flash artifacts such as specular reflections and red-eye. The threshold used may be automatically calculated or may be set by the user. For some applications it may be desirable to set separate thresholds for each dimension of the color space (such as red, green, and blue) used. Once again these separate thresholds may be automatically calculated or may be set by the user.

Claims

1. A method for removing flash artifacts comprising the steps of:

a) capturing a first digital image of a subject;

b) capturing a second digital image of said subject with the use of a flash;

c) creating a difference image of said first and second digital images;

d) applying a threshold to said difference image to create an artifact image; and

e) subtracting said artifact image from said second digital image, resulting in a final digital image.

2. A method for removing flash artifacts as recited in claim 1;

wherein said artifact image is multiplied by a factor before the step of subtracting said artifact image from said second digital image.

3. A method for removing flash artifacts as recited in claim 1;

wherein said artifact image is offset by a factor before said step of subtracting said artifact image from said second digital image.

4. A method for removing flash artifacts as recited in claim 1;

wherein said difference image comprises intensity data.

5. A method for removing flash artifacts as recited in claim 1, further comprising the step of:

f) storing said final digital image in a memory device.

6. A method for removing flash artifacts as recited in claim 1;

wherein said threshold can be set by a user.

7. A method for removing flash artifacts as recited in claim 1;

wherein said threshold is calculated from a histogram of said difference image.

8. A method for removing flash artifacts comprising the steps of:

a) capturing a first digital image of a subject;

b) capturing a second digital image of said subject with the use of a flash;

c) creating at least three difference images of said first and second digital images, including a red difference image, a green difference image, and a blue difference image;

d) applying a red threshold, a green threshold, and a blue threshold to said red, green, and blue difference images to create at least three artifact images, including a red artifact image, a green artifact image, and a blue artifact image; and

e) subtracting said red, green, and blue artifact images from said second digital image, resulting in a final digital image.

9. A method for removing flash artifacts as recited in claim 8;

wherein said red, green, and blue artifact images are multiplied by a factor before the step of subtracting said red, green, and blue artifact images from said second digital image.

10. A method for removing flash artifacts as recited in claim 8;

wherein said red, green, and blue artifact images are offset by a factor before the step of subtracting said red, green, and blue artifact images from said second digital image.

11. A method for removing flash artifacts as recited in claim 8;

wherein said red, green, and blue thresholds are set independently and comprise chromatic data.

12. A method for removing flash artifacts as recited in claim 8;

wherein said red, green, and blue thresholds are based on pixel intensity.

13. A method for removing flash artifacts as recited in claim 8;

wherein said pixel intensity is calculated according to the National Television System Committee (NTSC) standard.

14. A method for removing flash artifacts as recited in claim 8, further comprising the step of:

f) storing said final digital image in a memory device.

15. A method for removing flash artifacts as recited in claim 8;

wherein said red threshold, green threshold, and blue threshold can be set by a user.

16. A method for removing flash artifacts as recited in claim 8;

wherein said red threshold is calculated from a red histogram of said red difference image;

said green threshold is calculated from a green histogram of said green difference image; and

said blue threshold is calculated from a blue histogram of said blue difference image.

23. A method for removing flash artifacts as recited in claim 17, further comprising the step of:

f) storing said final digital image in a memory device.

17. A method for removing flash artifacts comprising the steps of:

a) capturing a first digital image of a subject;

b) capturing a second digital image of said subject with the use of a flash;

c) creating at least three difference images of said first and second digital images, including a yellow difference image, a cyan difference image, and a magenta difference image;

d) applying a yellow threshold, a cyan threshold, and a magenta threshold to said yellow, cyan, and magenta difference images to create at least three artifact images, including a yellow artifact image, a cyan artifact image, and a magenta artifact image; and

e) subtracting said yellow, cyan, and magenta artifact images from said second digital image, resulting in a final digital image.

18. A method for removing flash artifacts as recited in claim 17;

wherein said yellow, cyan, and magenta artifact images are multiplied by a factor before the step of subtracting said yellow, cyan, and magenta artifact images from said second digital image.

19. A method for removing flash artifacts as recited in claim 17;

wherein said yellow, cyan, and magenta artifact images are offset by a factor before the step of subtracting said yellow, cyan, and magenta artifact images from said second digital image.

20. A method for removing flash artifacts as recited in claim 17;

wherein said yellow, cyan, and magenta thresholds are set independently and comprise chromatic data.

21. A method for removing flash artifacts as recited in claim 17;

wherein said yellow, cyan, and magenta thresholds are based on pixel intensity.

22. A method for removing flash artifacts as recited in claim 21;

wherein said pixel intensity is calculated according to the National Television System Committee (NTSC) standard.

24. A method for removing flash artifacts as recited in claim 17;

wherein said yellow threshold, cyan threshold, and magenta threshold can be set by a user.

25. A method for removing flash artifacts as recited in claim 17;

wherein said yellow threshold is calculated from a yellow histogram of said yellow difference image;

said cyan threshold is calculated from a cyan histogram of said cyan difference image; and

said magenta threshold is calculated from a magenta histogram of said magenta difference image.

26. A device for removing flash artifacts comprising:

a first subtract block with inputs comprising flash-less digital image data and flash digital image data, wherein said flash-less digital image data is subtracted from said flash digital image data producing difference data;

a threshold block connected to said first subtract block, that determines a threshold from said difference data;

a clipping block connected to said threshold block, that receives said difference data and said threshold and outputs artifact data, wherein said artifact data contains said difference data for said pixels with a value greater than said threshold; and

a second subtract block connected to said clipping block, wherein said second subtract block subtracts said artifact data from said flash digital image data, resulting in a final digital image.

27. A device for removing flash artifacts as recited in claim 26;

wherein said difference data comprises intensity data.

28. A device for removing flash artifacts as recited in claim 26;

wherein said threshold block calculates said threshold from a histogram.

29. A device for removing flash artifacts as recited in claim 26;

wherein said threshold block determines said threshold from a user input.

30. A device for removing flash artifacts comprising:

a first subtract block with inputs comprising flash-less digital image data and flash digital image data, wherein said flash-less digital image data is subtracted from said flash digital image data producing difference data;

a threshold block connected to said first subtract block, that determines a threshold from said difference data;

a clipping block connected to said threshold block, that receives said difference data and said threshold and outputs artifact data, wherein said artifact data contains said difference data for said pixels with a value greater than said threshold;

a factor block connected to said clipping block that modifies said artifact data by a factor and outputs factored artifact data; and

a second subtract block connected to said factor block, wherein said second subtract block subtracts said factored artifact data from said flash digital image data, resulting in a final digital image.

31. A device for removing flash artifacts as recited in claim 30;

wherein said factor block multiplies said artifact data by said factor.

32. A device for removing flash artifacts as recited in claim 30;

wherein said factor block adds said artifact data to said factor.

33. A device for removing flash artifacts as recited in claim 30;

wherein said difference data comprises intensity data.

34. A device for removing flash artifacts as recited in claim 30;

wherein said threshold block calculates said threshold from a histogram.

35. A device for removing flash artifacts as recited in claim 30;

wherein said threshold block determines said threshold from a user input.

36. A device for removing flash artifacts comprising:

a first subtract block with inputs comprising flash-less digital image data and flash digital image data, wherein said flash-less digital image data is subtracted from said flash digital image data producing difference data;

a threshold block connected to said first subtract block, that determines a threshold from said difference data;

a clipping block connected to said threshold block, that receives said difference data and said threshold and outputs artifact data, wherein said artifact data contains said difference data for said pixels with a value greater than or equal to said threshold; and

a second subtract block connected to said clipping block, wherein said second subtract block subtracts said artifact data from said flash digital image data, resulting in a final digital image.

37. A device for removing flash artifacts as recited in claim 36;

wherein said difference data comprises intensity data.

38. A device for removing flash artifacts as recited in claim 36;

wherein said threshold block calculates said threshold from a histogram.

39. A device for removing flash artifacts as recited in claim 36;

wherein said threshold block determines said threshold from a user input.

40. A device for removing flash artifacts comprising:

a first subtract block with inputs comprising flash-less digital image data and flash digital image data, wherein said flash-less digital image data is subtracted from said flash digital image data producing difference data;

a threshold block connected to said first subtract block, that determines a threshold from said difference data;

a clipping block connected to said threshold block, that receives said difference data and said threshold and outputs artifact data, wherein said artifact data contains said difference data for said pixels with a value greater than or equal to said threshold;

a factor block connected to said clipping block that modifies said artifact data by a factor and outputs factored artifact data; and

a second subtract block connected to said factor block, wherein said second subtract block subtracts said factored artifact data from said flash digital image data, resulting in a final digital image.

41. A device for removing flash artifacts as recited in claim 40;

wherein said factor block multiplies said artifact data by said factor.

42. A device for removing flash artifacts as recited in claim 40;

wherein said factor block adds said artifact data to said factor.

43. A device for removing flash artifacts as recited in claim 40;

wherein said difference data comprises intensity data.

44. A device for removing flash artifacts as recited in claim 40;

wherein said threshold block calculates said threshold from a histogram.

45. A device for removing flash artifacts as recited in claim 40;

wherein said threshold block determines said threshold from a user input.

46. A device for removing flash artifacts comprising:

at least three first primary color subtraction blocks that subtract at least three primary color flash-less digital image data from at least three primary color flash digital image data producing at least three primary color difference data, wherein a first primary color subtraction block subtracts a first primary color flash-less digital image data from a first primary color flash digital image data producing a first primary color difference data, and a second primary color subtraction block subtracts a second primary color flash-less digital image data from a second primary color flash digital image data producing a second primary color difference data, and a third primary color subtraction block subtracts a third primary color flash-less digital image data from a third primary color flash digital image data producing a third primary color difference data;

at least three primary color threshold blocks connected to said at least three primary color subtraction blocks, that determine at least three primary color thresholds from said at least three primary color difference data, wherein a first primary color threshold block connected to said first primary color subtraction block determines a first primary color threshold from said first primary color difference data, and a second primary color threshold block connected to said second primary color subtraction block determines a second primary color threshold from said second primary color difference data, and a third primary color threshold block connected to said third primary color subtraction block determines a third primary color threshold from said third primary color difference data;

at least three clipping blocks connected to said at least three threshold blocks, that receive said difference data and said threshold and output at least three primary color artifact data, wherein said primary color artifact data contains said difference data for said pixels with a primary color value greater than said primary color thresholds, also wherein a first clipping block is connected to said first threshold block and receives said first difference data and said first threshold and outputs a first primary color artifact data, and a second clipping block is connected to said second threshold block and receives said second difference data and said second threshold and outputs a second primary color artifact data, and a third clipping block is connected to said third threshold block and receives said third difference data and said third threshold and outputs a third primary color artifact data;

a merge block connected to said clipping blocks, that merges said at least three primary color artifact data into full-color artifact data; and

a second subtract block connected to said merge block, that subtracts said artifact data from said flash digital image data, resulting in a final digital image.

47. A device for removing flash artifacts as recited in claim 46;

wherein said primary colors are red, green, and blue.

48. A device for removing flash artifacts as recited in claim 46;

wherein said primary colors are yellow, cyan, and magenta.

49. A device for removing flash artifacts comprising:

at least three first primary color subtraction blocks that subtract at least three primary color flash-less digital image data from at least three primary color flash digital image data producing at least three primary color difference data, wherein a first primary color subtraction block subtracts a first primary color flash-less digital image data from a first primary color flash digital image data producing a first primary color difference data, and a second primary color subtraction block subtracts a second primary color flash-less digital image data from a second primary color flash digital image data producing a second primary color difference data, and a third primary color subtraction block subtracts a third primary color flash-less digital image data from a third primary color flash digital image data producing a third primary color difference data;

at least three primary color threshold blocks connected to said at least three primary color subtraction blocks, that determine at least three primary color thresholds from said at least three primary color difference data, wherein a first primary color threshold block connected to said first primary color subtraction block determines a first primary color threshold from said first primary color difference data, and a second primary color threshold block connected to said second primary color subtraction block determines a second primary color threshold from said second primary color difference data, and a third primary color threshold block connected to said third primary color subtraction block determines a third primary color threshold from said third primary color difference data;

at least three clipping blocks connected to said at least three threshold blocks, that receive said difference data and said threshold and output at least three primary color artifact data, wherein said primary color artifact data contains said difference data for said pixels with a primary color value greater than said primary color thresholds, also wherein a first clipping block is connected to said first threshold block and receives said first difference data and said first threshold and outputs a first primary color artifact data, and a second clipping block is connected to said second threshold block and receives said second difference data and said second threshold and outputs a second primary color artifact data, and a third clipping block is connected to said third threshold block and receives said third difference data and said third threshold and outputs a third primary color artifact data;

at least three second subtract blocks connected to said clipping block, that subtract said artifact data from said flash digital image data producing at least three primary color final image data; and

a merge block connected to said second subtract blocks, that merges said at least three primary color final image data into full-color final digital image data.

50. A device for removing flash artifacts as recited in claim 49;

wherein said primary colors are red, green, and blue.

51. A device for removing flash artifacts as recited in claim 49;

wherein said primary colors are yellow, cyan, and magenta.

52. A computer program storage medium readable by a computer, tangibly embodying a computer program of instructions executable by the computer to perform method steps for removing flash artifacts from digital image data, the steps comprising:

a) capturing a first digital image of a subject;

b) capturing a second digital image of said subject with the use of a flash;

c) creating a difference image of said first and second digital images;

d) applying a threshold to said difference image to create an artifact image; and

e) subtracting said artifact image from said second digital image, resulting in a final digital image.

53. A computer program storage medium as recited in claim 52;

wherein said difference image comprises intensity data.

54. A computer program storage medium as recited in claim 52, the steps further comprising:

f) storing said final digital image in a memory device.

55. A computer program storage medium as recited in claim 52;

wherein said threshold can be set by a user.

56. A computer program storage medium as recited in claim 52;

wherein said threshold is calculated from a histogram of said difference image.

58. A computer program storage medium as recited in claim 57, wherein said difference image comprises intensity data.

59. A computer program storage medium as recited in claim 57, further comprising the step of:

f) storing said final digital image in a memory device.

60. A computer program storage medium as recited in claim 57, wherein said red threshold, green threshold, and blue threshold can be set by a user.

61. A computer program storage medium as recited in claim 57, wherein said red threshold is calculated from a red histogram of said red difference image;

said green threshold is calculated from a green histogram of said green difference image; and

said blue threshold is calculated from a blue histogram of said blue difference image.

57. A computer program storage medium readable by a computer, tangibly embodying a computer program of instructions executable by the computer to perform method steps for removing flash artifacts from digital image data, the steps comprising:

a) capturing a first digital image of a subject;

b) capturing a second digital image of said subject with the use of a flash;

c) creating at least three difference images of said first and second digital images, including a red difference image, a green difference image, and a blue difference image;

d) applying a red threshold, a green threshold, and a blue threshold to said red, green and blue difference images to create at least three artifact images, including a red artifact image, a green artifact image, and a blue artifact image; and

e) subtracting said artifact image from said second digital image, resulting in a final digital image.

62. A computer program storage medium readable by a computer, tangibly embodying a computer program of instructions executable by the computer to perform method steps for removing flash artifacts from digital image data, the steps comprising:

a) capturing a first digital image of a subject;

b) capturing a second digital image of said subject with the use of a flash;

c) creating at least three difference images of said first and second digital images, including a yellow difference image, a cyan difference image, and a magenta difference image;

d) applying a yellow threshold, a cyan threshold, and a magenta threshold to said yellow difference image, cyan difference image, and magenta difference image to create at least three artifact images including a yellow artifact image, a cyan artifact image, and a magenta artifact image; and

e) subtracting said artifact images from said second digital image, resulting in a final digital image.

63. A computer program storage medium as recited in claim 62, further comprising the step of:

f) storing said final digital image in a memory device.

64. A computer program storage medium as recited in claim 62, wherein said yellow threshold, cyan threshold, and magenta threshold can be set by a user.

65. A computer program storage medium as recited in claim 62, wherein said yellow threshold is calculated from a yellow histogram of said yellow difference image;

said cyan threshold is calculated from a cyan histogram of said cyan difference image; and

said magenta threshold is calculated from a magenta histogram of said magenta difference image.

66. A method for removing flash artifacts comprising the steps of:

a) capturing a first digital image of a subject;

b) capturing a second digital image of said subject with the use of a flash;

c) selecting an intensity threshold; and

d) for each pixel to be processed performing the sub-steps of;

i) subtracting intensity of current pixel in said first digital image from intensity of current pixel in said second digital image resulting in a pixel difference; and

ii) replacing current pixel in said second digital image with current pixel from said first digital image when said pixel difference is greater than said intensity threshold.

67. A method for removing flash artifacts as recited in claim 66, wherein said intensity threshold is set by a user.

68. A method for removing flash artifacts as recited in claim 66, further comprising the step of:

e) after all of said pixels to be processed have been processed, saving said second digital image in a memory device as a final digital image.

70. A method for removing flash artifacts as recited in claim 69, wherein said threshold is set by a user.

71. A method for removing flash artifacts as recited in claim 69, further comprising the step of:

e) after all of said pixels to be processed have been processed, saving said second digital image in a memory device as a final digital image.

69. A method for removing flash artifacts comprising the steps of:

a) capturing a first digital image of a subject;

b) capturing a second digital image of said subject with the use of a flash;

c) selecting an intensity threshold; and

d) for each pixel to be processed performing the sub-steps of;

i) subtracting intensity of current pixel in said first digital image from intensity of current pixel in said second digital image resulting in a pixel difference; and

ii) replacing current pixel in said second digital image with current pixel from said first digital image when said pixel difference is greater than or equal to said intensity threshold.

72. A method for removing flash artifacts comprising the steps of:

a) capturing a first digital image of a subject;

b) capturing a second digital image of said subject with the use of a flash;

c) selecting at least three primary color thresholds; and

d) for each primary color of each pixel to be processed performing the sub-steps of;

i) subtracting primary color value of current pixel in said first digital image from primary color value of current pixel in said second digital image resulting in a primary color pixel difference for the current primary color of the current pixel; and

ii) replacing primary color value of current pixel in said second digital image with primary color value of current pixel from said first digital image when said primary color pixel difference for the current primary color of the current pixel is greater than said primary color threshold for said current primary color.

73. A method for removing flash artifacts as recited in claim 72;

wherein said primary colors include red, green, and blue.

74. A method for removing flash artifacts as recited in claim 72;

wherein said primary colors include yellow, cyan, and magenta.

75. A method for removing flash artifacts as recited in claim 72;

wherein said primary color thresholds are set by a user.

76. A method for removing flash artifacts as recited in claim 72, further comprising the step of:

e) after all of said pixels to be processed have been processed, saving said second digital image in a memory device as a final digital image.

77. A method for removing flash artifacts comprising the steps of:

a) capturing a first digital image of a subject;

b) capturing a second digital image of said subject with the use of a flash;

c) selecting at least three primary color thresholds; and

d) for each primary color of each pixel to be processed performing the sub-steps of;

i) subtracting primary color value of current pixel in said first digital image from primary color value of current pixel in said second digital image resulting in a primary color pixel difference for the current primary color of the current pixel; and

ii) replacing primary color value of current pixel in said second digital image with primary color value of current pixel from said first digital image when said primary color pixel difference for the current primary color of the current pixel is greater than or equal to said primary color threshold for said current primary color.

78. A method for removing flash artifacts as recited in claim 77;

wherein said primary colors include red, green, and blue.

79. A method for removing flash artifacts as recited in claim 77;

wherein said primary colors include yellow, cyan, and magenta.

80. A method for removing flash artifacts as recited in claim 77;

wherein said primary color thresholds are set by a user.

81. A method for removing flash artifacts as recited in claim 77, further comprising the step of:

e) after all of said pixels to be processed have been processed, saving said second digital image in a memory device as a final digital image.

82. A method for removing flash artifacts comprising the steps of:

a) capturing a first digital image of a subject;

b) capturing a second digital image of said subject with the use of a flash;

c) selecting at least three primary color thresholds; and

d) for each primary color of each pixel to be processed performing the sub-steps of;

i) subtracting primary color value of current pixel in said first digital image from primary color value of current pixel in said second digital image resulting in a primary color pixel difference for the current primary color of the current pixel; and

ii) replacing primary color value of current pixel in said second digital image with a color value calculated from said primary color value of current pixel in said first digital image, and said primary color value of current pixel in said second digital image, when said primary color pixel difference for the current primary color of the current pixel is greater than said primary color threshold for said current primary color.

83. A method for removing flash artifacts as recited in claim 82;

wherein said primary colors include red, green, and blue.

84. A method for removing flash artifacts as recited in claim 82;

wherein said primary colors include yellow, cyan, and magenta.

85. A method for removing flash artifacts as recited in claim 82;

wherein said primary color thresholds are set by a user.

86. A method for removing flash artifacts as recited in claim 82, further comprising the step of:

j) after all of said pixels to be processed have been processed, saving said second digital image in a memory device as a final digital image.

87. A method for removing flash artifacts comprising the steps of:

a) capturing a first digital image of a subject;

b) capturing a second digital image of said subject with the use of a flash;

c) selecting at least three primary color thresholds; and

d) for each primary color of each pixel to be processed performing the sub-steps of;

i) subtracting primary color value of current pixel in said first digital image from primary color value of current pixel in said second digital image resulting in a primary color pixel difference for the current primary color of the current pixel; and

ii) replacing primary color value of current pixel in said second digital image with a color value calculated from said primary color value of current pixel in said first digital image, and said primary color value of current pixel in said second digital image, when said primary color pixel difference for the current primary color of the current pixel is greater than or equal to said primary color threshold for said current primary color.

88. A method for removing flash artifacts as recited in claim 87;

wherein said primary colors include red, green, and blue.

89. A method for removing flash artifacts as recited in claim 87;

wherein said primary colors include yellow, cyan, and magenta.

90. A method for removing flash artifacts as recited in claim 87;

wherein said primary color thresholds are set by a user.

91. A method for removing flash artifacts as recited in claim 87, further comprising the step of:

e) after all of said pixels to be processed have been processed, saving said second digital image in a memory device as a final digital image.