Identifying intrinsic pixel colors in a region of uncertain pixels

• US 7,317,826 B2
• Filed: 09/05/2003
• Issued: 01/08/2008
• Est. Priority Date: 04/26/1999
• Status: Expired due to Fees

• Current Assignee: Adobe Inc.
• Original Assignee: Adobe Systems Incorporated (Adobe Inc.)
• Inventors: Wilensky, Gregg D., Newell, Martin E.
• Primary Examiner(s): SHERALI, ISHRAT I
• Application Number:
  US10/656,678
• Publication Number:
  US 20040042662A1
• Time in Patent Office: 1,586 Days
• Field of Search: 382162-167, 382/173, 382/284, 382/194, 382/141, 382/100, 382/199, 382/268, 382155-159, 345/156, 345/428, 345/420, 345/505, 345589-619, 345/519, 348576-592, 358/1.9, 358/500, 358/3.26, 358/527, 358/537, 358/540, 700/200, 715/744, 352/87, 353/28
• US Class Current: 382/162
• CPC Class Codes: G06T 11/001 Texturing; Colouring; Generation of texture or colour inpainting G06T5/77
• CPC Class Codes: G06T 2207/20101 Interactive definition of point of interest, landmark or seed
• CPC Class Codes: G06T 2207/20104 Interactive definition of region of interest [ROI]
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06T 7/143 involving probabilistic approaches, e.g. Markov random field [MRF] modelling
• CPC Class Codes: G06T 7/187 involving region growing; involving region merging; involving connected component labelling
• CPC Class Codes: G06T 7/194 involving foreground-background segmentation
• CPC Class Codes: G06V 10/267 by performing operations on regions, e.g. growing, shrinking or watersheds
• CPC Class Codes: G06V 10/28 Quantising the image, e.g. histogram thresholding for discrimination between background and foreground patterns
• CPC Class Codes: G06V 10/56 relating to colour

Abstract

A digital image that includes first and second regions is processed. An intrinsic color of a given pixel located in an area of interest that is adjacent to at least one of the first and second regions is estimated by extrapolating from colors of multiple pixels in one of the first and second regions and multiple pixels in the other of the two regions.

Claims

1. A machine-based method for use in processing a digital image that includes first and second regions, the method comprising:

estimating an intrinsic color of a given pixel located in an area of interest that is adjacent to at least one of the first and second regions, the estimating comprises extrapolating from colors of multiple pixels in one of the first and second regions and multiple pixels in the other of the two regions, and

storing the intrinsic color of the pixel for later use.

2. The method of claim 1 in which the given pixel has an original color that relates to the original colors of pixels in both the first and second regions, and the estimated intrinsic color of the given pixel relates to original colors in only one or the other of the first and second regions.

3. The method of claim 1 in which the area of interest includes one of the first and second regions.

4. The method of claim 1 in which the area of interest is adjacent to both of the first and second regions.

5. The method of claim 1 in which the first region comprises a foreground object and the second region comprises a background.

6. The method of claim 1 in which the first and second regions may have any arbitrary degree of color variation in the visible spectrum over a spatial scale that is on the same order of magnitude or smaller than the minimum span of the area of interest.

7. The method of claim 1 in which the estimating comprises analyzing both the color and spatial proximity of pixels in the first and second regions.

8. The method of claim 1 in which the estimating comprises extrapolating from the closest pixels in the first and second regions.

9. The method of claim 1 in which the estimating comprises flowing colors into the area of interest from one or both of the first and second regions.

10. The method of claim 9 in which the flowing of colors comprises averaging of color values for each of a set of pixels in the first region and a set of pixels in the second region.

11. The method of claim 1 in which the digital image comprises layers of pixel information and the estimating is based on pixel information in only one of the layers.

12. The method of claim 1 in which the digital image comprises layers of pixel information and the estimating is based on pixel information in a composition of all the layers.

13. The method of claim 1 further comprising determining an opacity value for the given pixel, indicative of the extent to which the intrinsic color of the given pixel relates to original colors in the first and second regions, based on a result of the estimating of the intrinsic color.

14. The method of claim 13 in which the given pixel includes original opacity information, and the opacity value is also based on the original opacity information.

15. The method of claim 13 further comprising determining opacity values for other pixels that have intrinsic colors that relate to original colors in the first and second regions.

16. The method of claim 13 in which the opacity determination comprises use of a neural network trained on the image original colors and estimated intrinsic colors.

17. The method of claim 13 further comprising using the opacity value to composite one of the first and second regions with another digital image.

18. The method of claim 1 in which the estimating also includes extrapolating estimates of intrinsic colors of the first and second regions using searches in color space and image coordinate space.

19. The method of claim 1 in which the estimating assumes a linear blending model.

20. The method of claim 1 in which the estimating includes flowing colors from edges of the area of interest to fill the area of interest with estimates of the colors of the first and second regions.

21. The method of claim 1 further comprising

extracting from the digital image the intrinsic colors of the given pixel and of other pixels that have intrinsic colors that relate to original colors in the first region or second region.

22. The method of claim 21 further comprising using the extracted intrinsic colors to composite the first region or the second region with another digital image.

23. The method of claim 1 further comprising

receiving from an interactive user interface information that identifies the area of interest.

24. The method of claim 1 in which estimating the intrinsic color comprises

determining two color sample sets for the given pixel, each of the color sample sets being associated with one of the first and second regions, and

estimating the intrinsic color based on the two color sample sets.

25. The method of claim 24 in which estimating the intrinsic color comprises comparing the original color of the given pixel with colors in the color sample sets.

26. The method of claim 24 further comprising

determining an opacity for the given pixel indicative of the extent to which the intrinsic color of the given pixel relates to original colors in both of the first and second regions, where the determination of opacity includes comparing the original color of the given pixel with colors in the color sample sets.

27. The method of claim 26 in which the given pixel includes original opacity information and the determination of opacity is also based on the original opacity information.

28. The method of claim 24 in which the color sample sets are derived from colors of pixels in the first and second regions.

29. The method of claim 24 in which a single color is selected from each of the color sample sets based on an error minimization technique.

30. The method of claim 1 in which the intrinsic colors of all of the pixels in the area of interest are determined automatically.

31. A medium bearing a computer program capable of controlling a computer to process a digital image that includes first and second regions by:

estimating an intrinsic color of a given pixel located in an area of interest that is adjacent to at least one of the first and second regions, the estimating comprises extrapolating from colors of multiple pixels in one of the first and second regions and multiple pixels in the other of the two regions, and

storing the intrinsic color of the pixel for later use.

33. The medium of claim 31 in which the given pixel has an original color that relates to the original colors of pixels in both the first and second regions, and the estimated intrinsic color of the given pixel relates to original colors in only one or the other of the first and second regions.

34. The medium of claim 31 in which the area of interest includes one of the first and second regions.

35. The medium of claim 31 in which the area of interest is adjacent to both of the first and second regions.

36. The medium of claim 31 in which the first region comprises a foreground object and the second region comprises a background.

37. The medium of claim 31 in which the first and second regions may have any arbitrary degree of color variation in the visible spectrum over a spatial scale that is on the same order of magnitude or smaller than the minimum span of the area of interest.

38. The medium of claim 31 in which the estimating comprises analyzing both the color and spatial proximity of pixels in the first and second regions.

39. The medium of claim 31 in which the estimating comprises extrapolating from the closest pixels in the first and second regions.

40. The medium of claim 31 in which the estimating comprises flowing colors into the area of interest from one or both of the first and second regions.

41. The medium of claim 40 in which the flowing of colors comprises averaging of color values for each of a set of pixels in the first region and a set of pixels in the second region.

42. The medium of claim 31 in which the digital image comprises layers of pixel information and the estimating is based on pixel information in only one of the layers.

43. The medium of claim 31 in which the digital image comprises layers of pixel information and the estimating is based on pixel information in a composition of all the layers.

44. The medium of claim 31 in which the computer program is capable of controlling the computer to process the digital image by determining an opacity value for the given pixel, indicative of the extent to which the intrinsic color of the given pixel relates to original colors in the first and second regions, based on a result of the estimating of the intrinsic color.

45. The medium of claim 44 in which the given pixel includes original opacity information, and the opacity value is also based on the original opacity information.

46. The medium of claim 44 in which the computer program is capable of controlling the computer to process the digital image by determining opacity values for other pixels that have intrinsic colors that relate to original colors in the first and second regions.

47. The medium of claim 44 in which the opacity determination comprises use of a neural network trained on the image original colors and estimated intrinsic colors.

48. The medium of claim 44 in which the computer program is capable of controlling the computer to process the digital image by using the opacity value to composite one of the first and second regions with another digital image.

49. The medium of claim 31 in which the estimating also includes extrapolating estimates of intrinsic colors of the first and second regions using searches in color space and image coordinate space.

50. The medium of claim 31 in which the estimating assumes a linear blending model.

51. The medium of claim 31 in which the estimating includes flowing colors from edges of the area of interest to fill the area of interest with estimates of the colors of the first and second regions.

52. The medium of claim 31 in which the computer program is capable of controlling the computer to process the digital image by extracting from the digital image the intrinsic colors of the given pixel and of other pixels that have intrinsic colors that relate to original colors in the first region or second region.

53. The medium of claim 52 in which the computer program is capable of controlling the computer to process the digital image by using the extracted intrinsic colors to composite the first region or the second region with another digital image.

54. The medium of claim 31 in which the computer program is capable of controlling the computer to process the digital image by receiving from an interactive user interface information that identifies the area of interest.

55. The medium of claim 31 in which estimating the intrinsic color comprises

determining two color sample sets for the given pixel, each of the color sample sets being associated with one of the first and second regions, and

estimating the intrinsic color based on the two color sample sets.

56. The medium of claim 55 in which estimating the intrinsic color comprises comparing the original color of the given pixel with colors in the color sample sets.

57. The medium of claim 55 in which the computer program is capable of controlling the computer to process the digital image by determining an opacity for the given pixel indicative of the extent to which the intrinsic color of the given pixel relates to original colors in both of the first and second regions, where the determination of opacity includes comparing the original color of the given pixel with colors in the color sample sets.

58. The medium of claim 57 in which the given pixel includes original opacity information and the determination of opacity is also based on the original opacity information.

59. The medium of claim 55 in which the color sample sets are derived from colors of pixels in the first and second regions.

60. The medium of claim 55 in which a single color is selected from each of the color sample sets based on an error minimization technique.

61. The medium of claim 31 in which the intrinsic colors of all of the pixels in the area of interest are determined automatically.

32. A system for use in processing a digital image that includes first and second regions, the system comprising:

means for estimating an intrinsic color of a given pixel located in an area of interest that is adjacent to at least one of the first and second regions, the estimating comprises extrapolating from colors of multiple pixels in one of the first and second regions and multiple pixels in the other of the two regions, and

means for storing the intrinsic color of the pixel for later use.

62. The system of claim 32 in which the given pixel has an original color that relates to the original colors of pixels in both the first and second regions, and the estimated intrinsic color of the given pixel relates to original colors in only one or the other of the first and second regions.

63. The system of claim 32 in which the area of interest includes one of the first and second regions.

64. The system of claim 32 in which the area of interest is adjacent to both of the first and second regions.

65. The system of claim 32 in which the first region comprises a foreground object and the second region comprises a background.

66. The system of claim 32 in which the first and second regions may have any arbitrary degree of color variation in the visible spectrum over a spatial scale that is on the same order of magnitude or smaller than the minimum span of the area of interest.

67. The system of claim 32 in which the estimating comprises analyzing both the color and spatial proximity of pixels in the first and second regions.

68. The system of claim 32 in which the estimating comprises extrapolating from the closest pixels in the first and second regions.

69. The system of claim 32 in which the estimating comprises flowing colors into the area of interest from one or both of the first and second regions.

70. The system of claim 69 in which the flowing of colors comprises averaging of color values for each of a set of pixels in the first region and a set of pixels in the second region.

71. The system of claim 32 in which the digital image comprises layers of pixel information and the estimating is based on pixel information in only one of the layers.

72. The system of claim 32 in which the digital image comprises layers of pixel information and the estimating is based on pixel information in a composition of all the layers.

73. The system of claim 32 comprising means for determining an opacity value for the given pixel, indicative of the extent to which the intrinsic color of the given pixel relates to original colors in the first and second regions, based on a result of the estimating of the intrinsic color.

74. The system of claim 73 in which the given pixel includes original opacity information, and the opacity value is also based on the original opacity information.

75. The system of claim 73 comprising means for determining opacity values for other pixels that have intrinsic colors that relate to original colors in the first and second regions.

76. The system of claim 73 in which the opacity determination comprises use of a neural network trained on the image original colors and estimated intrinsic colors.

77. The system of claim 73 comprising means for using the opacity value to composite one of the first and second regions with another digital image.

78. The system of claim 32 in which the estimating also includes extrapolating estimates of intrinsic colors of the first and second regions using searches in color space and image coordinate space.

79. The system of claim 32 in which the estimating assumes a linear blending model.

80. The system of claim 32 in which the estimating includes flowing colors from edges of the area of interest to fill the area of interest with estimates of the colors of the first and second regions.

81. The system of claim 32 comprising means for extracting from the digital image the intrinsic colors of the given pixel and of other pixels that have intrinsic colors that relate to original colors in the first region or second region.

82. The system of claim 81 comprising means for using the extracted intrinsic colors to composite the first region or the second region with another digital image.

83. The system of claim 32 comprising means for receiving from an interactive user interface information that identifies the area of interest.

84. The system of claim 32 in which estimating the intrinsic color comprises

determining two color sample sets for the given pixel, each of the color sample sets being associated with one of the first and second regions, and

estimating the intrinsic color based on the two color sample sets.

85. The system of claim 84 in which estimating the intrinsic color comprises comparing the original color of the given pixel with colors in the color sample sets.

86. The system of claim 84 comprising means for determining an opacity for the given pixel indicative of the extent to which the intrinsic color of the given pixel relates to original colors in both of the first and second regions, where the determination of opacity includes comparing the original color of the given pixel with colors in the color sample sets.

87. The system of claim 86 in which the given pixel includes original opacity information and the determination of opacity is also based on the original opacity information.

88. The system of claim 84 in which the color sample sets are derived from colors of pixels in the first and second regions.

89. The system of claim 84 in which a single color is selected from each of the color sample sets based on an error minimization technique.

90. The system of claim 32 in which the intrinsic colors of all of the pixels in the area of interest are determined automatically.