Estimating sensor sensitivity

• US 8,452,116 B1
• Filed: 11/19/2008
• Issued: 05/28/2013
• Est. Priority Date: 09/19/2008
• Status: Active Grant

• Current Assignee: Adobe Inc.
• Original Assignee: Adobe Systems Incorporated (Adobe Inc.)
• Inventors: Chuang, Jason C., Winnemoeller, Holger
• Primary Examiner(s): Allison, Andrae S
• Application Number:
  US12/274,233
• Publication Number:
  US 20130128056A1
• Time in Patent Office: 1,651 Days
• Field of Search: 382162-167, 382/190, 382/254, 382/260, 348/272, 348180-191, 348/273, 348/229.1, 348/230.1, 348/250.8, 250/208.1, 250/226, 358/504
• US Class Current: 382/260
• CPC Class Codes: G06F 18/24 Classification techniques
• CPC Class Codes: G06V 10/147 Details of sensors, e.g. sensor lenses fingerprint or palmprint sensors G06V40/13; vascular sensors G06V40/145; eye sensors G06V40/19
• CPC Class Codes: G06V 10/30 Noise filtering
• CPC Class Codes: G06V 10/764 using classification, e.g. of video objects
• CPC Class Codes: H04N 17/002 for television cameras
• CPC Class Codes: H04N 23/71 Circuitry for evaluating the brightness variation

Abstract

A method, system, and computer-readable storage medium for determining an estimate of sensor sensitivity associated with an image. A noise level of an image is determined, then an estimate of sensor sensitivity associated with the image is automatically determined, e.g., by a trained classifier based on the determined noise level. Additionally, the sensor sensitivity estimate can be used to determine scene brightness.

Claims

1. A computer-implemented method, comprising:

de-noising an image to generate a de-noised image from the image;

determining a noise signature of the image, wherein said determining the noise signature is based on differences between the image and the de-noised image; and

automatically determining an estimate of sensor sensitivity associated with the image based on the determined noise signature, wherein the sensor sensitivity is of a sensor used to capture the image.

2. The computer-implemented method of claim 1, wherein said determining a noise signature of the image comprises:

de-noising the image, thereby generating a de-noised image;

decomposing each of the image and the de-noised image into respective and corresponding pluralities of patches;

determining at least one respective patch descriptor for each patch of the image based on the patch of the image and the corresponding patch of the de-noised image, wherein the patch descriptor characterizes noise in the patch of the image by summarizing differences between the patch of the image and the corresponding patch of the de-noised image; and

creating the noise signature for the image based on the patch descriptors.

3. The computer-implemented method of claim 2, wherein said decomposing each of the image and the de-noised image into respective and corresponding pluralities of patches comprises partitioning each image multiple times with respective different patch sizes.

4. The computer-implemented method of claim 2, wherein said determining at least one respective patch descriptor for each patch of the image comprises determining a patch descriptor for each patch per color channel.

5. The computer-implemented method of claim 2, wherein the respective patch descriptor further includes a texture descriptor that characterizes texture of the patch.

6. The computer-implemented method of claim 5, wherein said creating the noise signature for the image based on the patch descriptors comprises:

discarding descriptors of patches with high frequency texture, thereby leaving a subset of the patches with little or no high frequency texture; and

creating the noise signature based on the subset of the patches.

7. The computer-implemented method of claim 2, wherein said creating the noise signature for the image based on the patch descriptors comprises:

sorting the patch descriptors into bins based on intensity, color channel, and/or texture;

sorting the patch descriptors within each bin based on amount of pixel alteration;

selecting the patch descriptors with the lowest pixel alteration within each bin, wherein the amount of pixel alteration indicates the amount of noise in the corresponding patch; and

creating the noise signature based on a weighted average pixel alteration of the selected patch descriptors.

8. The computer-implemented method of claim 2, wherein said automatically determining an estimate of sensor sensitivity associated with the image based on the determined noise signature comprises:

providing the noise signature for the image to a trained classifier as input; and

the trained classifier generating the estimate of sensor sensitivity based on the noise signature.

9. The computer-implemented method of claim 8, further comprising training the classifier prior to said trained classifier generating the estimate of sensor sensitivity, wherein said training the classifier comprises:

determining noise signatures for each of a plurality of photographs for which sensor sensitivity is known; and

training the classifier using the determined noise signatures and corresponding sensor sensitivity values to the classifier as training input.

10. The computer-implemented method of claim 8, wherein the classifier comprises a plurality of classifiers, each directed to a respective camera model or make.

11. The computer-implemented method of claim 1, further comprising:

automatically determining an estimate of scene brightness based on the estimate of sensor sensitivity and metadata of the image, wherein the metadata comprises aperture information, exposure time information, and intensity information for the image; and

storing the estimate of scene brightness, wherein the estimate of scene brightness is useable to categorize the image.

12. The computer-implemented method of claim 11, further comprising:

categorizing the image based on the estimate of scene brightness, thereby determining a category for the image, wherein the category for the image is useable to perform semantic based image operations.

13. The computer-implemented method of claim 12, further comprising:

determining one or more keywords or tags for the image based on the determined category, wherein the one or more keywords or tags are useable to perform search, retrieval, and matching operations with respect to the image.

14. A non-transitory computer-readable storage medium that stores program instructions computer-executable to implement:

de-noising an image to generate a de-noised image from the image;

determining a noise signature of the image, wherein said determining the noise signature is based on differences between the image and the de-noised image; and

automatically determining an estimate of sensor sensitivity associated with the image based on the determined noise signature, wherein the sensor sensitivity is of a sensor used to capture the image.

15. The non-transitory computer-readable storage medium of claim 14, wherein said determining a noise signature of the image comprises:

de-noising the image, thereby generating a de-noised image;

decomposing each of the image and the de-noised image into respective and corresponding pluralities of patches;

determining at least one respective patch descriptor for each patch of the image based on the patch of the image and the corresponding patch of the de-noised image, wherein the patch descriptor characterizes noise in the patch of the image by summarizing differences between the patch of the image and the corresponding patch of the de-noised image; and

creating the noise signature for the image based on the patch descriptors.

16. The non-transitory computer-readable storage medium of claim 15, wherein said decomposing each of the image and the de-noised image into respective and corresponding pluralities of patches comprises partitioning each image multiple times with respective different patch sizes.

17. The non-transitory computer-readable storage medium of claim 15, wherein said determining at least one respective patch descriptor for each patch of the image comprises determining a patch descriptor for each patch per color channel.

18. The non-transitory computer-readable storage medium of claim 15, wherein the respective patch descriptor further includes a texture descriptor that characterizes texture of the patch.

19. The non-transitory computer-readable storage medium of claim 18, wherein said creating the noise signature for the image based on the patch descriptors comprises:

discarding descriptors of patches with high frequency texture, thereby leaving a subset of the patches with little or no texture; and

creating the noise signature based on the subset of the patches.

20. The non-transitory computer-readable storage medium of claim 15, wherein said creating the noise signature for the image based on the patch descriptors comprises:

sorting the patch descriptors into bins based on intensity, color channel, and/or texture;

sorting the patch descriptors within each bin based on amount of pixel alteration;

selecting the patch descriptors with the lowest pixel alteration within each bin, wherein the amount of pixel alteration indicates the amount of noise in the corresponding patch; and

creating the noise signature based on a weighted average pixel alteration of the selected patch descriptors.

21. The non-transitory computer-readable storage medium of claim 15, wherein said automatically determining an estimate of sensor sensitivity associated with the image based on the determined noise signature comprises:

providing the noise signature for the image to a trained classifier as input; and

the trained classifier generating the estimate of sensor sensitivity based on the noise signature.

22. The non-transitory computer-readable storage medium of claim 21, wherein the program instructions are further computer-executable to train the classifier prior to said trained classifier generating the estimate of sensor sensitivity, wherein said training the classifier comprises:

determining noise signatures for each of a plurality of photographs for which sensor sensitivity is known; and

training the classifier using the determined noise signatures and corresponding sensor sensitivity values to the classifier as training input.

23. The non-transitory computer-readable storage medium of claim 21, wherein the classifier comprises a plurality of classifiers, each directed to a respective camera model or make.

24. The non-transitory computer-readable storage medium of claim 14, wherein the program instructions are further computer-executable to implement:

automatically determining an estimate of scene brightness based on the estimate of sensor sensitivity and metadata of the image, wherein the metadata comprises aperture information, exposure time information, and intensity information for the image; and

storing the estimate of scene brightness, wherein the estimate of scene brightness is useable to categorize the image.

25. The non-transitory computer-readable storage medium of claim 24, wherein the program instructions are further computer-executable to implement:

categorizing the image based on the estimate of scene brightness, thereby determining a category for the image, wherein the category for the image is useable to perform semantic based image operations.

26. The non-transitory computer-readable storage medium of claim 25, wherein the program instructions are further computer-executable to implement:

determining one or more keywords or tags for the image based on the determined category, wherein the one or more keywords or tags are useable to perform search, retrieval, and matching operations with respect to the image.

27. A system, comprising:

at least one processor; and

a memory coupled to the at least one processor, wherein the memory stores program instructions, wherein the program instructions are executable by the at least one processor to;

de-noise an image to generate a de-noised image from the image;

determine a noise signature of the image, wherein said determining the noise signature is based on differences between the image and the de-noised image; and

automatically determine an estimate of sensor sensitivity associated with the image based on the determined noise signature, wherein the sensor sensitivity is of a sensor used to capture the image.

28. The system of claim 27, wherein to determining a noise signature of the image, the program instructions are computer-executable to:

de-noise the image, thereby generating a de-noised image;

decompose each of the image and the de-noised image into respective and corresponding pluralities of patches;

determine at least one respective patch descriptor for each patch of the image based on the patch of the image and the corresponding patch of the de-noised image, wherein the patch descriptor characterizes noise in the patch of the image by summarizing differences between the patch of the image and the corresponding patch of the de-noised image; and

create the noise signature for the image based on the patch descriptors.

29. The system of claim 28, wherein said decomposing each of the image and the de-noised image into respective and corresponding pluralities of patches comprises partitioning each image multiple times with respective different patch sizes.

30. The system of claim 28, wherein to determine at least one respective patch descriptor for each patch of the image, the program instructions are computer-executable to determine a patch descriptor for each patch per color channel.

31. The system of claim 28, wherein the respective patch descriptor further includes a texture descriptor that characterizes texture of the patch.

32. The system of claim 31, wherein to create the noise signature for the image based on the patch descriptors, the program instructions are computer-executable to:

discard descriptors of patches with high frequency texture, thereby leaving a subset of the patches with little or no texture; and

create the noise signature based on the subset of the patches.

33. The system of claim 28, wherein to create the noise signature for the image based on the patch descriptors, the program instructions are computer-executable to:

sort the patch descriptors into bins based on intensity, color channel, and/or texture;

sort the patch descriptors within each bin based on amount of pixel alteration;

select the patch descriptors with the lowest pixel alteration within each bin, wherein the amount of pixel alteration indicates the amount of noise in the corresponding patch; and

create the noise signature based on a weighted average pixel alteration of the selected patch descriptors.

34. The system of claim 28, wherein to automatically determine an estimate of sensor sensitivity associated with the image based on the determined noise signature, the program instructions are computer-executable to:

provide the noise signature for the image to a trained classifier as input, wherein the trained classifier generates the estimate of sensor sensitivity based on the noise signature.

35. The system of claim 34, wherein the program instructions are further executable to train the classifier prior to said trained classifier generating the estimate of sensor sensitivity, wherein to train the classifier the program instructions are computer-executable to:

determine noise signatures for each of a plurality of photographs for which sensor sensitivity is known; and

train the classifier using the determined noise signatures and corresponding sensor sensitivity values to the classifier as training input.

36. The system of claim 34, wherein the classifier comprises a plurality of classifiers, each directed to a respective camera model or make.

37. The system of claim 27, wherein the program instructions are further executable to:

automatically determine an estimate of scene brightness based on the estimate of sensor sensitivity and metadata of the image, wherein the metadata comprises aperture information, exposure time information, and intensity information for the image; and

store the estimate of scene brightness, wherein the estimate of scene brightness is useable to categorize the image.

38. The computer-readable storage medium of claim 37, wherein the program instructions are further executable to:

categorize the image based on the estimate of scene brightness, thereby determining a category for the image, wherein the category for the image is useable to perform semantic based image operations.

39. The computer-readable storage medium of claim 38, wherein the program instructions are further executable to:

determine one or more keywords or tags for the image based on the determined category, wherein the one or more keywords or tags are useable to perform search, retrieval, and matching operations with respect to the image.

40. A computer-implemented method, comprising:

executing instructions on a computing platform, the instructions for de-noising an image so that a de-noised image is generated from the image;

executing instructions on a computing platform so that binary digital electronic signals representing a noise signature of the image are determined, wherein said determined noise signature is based on differences between the image and the de-noised image;

executing instructions on the computing platform so that binary digital electronic signals representing an estimate of sensor sensitivity associated with the image are determined based on the noise signature, wherein the sensor sensitivity is of a sensor used to capture the image; and

storing an indication of the determined estimate of sensor sensitivity.