Efficient storage for production of high dynamic range images

• US 10,467,732 B2
• Filed: 11/30/2017
• Issued: 11/05/2019
• Est. Priority Date: 06/01/2015
• Status: Active Grant

• Current Assignee: Arm Ltd. (United Kingdom) (SoftBank Group Corp.)
• Original Assignee: Apical Limited (SoftBank Group Corp.)
• Inventors: Dewhurst, Andrew
• Primary Examiner(s): Werner, Brian
• Application Number:
  US15/827,120
• Publication Number:
  US 20180082408A1
• Time in Patent Office: 705 Days
• CPC Class Codes: G06T 2207/10144 Varying exposure
• CPC Class Codes: G06T 2207/20208 High dynamic range [HDR] image processing
• CPC Class Codes: G06T 5/40 using histogram techniques
• CPC Class Codes: G06T 5/50 using two or more images, e.g. averaging or subtraction
• CPC Class Codes: G06T 5/90 Dynamic range modification of images or parts thereof
• CPC Class Codes: H04N 23/741 by increasing the dynamic range of the image compared to the dynamic range of the electronic image sensors

Abstract

Methods, apparatus and computer programs are provided for storing a set of sensor output images for multi-exposure high dynamic range processing to produce a processed image. The multi-exposure high dynamic range processing combines pixel values from one or more of the sensor output images to produce the processed image. In one example, a method comprises receiving from an image sensor a first sensor output image. The first sensor output image has a first exposure. The method also comprises receiving from the image sensor a second sensor output image. The second sensor output image has a second exposure different than the first exposure. First image pixels of the first sensor output image having pixel values which are unlikely to have a substantial contribution to the processed image are identified, and the amount of information stored in relation to the first image pixels is reduced on the basis of the identifying step.

Claims

1. A method for storing a set of sensor output images for multi-exposure high dynamic range processing to produce a processed image, wherein the multi-exposure high dynamic range processing combines pixel values from one or more of the sensor output images to produce the processed image, the method comprising:

receiving, from an image sensor, a first sensor output image, wherein the first sensor output image has a first exposure;

receiving, from the image sensor, a second sensor output image, wherein the second sensor output image has a second exposure different than the first exposure;

identifying first image pixels of the first sensor output image having pixel values which are unlikely to have a substantial contribution to the processed image; and

reducing an amount of information stored in relation to the first image pixels on the basis of the identifying step.

2. The method according to claim 1, comprising:

identifying second image pixels of the second sensor output image having pixel values which are unlikely to have a substantial contribution to the processed image; and

reducing the amount of information stored in relation to the second image pixels on the basis of the identifying step.

3. The method according to claim 1, comprising processing the first and second images to produce a multi-exposure high dynamic range image, wherein the processing excludes pixels identified as having pixel values which are unlikely to have a substantial contribution to the processed image.

4. The method according to claim 1, wherein the second exposure is smaller than the first exposure and wherein the first image pixels are identified on the basis of having a pixel value which is more than or equal to a predetermined threshold.

5. The method according to claim 1, wherein the second exposure is greater than the first exposure and wherein the first image pixels are identified on the basis of having a pixel value which is less than or equal to a predetermined threshold.

6. The method according to claim 4, comprising:

receiving from the image sensor a third image, wherein the third image has a third exposure smaller than the first exposure and greater than the second exposure; and

identifying as excluded from the third image pixels of the third image having pixel value between two predetermined thresholds.

7. The method according to claim 4, wherein at least one of the thresholds is selected based on a noise profile of the image sensor.

8. The method according to claim 6, wherein:

at least one of the thresholds is based on a ratio of the first exposure to the third exposure;

or

at least one of the thresholds is based on a ratio of the third exposure to the second exposure;

or

at least one of the thresholds is based on a ratio of the first exposure to the second exposure.

9. The method according to claim 1, wherein reducing the amount of information stored in relation to the first image pixels comprises representing the first image pixels using a run-length encoding algorithm.

10. The method according to claim 4, wherein the predetermined threshold comprises a maximum pixel value of the first image.

11. The method according to claim 5, wherein the predetermined threshold comprises a minimum pixel value of the second image.

12. An apparatus for storing a set of sensor output images for multi-exposure high dynamic range processing to produce a processed image, wherein the multi-exposure high dynamic range processing combines pixel values from one or more of the sensor output images to produce the processed image, the apparatus comprising:

an image sensor;

a first encoder configured to;

receive from the image sensor a first sensor output image, wherein the first sensor output image has a first exposure; and

produce a modified version of the first sensor output image, wherein producing the modified version of the first image comprises;

identifying first image pixels of the first sensor output image having pixel values which are unlikely to have a substantial contribution to the processed image, and

reducing an amount of information stored in relation to the first image pixels on the basis of the identifying step;

a first buffer storage configured to receive and store the modified version of the first image;

a second encoder configured to;

receive from the image sensor a second sensor output image, wherein the second sensor output image has a second exposure different than the first exposure; and

produce a modified version of the second sensor output image, wherein producing the modified version of the second image comprises;

identifying second image pixels of the second sensor output image having pixel values which are unlikely to have a substantial contribution to the processed image; and

reducing an amount of information stored in relation to the second image pixels on the basis of the identifying step of the second image pixels, and

a second buffer storage configured to receive and store the modified version of the second image.

13. The apparatus according to claim 12, comprising a high dynamic range image constructor configured to:

receive the modified version of the first sensor output image from the first buffer;

receive the modified version of the second sensor output image from the second buffer; and

process the modified version of the first sensor output image and the modified version of the second sensor output image to produce the processed image, wherein the processing excludes pixels identified as having pixel values which are unlikely to have a substantial contribution to the processed image.

14. The apparatus according to claim 12, wherein the first buffer stores the modified version of the first sensor output image as a first compressed image and/or the second buffer stores the modified version of the second sensor output image as a second compressed image.

15. The apparatus according to claim 14, wherein storing the first and/or second compressed images comprises using a run-length encoding algorithm to indicate pixels excluded from the modified versions of the first and/or second images.

16. The apparatus according to claim 12, wherein the first and second buffers are configured to simultaneously transmit the modified versions of the first and second sensor output images to a high dynamic range image constructor.

17. A non-transitory, computer-readable storage medium comprising a set of computer-readable instructions stored thereon which, when executed by at least one processor, cause the at least one processor to:

receive, from an image sensor, a first sensor output image, wherein the first sensor output image has a first exposure;

identify, as excluded from the first sensor output image, pixels of the first sensor output image having pixel value equal to or greater than a first threshold, the first threshold corresponding to a first image intensity;

receive, from the image sensor, a second sensor output image, wherein the second sensor output image has a second exposure smaller than the first exposure;

identify, as excluded from the second sensor output image, pixels of the second sensor output image having pixel value less than or equal to a second threshold, the second threshold corresponding to a second image intensity lower than the first image intensity;

receive, from the image sensor, a third sensor output image, wherein the third sensor output image has a third exposure smaller than the first exposure and greater than the second exposure;

identify, as excluded from the third sensor output image, pixels of the third sensor output image having pixel value greater than or equal to a third threshold and less than or equal to a fourth threshold, wherein the third threshold corresponds to a third image intensity lower than the first image intensity and the fourth threshold corresponds to a fourth image intensity higher than the second image intensity; and

process the first, second and third sensor output images to produce a multi-exposure high dynamic range image, wherein the processing excludes pixels identified as excluded from the first sensor output image, excludes pixels identified as excluded from the second sensor output image and excludes pixels identified as excluded from the third sensor output image.