Scene exposure auto-compensation for differential image comparisons

• US 9,390,487 B2
• Filed: 10/20/2014
• Issued: 07/12/2016
• Est. Priority Date: 10/20/2014
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Hoof, Jonathan Ross, Walker, Benjamin William, Cheng, Beibei
• Primary Examiner(s): Liu, Li
• Application Number:
  US14/518,915
• Publication Number:
  US 20160110856A1
• Time in Patent Office: 631 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G06T 11/00 2D [Two Dimensional] image generation
• CPC Class Codes: G06T 5/50 using two or more images, e.g. averaging or subtraction
• CPC Class Codes: G06T 5/94 based on local image properties, e.g. for local contrast enhancement
• CPC Class Codes: H04N 13/133 Equalising the characteristics of different image components, e.g. their average brightness or colour balance
• CPC Class Codes: H04N 13/25 using two or more image sensors with different characteristics other than in their location or field of view, e.g. having different resolutions or colour pickup characteristics; using image signals from one sensor to control the characteristics of another sensor
• CPC Class Codes: H04N 23/76 by influencing the image signals

Abstract

Changes in image exposure setting may be compensated by creating a table of aggregate differences between corresponding pixels in two images and applying the table of aggregate differences to a portion of one of the two images. The images may be in any color space for visible light, or images not of visible light, such as infrared or depth images. In various configurations, the differences may be aggregated by averaging the differences for pixels of a certain value in one of the two images.

Claims

1. An apparatus comprising:

a processor; and

memory coupled to the processor, the memory comprising executable instructions that, when executed by the processor, cause the processor to effectuate operations comprising;

generating a masked reference image by applying a mask to a reference image, the masked reference image comprising a plurality of pixels that represent a part of a scene, wherein the mask indicates which pixels of the reference image are likely to break an assumption of a compensation estimation process;

generating an estimate of compensation for compensating a particular pixel value of pixels contained in the masked reference image by aggregating differences between the particular pixel value and values of pixels in a second image that correspond to pixels in the masked reference image with the particular pixel value, wherein a pixel in the reference image corresponds to a pixel in the second image when pixels represent the same part of the scene; and

generating a compensated image portion of the reference image by adding the estimate to the value of pixels that are both in a compensation portion of the reference image and also have pixel values equal to the particular pixel value.

2. The apparatus of claim 1, the operations further comprising:

generating the mask by logically ORing a binary mask for the reference image with a binary mask for the second image.

3. The apparatus of claim 1, the operations further comprising:

determining a heartrate of an object represented in the reference image and the second image based at least in part on the compensated image portion of the reference image.

4. The apparatus of claim 3, wherein:

the determined heartrate is further based on a portion of the second image corresponding to the compensation portion.

5. The apparatus of claim 1 wherein the mask comprises at least one of a portion of the images that are moving or the compensation portion.

6. The apparatus of claim 1, wherein the differences are aggregated by averaging the differences.

7. The apparatus of claim 1, wherein the pixel values represent at least one of:

red, green, blue, luminance, red-difference chroma, blue-difference chroma, infrared, or a portion of the light spectrum.

8. A method comprising:

determining a difference between a first value of a feature of a pixel of a first representation of a scene and a second value of the feature of a corresponding pixel of a second representation of the scene;

determining, for the first value, an aggregate based on the determined difference and at least one other value;

excluding a pixel from the aggregate if the pixel is contained in a mask; and

adjusting the feature of a pixel in an adjustment portion of at least one of the first representation or the second representation based on the aggregate value.

9. The method of claim 8, further comprising:

constructing the mask by logically ORing a binary mask for the first representation with a binary mask for the second representation.

10. The method of claim 8, further comprising:

constructing the mask by including in the mask at least one of;

a portion of the representations that are moving, and the adjustment portion.

11. The method of claim 8, further comprising:

detecting a heartrate of an object in the first representation and the second representation based at least in part on the compensated image portion of the reference image.

12. The method of claim 8, further comprising:

determining the aggregate by averaging a set of determined differences, wherein the set includes pixels from a reference frame having the same first value of the feature, and wherein the reference frame is either the first representation or the second representation.

13. A system comprising:

a processor; and

a memory coupled to the processor, the memory comprising executable instructions that, when executed by the processor, cause the processor to effectuate operations comprising;

determining a difference between a first value of a feature of a pixel of a first representation of a scene and a second value of the feature of a corresponding pixel of a second representation of the scene;

for the first value, determining an aggregate based on the determined difference and at least one other value;

excluding a pixel from the aggregate if the pixel is contained in a mask; and

adjusting the feature of an adjustment portion of at least one of the first representation or the second representation based on the aggregate value.

14. The system of claim 13, the operations further comprising:

constructing the mask by logically ORing a binary mask for the first representation with a binary mask for the second representation.

15. The system of claim 13, the operations further comprising:

constructing the mask by including in the mask at least one of;

a portion of the representations that are moving, and the adjustment portion.

16. The system of claim 13, the operations further comprising:

detecting a heartrate of an object in the first representation and the second representation based at least in part on the compensated image portion of the reference image.

17. The system of claim 13, the operations further comprising:

determining the aggregate by averaging a set of determined differences, wherein the set includes only pixels from a reference frame having the same first value of the feature, and wherein the reference frame is either the first representation or the second representation.

18. The system of claim 13, wherein the feature of a pixel is a numeric value for at least one of:

red, green, blue, luminance, red-difference chroma, blue-difference chroma, infrared, or other part of the light spectrum.