Systems and methods for illumination correction of an image

• US 9,008,457 B2
• Filed: 05/31/2011
• Issued: 04/14/2015
• Est. Priority Date: 05/31/2010
• Status: Active Grant

• Current Assignee: Personify Incorporated (Wilson Human Capital Group, Inc.)
• Original Assignee: Pesonify Incorporated
• Inventors: Dikmen, Mert, Patel, Sanjay J., Lin, Dennis, Nguyen, Quang H., Do, Minh N.
• Primary Examiner(s): Patel, Jayesh A
• Application Number:
  US13/149,715
• Publication Number:
  US 20110293179A1
• Time in Patent Office: 1,414 Days
• Field of Search: None
• US Class Current: 382/274
• CPC Class Codes: G06T 5/94 based on local image properties, e.g. for local contrast enhancement

Abstract

An RGB color image and an infrared intensity image of a live video are received. The RGB color image is converted to a colorspace image comprising a channel corresponding to a brightness value. Each pixel of the converted colorspace image is evaluated to determine whether the brightness channel of the pixel exceeds a threshold value. If the brightness channel of the pixel exceeds the threshold value, the infrared intensity value of a corresponding pixel from the infrared intensity image is mixed into the pixel'"'"'s channel value that corresponds to brightness. The converted colorspace image is converted back to an RGB color image.

Claims

1. A method for illumination correction of an image, the method comprising:

receiving, using a computer, a color image, a depth image and an infrared intensity image;

converting the color image to a colorspace image comprising a brightness channel corresponding to a brightness value for each pixel;

based at least in part on the depth image, identifying pixels in the colorspace image that correspond to a user;

comparing, for each pixel of the colorspace image corresponding to the user, the brightness value of the pixel to a brightness threshold value; and

creating a hybrid intensity image of the user by using a fusion formula based on the brightness value of a pixel of the colorspace image corresponding to the user and an infrared intensity value of a corresponding pixel of the infrared intensity image by applying the fusion formula to each pixel with a brightness value greater than the brightness threshold value, and by using the brightness channel for each pixel with a brightness value lower than the brightness threshold value.

2. The method as set forth in claim 1, wherein the use of the fusion formula comprises modifying the brightness channel value of a pixel of the colorspace image based on the infrared intensity value of the corresponding pixel of the infrared intensity image.

3. The method as set forth in claim 1, wherein the color image comprises an RGB color image, the colorspace image comprises a CIE-Lab colorspace image, the channel corresponding to the brightness value comprises the luminance (L) channel of the CIE-Lab colorspace image.

4. The method as set forth in claim 1, wherein the color image comprises an RGB color image, the colorspace image comprises a Hue-Saturation-Value (HSV) colorspace image, the channel corresponding to the brightness value comprises the value (V) channel of the HSV colorspace image.

5. The method as set forth in claim 1, further comprising transforming the infrared intensity image to match a viewpoint of the color image based on a displacement between an infrared sensor generating the infrared intensity image and a color sensor generating the color image.

6. The method as set forth in claim 1, further comprising applying a brightness adjustment and a contrast adjustment of a camera generating the color image, the adjustments are performed until a mean sample value of the brightness value of the colorspace image approximates a target mean sample value.

7. The method as set forth in claim 1, further comprising shining an infrared light onto a user, the infrared intensity image comprises the infrared light that is shined onto the user and returned to an infrared sensor.

8. A system, comprising at least one processor and memory, for illumination correction of an image, the system comprising:

a module for receiving a color image, a depth image and an infrared intensity image;

a module for converting the color image to a colorspace image comprising a brightness channel corresponding to a brightness value for each pixel;

based at least in part on the depth image, identifying pixels in the colorspace image that correspond to a user;

a module for comparing, for each pixel of the colorspace image corresponding to the user, the brightness value of the pixel to a brightness threshold value; and

a module for creating a hybrid intensity image of the user by using a fusion formula based on the brightness value of a pixel of the colorspace image corresponding to the user and an infrared intensity value of a corresponding pixel of the infrared intensity image by applying the fusion formula to each pixel with a brightness value greater than the brightness threshold value, and by using the brightness channel for each pixel with a brightness value lower than the brightness threshold value.

9. The system as set forth in claim 8, wherein the use of the fusion formula comprises modifying the brightness channel value of a pixel of the colorspace image based on the infrared intensity value of the corresponding pixel of the infrared intensity image.

10. The system as set forth in claim 8, wherein the color image comprises an RGB color image, the colorspace image comprises a CIE-Lab colorspace image, the channel corresponding to the brightness value comprises the luminance (L) channel of the CIE-Lab colorspace image.

11. The system as set forth in claim 8, wherein the color image comprises an RGB color image, the colorspace image comprises a Hue-Saturation-Value (HSV) colorspace image, the channel corresponding to the brightness value comprises the value (V) channel of the HSV colorspace image.

12. The system as set forth in claim 8, further comprising a module for transforming the infrared intensity image to match a viewpoint of the color image based on a displacement between an infrared sensor generating the infrared intensity image and a color sensor generating the color image.

13. The system as set forth in claim 8, further comprising a module for applying a brightness adjustment and a contrast adjustment of a camera generating the color image, the adjustments are performed until a mean sample value of the brightness value of the colorspace image approximates a target mean sample value.

14. The system as set forth in claim 8, further comprising a module for shining an infrared light onto a user, the infrared intensity image comprises the infrared light that is shined onto the user and returned to an infrared sensor.

15. A non-transitory computer readable medium carrying one or more instructions for illumination correction of an image, wherein the one or more instructions, when executed by one or more processors, causes the one or more processors to perform the steps of:

receiving a color image, a depth image, and an infrared intensity image;

converting the color image to a colorspace image comprising a brightness channel corresponding to a brightness value for each pixel;

based at least in part on the depth image, identifying pixels in the colorspace image that correspond to a user;

comparing, for each pixel of the colorspace image corresponding to the user, the brightness value of the pixel to a brightness threshold value; and

creating a hybrid intensity image of the user by modifying the brightness channel using a fusion formula based on the brightness value of a pixel of the colorspace image corresponding to the user and an infrared intensity value of a corresponding pixel of the infrared intensity image by applying the fusion formula to each pixel with a brightness value greater than the brightness threshold value, and by using the brightness channel for each pixel with a brightness value lower than the brightness threshold value.

16. The non-transitory computer readable medium as set forth in claim 15, wherein the use of the fusion formula comprises modifying the brightness channel value of a pixel of the colorspace image based on the infrared intensity value of the corresponding pixel of the infrared intensity image.

17. The non-transitory computer readable medium as set forth in claim 15, wherein the color image comprises an RGB color image, the colorspace image comprises a CIE-Lab colorspace image, the channel corresponding to the brightness value comprises the luminance (L) channel of the CIE-Lab colorspace image.

18. The non-transitory computer readable medium as set forth in claim 15, wherein the color image comprises an RGB color image, the colorspace image comprises a Hue-Saturation-Value (HSV) colorspace image, the channel corresponding to the brightness value comprises the value (V) channel of the HSV colorspace image.

19. The non-transitory computer readable medium as set forth in claim 15, wherein the steps further comprise transforming the infrared intensity image to match a viewpoint of the color image based on a displacement between an infrared sensor generating the infrared intensity image and a color sensor generating the color image.

20. The non-transitory computer readable medium as set forth in claim 15, wherein the steps further comprise applying a brightness adjustment and a contrast adjustment of a camera generating the color image, the adjustments are performed until a mean sample value of the brightness value of the colorspace image approximates a target mean sample value.