Methods and systems for combining foreground video and background video using chromatic matching

• US 9,883,155 B2
• Filed: 06/14/2016
• Issued: 01/30/2018
• Est. Priority Date: 06/14/2016
• Status: Active Grant

• Current Assignee: Personify Incorporated (Wilson Human Capital Group, Inc.)
• Original Assignee: Personify Incorporated (Wilson Human Capital Group, Inc.)
• Inventors: Patel, Sanjay, Yarkony, Elad
• Primary Examiner(s): Tran, Trang U
• Application Number:
  US15/181,653
• Publication Number:
  US 20170359559A1
• Time in Patent Office: 595 Days
• Field of Search: 348584, 348586, 348587, 348589, 348582
• CPC Class Codes: H04N 5/275 Generation of keying signals
• CPC Class Codes: H04N 9/73 Colour balance circuits, e.g. white balance circuits or colour temperature control camera processing pipelines for colour balance H04N23/88
• CPC Class Codes: H04N 9/76 for mixing of colour signals H04N9/75 takes precedence

Abstract

Disclosed herein are methods and systems for combining foreground video and background video using chromatic matching. In an embodiment, a system obtains foreground video data. The system obtains background video data. The system determines a color-distribution dimensionality of the background video data to be either high-dimensional chromatic or low-dimensional chromatic. The system selects a chromatic-adjustment technique from a set of chromatic-adjustment techniques based on the determined color-distribution dimensionality of the background video data. The system adjusts the foreground video data using the selected chromatic-adjustment technique. The system generates combined video data at least in part by combining the background video data with the adjusted foreground video data. The system outputs the combined video for display.

Claims

1. A method comprising:

obtaining foreground video data;

obtaining background video data;

determining a color-distribution dimensionality of the background video data to be either high-dimensional chromatic or low-dimensional chromatic;

selecting a chromatic-adjustment technique from a set of chromatic-adjustment techniques based on the determined color-distribution dimensionality of the background video data;

adjusting the foreground video data using the selected chromatic-adjustment technique;

generating combined video data at least in part by combining the background video data with the adjusted foreground video data; and

outputting the combined video data for display.

2. The method of claim 1, wherein determining the color-distribution dimensionality of the background video data to be either high-dimensional chromatic or low-dimensional chromatic comprises:

converting pixels of the background video data from an {R,G,B} color space to an {L,a,b} color space;

calculating an {a,b} variance of the converted background pixels;

comparing the calculated {a,b} variance to an {a,b}-variance threshold;

determining the color-distribution dimensionality of the background video data to be high-dimensional chromatic if the calculated {a,b} variance exceeds the {a,b}-variance threshold; and

determining the color-distribution dimensionality of the background video data to be low-dimensional chromatic if the calculated {a,b} variance does not exceed the {a,b}-variance threshold.

3. The method of claim 2, wherein:

calculating the {a,b} variance of the converted background pixels comprises determining how many luminance levels in the converted background pixels have more than a luminance-level-specific degree of {a,b} variance; and

the {a,b}-variance threshold is a threshold number of luminance levels.

4. The method of claim 3, wherein the threshold number of luminance levels is zero.

5. The method of claim 3, wherein the threshold number of luminance levels is greater than zero.

6. The method of claim 2, wherein calculating the {a,b} variance of the converted background pixels comprises:

determining a respective luminance-level-specific {a,b} variance for each of a plurality of luminance levels that are represented in the converted background pixels; and

calculating the {a,b} variance of the converted background pixels to be a sum of the determined luminance-level-specific {a,b} variances.

7. The method of claim 2, wherein calculating the {a,b} variance of the converted background pixels comprises:

determining a respective luminance-level-specific {a,b} variance for each luminance level represented in the converted background pixels; and

calculating the {a,b} variance of the converted background pixels to be a sum of the determined luminance-level-specific {a,b} variances.

8. The method of claim 1, wherein determining the color-distribution dimensionality of the background video data to be either high-dimensional chromatic or low-dimensional chromatic comprises:

determining the color-distribution dimensionality of the background video data to be low-dimensional chromatic if a background-color distribution of the background video data in an {L,a,b} color space is supported by a relationship defined by;

{(L,a,b)|a=f_a(L),b=f_b(L)},

where f_a and f_b are functions; and

otherwise determining the color-distribution dimensionality of the background video data to be high-dimensional chromatic.

9. The method of claim 1, wherein:

the set of chromatic-adjustment techniques includes a white-balancing technique and a chromatic-replacement technique; and

selecting a chromatic-adjustment technique based on the determined color-distribution dimensionality comprises;

selecting the white-balancing technique when the color-distribution dimensionality of the background video data is determined to be high-dimensional chromatic; and

selecting the chromatic-replacement technique when the color-distribution dimensionality of the background video data is determined to be low-dimensional chromatic.

10. The method of claim 9, wherein adjusting the foreground video data using the white-balancing technique comprises:

determining a foreground average of pixels of the foreground video data in an {R,G,B} color space;

determining a background average of pixels of the background video data in the {R,G,B} color space;

converting the foreground average and the background average from the {R,G,B} color space to a second color space;

determining a transform matrix in the second color space from the converted foreground average to the converted background average;

converting the pixels of the foreground video data from the {R,G,B} color space to the second color space;

transforming the converted foreground pixels in the second color space using the determined transform matrix; and

converting the transformed foreground pixels from the second color space to the {R,G,B} color space.

11. The method of claim 10, wherein the determined transform matrix comprises dimension-wise ratios in the second color space of the converted background average to the converted foreground average.

12. The method of claim 10, wherein the second color space is an {L,a,b} color space.

13. The method of claim 10, wherein the second color space is an {X,Y,Z} color space.

14. The method of claim 13, wherein converting the foreground pixels from the {R,G,B} color space to the second color space comprises converting the foreground pixels from the {R,G,B} color space to an {L,a,b} color space and then from the {L,a,b} color space to the {X,Y,Z} color space.

15. The method of claim 9, further comprising converting pixels of the background video data from an {R,G,B} color space to an {L,a,b} color space, wherein adjusting the foreground video data using the chromatic-replacement technique comprises:

generating an L-to-{a,b} lookup table based on the converted background pixels;

converting pixels of the foreground video data from the {R,G,B} color space to the {L,a,b} color space;

transforming the converted foreground pixels at least in part by;

using the respective L values of the respective converted foreground pixels to select respective replacement {a,b} values for the respective converted foreground pixels based on the L-to-{a,b} lookup table; and

replacing the respective {a,b} values of the respective converted foreground pixels with the corresponding respective selected replacement {a,b} values; and

converting the transformed foreground pixels from the {L,a,b} color space to the {R,G,B} color space.

16. The method of claim 15, wherein using the respective L values of the respective converted foreground pixels to select the respective replacement {a,b} values for the respective converted foreground pixels based on the L-to-{a,b} lookup table comprises:

retrieving the respective replacement {a,b} values from the L-to-{a,b} lookup table in cases where the respective L value of the respective converted foreground pixel is listed in the L-to-{a,b} lookup table.

17. The method of claim 16, wherein using the respective L values of the respective converted foreground pixels to select the respective replacement {a,b} values for the respective converted foreground pixels based on the L-to-{a,b} lookup table further comprises:

using interpolated {a,b} values based on one or more entries in the L-to-{a,b} lookup table as the respective replacement {a,b} values in cases where the respective L value of the respective converted foreground pixel is not listed in the L-to-{a,b} lookup table.

18. The method of claim 17, wherein the interpolated {a,b} values are copied from a nearest L value that is listed in the L-to-{a,b} lookup table.

19. The method of claim 17, wherein the interpolated {a,b} values are average {a,b} values of two or more proximate entries in the L-to-{a,b} lookup table.

20. The method of claim 1, further comprising:

obtaining second foreground video data; and

adjusting the second foreground video data using the selected chromatic-adjustment technique,

wherein generating the combined video data comprises combining the background video data with both the adjusted foreground video data and the adjusted second foreground video data.

21. A system comprising:

a communication interface;

a processor; and

a non-transitory computer-readable medium storing instructions executable by the processor for causing the system to perform functions including;

obtaining foreground video data;

obtaining background video data;

determining a color-distribution dimensionality of the background video data to be either high-dimensional chromatic or low-dimensional chromatic;

selecting a chromatic-adjustment technique from a set of chromatic-adjustment techniques based on the determined color-distribution dimensionality of the background video data;

adjusting the foreground video data using the selected chromatic-adjustment technique;

generating combined video data at least in part by combining the background video data with the adjusted foreground video data; and

outputting the combined video data for display.