Method, system, and device for automatic determination of nominal backing color and a range thereof

• US 7,508,455 B2
• Filed: 03/07/2005
• Issued: 03/24/2009
• Est. Priority Date: 03/26/2004
• Status: Active Grant

• Current Assignee: Roos Video's Live Production Technology Co.
• Original Assignee: Ross Video Live Production Technology
• Inventors: Fryer, Alun John, Ross, David Allan, Liu, Yu
• Primary Examiner(s): Kostak; Victor R
• Application Number:
  US11/072,502
• Publication Number:
  US 20050212820A1
• Time in Patent Office: 1,478 Days
• Field of Search: 348/587, 348/592, 348/722, 348/586, 345/473, 345/581, 345/620, 352/352, 382/195, 382/203, 382/204
• US Class Current: 348/587
• CPC Class Codes: H04N 9/75 Chroma key

Abstract

A method, system, and device for generating a clean clipping signal α for a chromakey function or a video composition, including identifying background colors formed by a solid color background, shadows cast by still and moving subjects, a non-uniform reflection caused by spot lighting and non-flat backdrop or flaw wall, and translucent foreground objects, with a 3D volume in a 3D color space; determining parameters defining same by using a dirty alpha α; generating a clean clipping signal α_shd background colors, and a clean clipping signal α_tsl for translucency colors; identifying foreground colors formed by the still and moving subjects with a 3D volume in a 3D color space; classifying colors into transition colors; and generating an area map for mapping each pixel into background, shadow, translucent, foreground, and transition areas.

Claims

1. A method for generating a clean clipping signal α

for a chromakey function or a video composition where still and moving subjects shot against a solid color background by a video camera are clipped out and overlaid on a new complex background, comprising the steps of;

identifying background colors which are formed by the solid color background, shadows cast by the still and moving subjects, non-uniform reflection caused by spot lighting and non-flat backdrop or flaw wall, and translucent foreground objects, with a 3D volume in a 3D color space;

automatically determining parameters which define the 3D volume by using a dirty alpha α

;

generating a clean clipping signal α

_shd for the background colors, which occurs to the colors identified as the shadow colors or confined by the 3D volume for the shadow colors, and a clean clipping signal α

_tsl for the translucency colors, which occur to the colors identified as the translucent areas'"'"' colors;

identifying foreground colors which are formed by the still and moving subjects with a 3D volume in a 3D color space;

classifying into transition colors those which are not identified as either the background or the foreground; and

generating an area map which labels each pixel as one of the areas such as background, shadow, translucent area, foreground and transition areas.

2. The method of claim 1, wherein identifying the background colors comprises identifying the background colors using a bounding box or a bounding rectangle which confines the 3D volume for the background colors to a box in a 3D color space.

3. The method of claim 1, wherein identifying the background colors comprises identifying a nominal backing color and a shadow axis of the 3D volume for the background colors in a 3D color space, the shadow axis being a semi line starting from the origin and passing through the nominal backing color, which confines the 3D volume for the shadow colors within a wedged box in a 3D color space.

4. The method of claim 1, wherein identifying the shadow colors comprises identifying the shadow colors using a two-semi-ellipse cylindrical segment which is confined by the wedged box in a 3D color space.

5. The method of claim 1, wherein the step of generating a clean clipping signal α

_shd further includes the projection of a pixel color onto the shadow axis, which is represented by;

6. The method of claim 5, wherein generating the clean clipping signal α

_shd further comprises generating a square root √

{square root over (α

_shd)} of α

_shd.

7. The method of claim 1, wherein identifying the background colors comprises identifying one or multiple translucency axis of the 3D volume for the background colors in a 3D color space, the translucency axis being a semi line starting from a lighting color reflected from translucent objects in foreground and passing through the nominal backing color, and the lighting color being a different color, including mostly a white color.

8. The method of claim 1, wherein the step of generating a clean clipping signal α

_tsl further includes the projection of a pixel color onto a translucency axis, which is represented by;

9. The method of claim 8, wherein generating the clean clipping signal α

_tsl further comprises generating a square root √

{square root over (α

_tsl)} of α

_tsl.

10. The method of claim 1, wherein identifying foreground colors comprises identifying foreground colors using, as the 3D volume for foreground colors, a 3D volume which may overlap the 3D volume for the background colors.

11. The method of claim 1, wherein the step of automatically determining parameters comprises the steps of:

initializing the 3D volume for the background colors;

generating a low-cost dirty clipping signal or the dirty alpha α

;

using the dirty alp ha or a value derived from the dirty alpha to construct a part of a weight;

generating a norm distance Δ

or a modified norm distance δ

from a pixel color of interest to the shadow axis or from a pixel color along the shadow axis to the nominal backing color;

using the norm distance Δ

or δ

to construct another part of a weight;

generating weighed color components by multiplying each color component with the weights;

accumulating each weighed color component pixel by pixel and averaging the accumulated weighed color component;

deriving parameters from the average of accumulated weighed color component and using them to determine the nominal backing color and a bounding box which defines two semi-ellipses;

repeatedly implementing the initializing to accumulating steps N times (N≧

3) to achieve the best results during initialization; and

recursively implementing the initializing to accumulating steps every M (M≧

1) frames to update the parameters on the fly.

12. The method of claim 11, wherein the step of initializing the 3D volume for the background colors comprises generating a chromatic plane divided into 6 equal sectors such that each sector center is exclusively passed through by one of 3 principal colors (red, green, blue) and 3 complementary colors (yellow, magenta, cyan), and the initial 3D volume for the background colors stands on one of the 6 sectors.

13. The method of claim 11, wherein the step of generating a dirty alpha comprises either using a standalone function or using intermediate results from a function which generates a clean alpha.

14. The method of claim 11, wherein the step of constructing a weight w comprises either directly multiplying a low-cost dirty alpha with a norm distance or generating an arithmetic combination of a low-cost dirty alpha and a norm distance, including a function ƒ

in terms of Δ and

α

such that;

w=ƒ

(α

, Δ

).

15. The method of claim 11, wherein generating weighted color components comprises multiplying, with the weights, each color component in one of the following:

luminance and color difference (L, Cr, Cb), luminance and modified color difference vector (L, U, V), (L, I, Q), (L, Pr, Pb), (L, H, S), luminance and a color difference derived from CIE XYZ color system, and principal color R, G, B.

16. The method of claim 11, wherein the step of accumulating each weighed color component comprises accumulating of an entire frame of a picture or an entire field of a picture or most parts of a picture.

17. The method of claim 11, wherein the step of deriving from the averages of accumulated weighed color components further comprises generating a series of functions which receive the averages of accumulated weighed color components as inputs and generate parameters as outputs for the wedged bounding box, the two semi-ellipse cylindrical segment, the nominal backing color.

18. The method of claim 1, wherein the step of automatically determining parameters includes using the dirty alpha from a low-cost clipping signal generator or intermediate results from a high-cost clipping signal generator to form a part of weight signals which weigh chrominance-luminance signals to determine the parameters.

19. The method of claim 1, wherein the step of generating an area map includes employing information from an identification or classification process in a chromatic plane or a color space and forming a map or display to show different areas, including at least one of a solid backing color area, a shadow area, translucent areas, and transition areas.

20. The method of claim 1, wherein the area map includes its application to a part of user interface in a chromakey-based composition systems.

21. A computer program product stored on a computer-readable medium for generating a clean clipping signal α

for a chromakey function or a video composition, and including one or more computer-readable instructions configured to cause one or more computer processors to perform the steps recited in claim 1.

22. A computer system for generating a clean clipping signal α

for a chromakey function or a video composition and configured to cause one or more computer processors to perform the steps recited in claim 1.

23. A device for generating a clean clipping signal α

for a chromakey function or a video composition and configured to cause one or more computer processors to perform the steps recited in claim 1.

24. A device for generating a clean clipping signal α

for a chromakey function or a video composition where still and moving subjects shot against a solid color background by a video camera are clipped out and overlaid on a new complex background, the device comprising;

means for identifying background colors which are formed by the solid color background, shadows cast by the still and moving subjects, non-uniform reflection caused by spot lighting and non-flat backdrop or flaw wall, and translucent foreground objects, with a 3D volume in a 3D color space;

means for automatically determining parameters which define the 3D volume by using a dirty alpha α

;

means for generating a clean clipping signal α

_shd for the background colors, which occurs to the colors identified as the shadow colors or confined by the 3D volume for the shadow colors, and a clean clipping signal α

_tsl for the translucency colors, which occur to the colors identified as the translucent areas'"'"' colors;

means for identifying foreground colors which are formed by the still and moving subjects with a 3D volume in a 3D color space;

means for classifying into transition colors those which are not identified as either the background or the foreground; and

means for generating an area map which labels each pixel as one of the areas such as background, shadow, translucent area, foreground and transition areas.

25. The device of claim 24, wherein the 3D volume for identifying the background colors further comprises a bounding box or a bounding rectangle which confines the 3D volume for the background colors to a box in a 3D color space.

26. The device of claim 24, wherein the 3D volume for the background colors further comprises a nominal backing color and a shadow axis in a 3D color space, and the shadow axis is a semi line starting from the origin and passing through the nominal backing color, which confines the 3D volume for the shadow colors within a wedged box in a 3D color space.

27. The device of claim 24, wherein the 3D volume for identifying the shadow colors comprises a two-semi-ellipse cylindrical segment which is confined for shadow area'"'"'s colors by the wedged box in a 3D color space.

28. The device of claim 24, wherein the means for generating a clean clipping signal α

_shd further includes means for projecting a pixel color onto the shadow axis, which is represented by;

29. The device of claim 28, wherein the clean clipping signal α

_shd further comprises a square root √

{square root over (α

_shd)} of α

_shd.

30. The device of claim 24, wherein the 3D volume for identifying the background colors further comprises one or multiple translucency axis in a 3D color space, the translucency axis is a semi line starting from a lighting color reflected from translucent objects in foreground and passing through the nominal backing color, and the lighting color is a different color, including mostly a white color.

31. The device of claim 24, wherein the means for generating a clean clipping signal α

_tsl further includes means for projecting a pixel color onto a translucency axis, which is represented by;

32. The device of claim 31, wherein the clean clipping signal α

_tsl further comprises a square root √

{square root over (α

_tsl)} of √

α

_tsl.

33. The device of claim 24, wherein the 3D volume for identifying foreground colors further comprises a 3D volume which may overlap the 3D volume for the background colors.

34. The device of claim 24, wherein the means for automatically determining parameters comprises:

means for initializing the 3D volume for the background colors;

means for generating a low-cost dirty clipping signal or the dirty alpha α

;

means for using the dirty alpha or a value derived from the dirty alpha to construct a part of a weight;

means for generating a norm distance Δ

or a modified norm distance δ

from a pixel color of interest to the shadow axis or from a pixel color along the shadow axis to the nominal backing color;

means for using the norm distance Δ

or δ

to construct another part of a weight;

means for generating weighed color components by multiplying each color component with the weights;

means for accumulating each weighed color component pixel by pixel and averaging the accumulated weighed color component;

means for deriving parameters from the average of accumulated weighed color component and using them to determine the nominal backing color and a bounding box which defines two semi-ellipses;

means for repeatedly implementing the processing of the initializing to accumulating means N times (N≧

3) to achieve the best results during initialization; and

means for recursively implementing the processing of the initializing to accumulating means every M (M≧

1) frames to update the parameters on the fly.

35. The device of claim 34, wherein the means for initializing the 3D volume for the background colors comprises means for generating a chromatic plane divided into 6 equal sectors such that each sector center is exclusively passed through by one of 3 principal colors (red, green, blue) and 3 complementary colors (yellow, magenta, cyan), and the initial 3D volume for the background colors stands on one of the 6 sectors.

36. The device of claim 34, wherein the means for generating a dirty alpha comprises either means for using a standalone function or means for using intermediate results from a function which generates a clean alpha.

37. The device of claim 34, wherein the means for constructing a weight w comprises either means for directly multiplying a low-cost dirty alpha with a norm distance or means for generating an arithmetic combination of a low-cost dirty alpha and a norm distance, including a function ƒ

in terms of Δ and

α

such that;

w=ƒ

(α

, Δ

).

38. The device of claim 34, wherein the color component comprises a color component in one of the following:

luminance and color difference (L, Cr, Cb), luminance and modified color difference vector (L, U, V), (L, I, Q), (L, Pr, Pb), (L, H, S), luminance and a color difference derived from CIE XYZ color system, and principal color R, G, B.

39. The device of claim 34, wherein the means for accumulating each weighed color component comprises means for accumulating of an entire frame of a picture or an entire field of a picture or most parts of a picture.

40. The device of claim 34, wherein the means for deriving from the averages of accumulated weighed color components further comprises means for generating a series of functions which receive the averages of accumulated weighed color components as inputs and generate parameters as outputs for the wedged bounding box, the two semi-ellipse cylindrical segment, the nominal backing color.

41. The device of claim 34, wherein the means for automatically determining parameters includes means for using the dirty alpha from a low-cost clipping signal generator or intermediate results from a high-cost clipping signal generator to form a part of weight signals which weigh chrominance-luminance signals to determine the parameters.

42. The device of claim 24, wherein the means for generating an area map includes means for employing information from an identification or classification process in a chromatic plane or a color space, and means for forming a map or display to show different areas, including at least one of a solid backing color area, a shadow area,

translucent areas, and transition areas.

43. The device of claim 24, wherein the area map includes its application to a part of user interface in a chromakey-based composition system.

44. A system for generating a clean clipping signal α

for a chromakey function or a video composition where still and moving subjects shot against a solid color background by a video camera are clipped out and overlaid on a new complex background, the system comprising;

means for identifying background colors which are formed by the solid color background, shadows cast by the still and moving subjects, non-uniform reflection caused by spot lighting and non-flat backdrop or flaw wall, and translucent foreground objects, with a 3D volume in a 3D color space;

means for automatically determining parameters which define the 3D volume by using a dirty alpha α

;

means for generating a clean clipping signal α

_shd for the background colors, which occurs to the colors identified as the shadow colors or confined by the 3D volume for the shadow colors, and a clean clipping signal α

_tsl for the translucency colors, which occur to the colors identified as the translucent areas'"'"' colors;

means for identifying foreground colors which are formed by the still and moving subjects with a 3D volume in a 3D color space;

means for classifying into transition colors those which are not identified as either the background or the foreground; and

means for generating an area map which labels each pixel as one of the areas such as background, shadow, translucent area, foreground and transition areas.

45. The system of claim 44, wherein the 3D volume for identifying the background colors further comprises a bounding box or a bounding rectangle which confines the 3D volume for the background colors to a box in a 3D color space.

46. The system of claim 44, wherein the 3D volume for the background colors further comprises a nominal backing color and a shadow axis in a 3D color space, and the shadow axis is a semi line starting from the origin and passing through the nominal backing color, which confines the 3D volume for the shadow colors within a wedged box in a 3D color space.

47. The system of claim 44, wherein the 3D volume for identifying the shadow colors comprises a two-semi-ellipse cylindrical segment which is confined for shadow area'"'"'s colors by the wedged box in a 3D color space.

48. The system of claim 44, wherein the means for generating a clean clipping signal α

_shd further includes means for projecting a pixel color onto the shadow axis, which is represented by;

49. The system of claim 44, wherein the clean clipping signal α

_shd further comprises a square root √

{square root over (α

_shd)} of α

_shd.

50. The system of claim 44, wherein the 3D volume for identifying the background colors further comprises one or multiple translucency axis in a 3D color space, the translucency axis is a semi line starting from a lighting color reflected from translucent objects in foreground and passing through the nominal backing color, and the lighting color is mostly a white color but can be any different color.

51. The system of claim 44, wherein the means for generating a clean clipping signal α

_tsl further includes means for projecting a pixel color onto a translucency axis, which is represented by;

52. The system of claim 51, wherein the clean clipping signal α

_tsl further comprises a square root √

{square root over (α

_tsl)} of α

_tsl.

53. The system of claim 44, wherein the 3D volume for identifying foreground colors further comprises a 3D volume which may overlap the 3D volume for the background colors.

54. The system of claim 44, wherein the means for automatically determining parameters comprises:

means for initializing the 3D volume for the background colors;

means for generating a low-cost dirty clipping signal or the dirty alpha α

;

means for using the dirty alpha or a value derived from the dirty alpha to construct a part of a weight;

means for generating a norm distance Δ

or a modified norm distance δ

from a pixel color of interest to the shadow axis or from a pixel color along the shadow axis to the nominal backing color;

means for using the norm distance Δ

or δ

to construct another part of a weight;

means for generating weighed color components by multiplying each color component with the weights;

means for accumulating each weighed color component pixel by pixel and averaging the accumulated weighed color component;

means for deriving parameters from the average of accumulated weighed color component and using them to determine the nominal backing color and a bounding box which defines two semi-ellipses;

means for repeatedly implementing the processing of the initializing to accumulating means N times (N≧

3) to achieve the best results during initialization; and

means for recursively implementing the processing of the initializing to accumulating means every M (M≧

1) frames to update the parameters on the fly.

55. The system of claim 54, wherein the means for initializing the 3D volume for the background colors comprises means for generating a chromatic plane divided into 6 equal sectors such that each sector center is exclusively passed through by one of 3 principal colors (red, green, blue) and 3 complementary colors (yellow, magenta, cyan), and the initial 3D volume for the background colors stands on one of the 6 sectors.

56. The system of claim 54, wherein the means for generating a dirty alpha comprises either means for using a standalone function or means for using intermediate results from a function which generates a clean alpha.

57. The system of claim 54, wherein the means for constructing a weight w comprises either means for directly multiplying a low-cost dirty alpha with a norm distance or means for generating an arithmetic combination of a low-cost dirty alpha and a norm distance, including a function ƒ

in terms of Δ and

α

such that;

w=ƒ

(α

, Δ

).

58. The system of claim 54, wherein the color component comprises a color component in one of the following:

luminance; and

color difference (L, Cr, Cb), luminance and modified color difference vector (L, U, V), (L, I, Q), (L, Pr, Pb), (L, H, S), luminance and a color difference derived from CIL XYZ color system, and principal color R, G, B.

59. The system of claim 54, wherein the means for accumulating each weighed color component comprises means for accumulating of an entire frame of a picture or an entire field of a picture or most parts of a picture.

60. The system of claim 54, wherein the means for deriving from the averages of accumulated weighed color components further comprises means for generating a series of functions which receive the averages of accumulated weighed color components as inputs and generate parameters as outputs for the wedged bounding box, the two semi-ellipse cylindrical segment, the nominal backing color.

61. The system of claim 44, wherein the means for automatically determining parameters includes means for using the dirty alpha from a low-cost clipping signal generator or intermediate results from a high-cost clipping signal generator to form a part of weight signals which weigh chrominance-luminance signals to determine the parameters.

62. The system of claim 44, wherein the means for generating an area map includes means for employing information from an identification or classification process in a chromatic plane or a color space, and means for forming a map or display to show different areas, including at least one of a solid backing color area, a shadow area, translucent areas, and transition areas.

63. The system of claim 44, wherein the area map includes its application to a part of user interface in a chromakey-based composition system.