System and method for real time insertion into video with occlusion on areas containing multiple colors

• US 20020027617A1
• Filed: 12/13/2000
• Published: 03/07/2002
• Est. Priority Date: 12/13/1999
• Status: Active Grant

• Current Assignee: Disney Enterprises Incorporated (The Walt Disney Company)
• Original Assignee: PVI Virtual Media Services, LLC (Altice USA, Inc.)
• Inventors: House, Gregory, Harney, Kevin, Jeffers, James L.
• Granted Patent: US 7,015,978 B2
• Time in Patent Office: Days
• US Class Current: 348/587
• CPC Class Codes: H04N 5/2723 Insertion of virtual advertisement; Replacing advertisements physical present in the scene by virtual advertisement data processing systems or methods specially adapted for marketing G06Q30/02
• CPC Class Codes: H04N 9/74 for obtaining special effects H04N9/65 - H04N9/73 take precedence

Abstract

A system, method and computer program product for real-time insertion of indicia (e.g., an advertisement) into a live (or taped) broadcast. Typically, the invention allow insertion with occlusion onto surfaces such as tennis courts, the wall behind home plate on a baseball field, the grass or turf on a soccer or football stadium, and the like. The occlusion processing described herein can handle multiple colors in the background image during the insertion of the indicia into a video image. The invention can thus adapt to changing light conditions in the video image. The process includes obtaining a video image from a camera. The video image is typically digitized. Next, the pixels within the video image are sampled. A plurality of background colors are then identified for the sampled pixels. An opacity value is then assigned to each pixel in the indicia based on whether the color of a positionally corresponding pixel in the video image is the same color as one of the plurality of background colors. Finally, pixels within the indicia are displayed in the video image if the color of the positionally corresponding pixel in the video image is the same color as one of the background colors and a pixel in the indicia is not displayed in the video image if the positionally corresponding pixel in the video image is not the same color of as one of the background colors.

Claims

1. A method of occlusion processing that identifies background colors in a video image for the purposes of inserting an indicia into the video image, wherein the background is comprised of two or more colors, comprising:

(a) sampling pixels in the video image to obtain a color component value for each pixel;

(b) generating a histogram for each of said color component values, wherein each said histogram represents a total number of occurrences of each of said color component values for said sampled pixels;

(c) selecting a dominant node from each of said histograms, wherein said dominant nodes identify a background color;

(d) is carding each of said sampled pixels having color component values that fall within said dominant nodes from said sampled pixels; and

(e) repeating steps (b) through (d) to identify an additional background color.

2. The method of claim 1, wherein said color component values are Y, U and V.

3. The method of claim 1, wherein said color component values are R, G and B or the component value of a color format that can be derived from R, G and B.

4. The method of claim 1, further comprises repeating steps (b) through (d) to identify additional background colors until no further dominant nodes can be selected.

5. The method of claim 1, wherein step (a) further comprises sampling pixels within a predetermined region within the video image to obtain a Y, U, and V value for each pixel.

6. The occlusion processing method of claim 5, wherein said predetermined region within the video image is an insert location for inserting the indicia in the video image.

7. The method of claim 1, further comprising:

(f) dividing the video image into a plurality of regions;

wherein step (i) precedes step (a), and wherein steps (a) through (e) are performed for each of said plurality of regions.

8. The method of claim 1, further comprising (f) identifying said discarded sampled pixels as background pixels and undiscarded pixels as foreground pixels.

9. A method of occlusion processing for inserting an indicia into a video image, comprising:

sampling pixels in the video image to obtain a color component value for each pixel;

identifying a plurality of background colors from said color component values for said sampled pixels; and

assigning an opacity value to each pixel in the indicia based on whether the color of a positionally corresponding pixel in the video image is the same color as one of said plurality of background colors, whereby a pixel within the indicia is displayed in the video image if the color of said positionally corresponding pixel in the video image is the same color as one of said background colors and a pixel in the indicia is not displayed in the video image if said positionally corresponding pixel in the video image is not the same color of as one of said background colors or the component value of a color format that can be derived from R, G and B.

10. The method of claim 9, wherein said identifying includes:

(a) generating a histogram for each of said color component values, wherein each said histogram represents a total number of occurrences of each of said color component values for said sampled pixels;

(b) selecting a dominant node from each of said histograms, wherein said dominant nodes identify a background color;

(c) discarding each of said sampled pixels having color component values that fall within said dominant nodes from said sampled pixels; and

(d) repeating steps (a) through (c) to identify an additional background color.

11. The method of claim 9, wherein said color component values are Y, U and V.

12. The method of claim 9, wherein said color component values are R, G and B

13. The method of claim 9, further comprises repeating steps (a) through (c) to identify additional background colors until no further dominant nodes can be selected.

14. The method of claim 9, wherein sampling further comprises sampling pixels within a predetermined region within the video image to obtain a color component value for each pixel.

15. The occlusion processing method of claim 14, wherein said predetermined region within the video image is an insert location for inserting the indicia in the video image.

16. The method of claim 9, further comprising dividing the video image into a plurality of regions, wherein steps (a) through (d) are performed for each of said plurality of regions.

17. The method of claim 9, further comprising (f) identifying said discarded sampled pixels as background pixels and undiscarded pixels as foreground pixels.

18. The method of claim 9, wherein said assigning said opacity value comprises:

constructing a color component look-up table for each background color, wherein each said look-up table defines opacity values for pixels;

addressing each said look-up table for each background color based on said color component values for said positionally corresponding pixel to select an opacity value for each background color;

combining through a mathematical operation said opacity values for each background color to produce a color opacity value; and

combining through a mathematical operation said color opacity values and assigning it to each said pixel in the indicia.

19. The method of claim 9, wherein said color component values are Y, U and V, and wherein each color component value has a separate look-up table.

20. The method of claim 18, wherein said color opacity values have a range from transparent to opaque.

21. The method of claim 9, wherein said assigning said opacity value includes comparing the color of a positionally corresponding pixel in the video image to one ore more thresholds.

22. A method of occlusion processing for inserting an indicia into a video image, comprising:

obtaining a video image from a camera;

sampling pixels in said video image;

identifying a plurality of background colors for said sampled pixels;

assigning an opacity value to each pixel in the indicia based on whether the color of a positionally corresponding pixel in the video image is the same color as one of said plurality of background colors; and

displaying said video image having said pixels associated with the indicia inserted therein, wherein pixels of the indicia are displayed based on said opacity value.