System for establishing a three-dimensional garbage matte which enables simplified adjusting of spatial relationships between physical and virtual scene elements

• US 6,034,739 A
• Filed: 06/09/1997
• Issued: 03/07/2000
• Est. Priority Date: 06/09/1997
• Status: Expired due to Term

• Current Assignee: RT-Set Real Time Synthesized Entertainment Technology Ltd.
• Original Assignee: Evans & Sutherland Computer Corp. (Elevate Entertainment, Inc.)
• Inventors: Rohlfing, Thomas R., Houskeeper, James H.
• Primary Examiner(s): Faile, Andrew I.
• Assistant Examiner(s): BROWN, RUEBEN M
• Application Number:
  US08/870,756
• Time in Patent Office: 1,002 Days
• Field of Search: 348/584-589, 348/566, 348/576, 348/577, 348/592, 348/598, 345/819, 345/113, 345/114, 345/420, 345/423, 345/426, 345/434
• US Class Current: 348/586
• CPC Class Codes: H04N 5/222 Studio circuitry; Studio devices; Studio equipment cameras or camera modules comprising electronic image sensors, or control thereof H04N23/00
• CPC Class Codes: H04N 5/2224 related to virtual studio applications
• CPC Class Codes: H04N 5/272 Means for inserting a foreground image in a background image, i.e. inlay, outlay
• CPC Class Codes: H04N 9/74 for obtaining special effects H04N9/65 - H04N9/73 take precedence

Abstract

A new method and apparatus for creating, positioning and manipulating a garbage matte which obscures everything outside of a three-dimensional physical set or blue stage. Effectively, the garbage matte is a "negative" of a blue stage. What is important is that the garbage matte is created either as a virtual enclosure around all virtual objects in the virtual environment, or simply as a background color of the graphics engine generating the garbage matte. In this way, the garbage matte provides extensions of blue screens which comprise the blue stage, extending all the way around and behind the television cameras. Consequently, a compositor will not include in a composited output image those physical set elements which are masked by the virtual garbage matte. The compositor will replace areas hidden by the garbage matte with image data supplied by the computer image generator which stores the virtual environment. Furthermore, it is now possible to move the virtual location of the physical set (including the television cameras) within the virtual environment without having to readjust the position of the garbage matte relative to the model by anchoring virtual objects together so as to maintain relative position relationships at all times.

Claims

1. A method for generating a garbage matte which masks undesired objects in view of a television camera transmitting video images of a physical set, said video images being combined with virtual images to thereby generate a composite output image without the undesired objects, said method comprising the steps of:

(1) modeling the physical set in a virtual environment;

(2) generating a virtual garbage matte enclosure in the virtual environment which surrounds the model of the physical set; and

(3) rendering the model of the physical set from the television camera'"'"'s current position to thereby remove the model of the physical set such that a resulting shape for the garbage matte is a negative image of the model of the physical set.

2. The method as defined in claim 1 wherein the method further comprises the step of anchoring the garbage matte and the model of the physical set to a relative position of the television camera so that the garbage matte is always positioned in a same relative position with respect to the model of the physical set and the television camera, regardless of where the television camera and the model of the physical set are relocated within the virtual environment or repositioned with respect to an original orientation.

3. The method as defined in claim 1 wherein the method further comprises the step of including within the virtual garbage matte enclosure all virtual objects which are defined within a virtual object database which corresponds to virtual objects which can appear within the virtual environment.

4. The method as defined in claim 1 wherein the method further comprises the steps of:

(1) generating the model of the physical set utilizing at least one polygon which is defined in the virtual environment; and

(2) generating the virtual garbage matte enclosure utilizing a plurality of polygons which are defined in the virtual environment.

5. The method as defined in claim 4 wherein the method further comprises the step of anchoring the plurality of polygons which define the physical set so that they can be manipulated as a single unit.

6. The method as defined in claim 4 wherein the method further comprises the steps of:

(1) assigning a first color to the at least one polygon which defines the physical set; and

(2) assigning a second color to the plurality of polygons which define the virtual garbage matte enclosure.

7. The method as defined in claim 6 wherein the method further comprises the step of selecting a second color for the garbage matte according to a compositor which utilizes the second color for masking out undesired objects seen by the television camera, and the first color is utilized by the compositor as representing areas of the composite output image where video signals should come from the television camera.

8. The method as defined in claim 7 wherein the method further comprises the steps of:

(1) assigning the first color a red/green/blue (RGB) value of white; and

(2) assigning the second color an RGB value of black.

9. The method as defined in claim 6 wherein the method further comprises the step of rendering in the virtual environment the garbage matte from the virtual garbage matte enclosure by rendering the first color everywhere within the garbage matte except where the second color pixels of the model of the physical set cover the garbage matte.

10. The method as defined in claim 1 wherein the method further comprises the step of providing a user interactive method of manipulating a position and an orientation of a virtual camera and the virtual models of all anchored objects to thereby maintain the anchored objects in desired relative locations.

11. The method as defined in claim 2 wherein the method further comprises the step of generating at least one virtual object which is not anchored to a position relative to the model of the physical set, such that moving the model of the physical set within the virtual environment does not cause a corresponding relative movement of the at least one virtual object.

12. The method as defined in claim 1 wherein the method further comprises the step of supplying virtual images to replace areas of the composite output image which show the garbage matte.

13. The method as defined in claim 1 wherein the method further comprises the step of generating at least one new garbage matte polygon separate from the garbage matte to thereby mask at least one undesirable object which is completely within boundaries of the model of the physical set.

14. The method as defined in claim 1 wherein the method further comprises the step of generating at least one object within the virtual environment which is anchored to the physical set.

15. The method as defined in claim 1 wherein the method further comprises the step of providing a user interactive method of manipulating a position and an orientation of the model of the physical set with respect to the virtual environment.

16. The method as defined in claim 1 wherein the method further comprises the step of generating a plurality of objects within the virtual environment which are anchored to the physical set.

17. The method as defined in claim 16 wherein the method further comprises the step of providing a user interactive method of manipulating a position and an orientation of each of the plurality of objects with respect to the plurality of objects which remain anchored to each other.

18. A method for generating a garbage matte which masks undesired objects in view of a television camera transmitting video images of a physical set, said video images being combined with virtual images to thereby generate a composite output image without the undesired objects, said method comprising the steps of:

(1) modeling the physical set in a virtual environment;

(2) assigning a first color to the model of the physical set;

(3) assigning a second color as a background color of a graphics engine generating the garbage matte; and

(4) not changing the second color anywhere except where the first color covers the background from the television camera'"'"'s current position to thereby remove the model of the physical set such that a resulting shape for the garbage matte is a negative image of the model of the physical set.

19. The method as defined in claim 18 wherein the method further comprises the step of anchoring the garbage matte and the model of the physical set to a relative position of the television camera so that the garbage matte is always positioned in a same relative position with respect to the model of the physical set and the television camera, regardless of where the television camera and the model of the physical set are relocated within the virtual environment or repositioned with respect to an original orientation.

20. The method as defined in claim 18 wherein the method further comprises the step of generating the model of the physical set utilizing at least one polygon which is defined in the virtual environment.

21. The method as defined in claim 20 wherein the method further comprises the step of anchoring the plurality of polygons which define the physical set so that they can be manipulated as a single unit.

22. The method as defined in claim 18 wherein the method further comprises the step of selecting a second color for the background according to a compositor which utilizes the second color for masking out undesired objects seen by the television camera, and the first color is utilized by the compositor as representing areas of the composite output image where video signals should come from the television camera.

23. The method as defined in claim 22 wherein the method further comprises the steps of:

(1) assigning the first color a red/green/blue (RGB) value of white; and

(2) assigning the second color an RGB value of black.

24. The method as defined in claim 18 wherein the method further comprises the step of providing a user interactive method of manipulating a position and an orientation of a virtual camera and the virtual models of all anchored objects to thereby maintain the anchored objects in desired relative locations.

25. The method as defined in claim 18 wherein the method further comprises the step of generating at least one virtual object which is not anchored to a position relative to the model of the physical set, such that moving the model of the physical set within the virtual environment does not cause a corresponding relative movement of the at least one virtual object.

26. The method as defined in claim 18 wherein the method further comprises the step of supplying virtual images to replace areas of the composite output image which show the garbage matte.

27. The method as defined in claim 18 wherein the method further comprises the step of generating at least one new garbage matte polygon separate from the garbage matte to thereby mask at least one undesirable object which is completely with in boundaries of the model of the physical set.

28. The method as defined in claim 18 wherein the method further comprises the step of generating at least one object within the virtual environment which is anchored to the physical set.

29. The method as defined in claim 18 wherein the method further comprises the step of providing a user interactive method of manipulating a position and an orientation of the model of the physical set with respect to the virtual environment.

30. The method as defined in claim 18 wherein the method further comprises the step of generating a plurality of objects within the virtual environment which are anchored to the physical set.

31. The method as defined in claim 30 wherein the method further comprises the step of providing a user interactive method of manipulating a position and an orientation of each of the plurality of objects with respect to the plurality of objects which remain anchored to each other.

32. A method for manipulating virtual objects which at least include a model of a physical set, and at least one garbage matte which masks undesired objects in view of a television camera transmitting video images of a physical set, said video images being combined with virtual images to thereby generate a composite output image without the undesired objects, said method comprising the steps of:

(1) modeling the physical set in a virtual environment;

(2) generating a garbage matte which is a negative image of the model of the physical set;

(3) anchoring the garbage matte to the model of the physical set and a model of the television camera to form a single unit of virtual objects; and

(4) manipulating a location of the single unit within the virtual environment without disturbing relative positions such that the virtual objects within the single unit are always positioned in a same relative position.

33. The method as defined in claim 32 further comprising the steps of:

(1) unanchoring the virtual objects within the single unit;

(2) repositioning at least one of the virtual objects with respect to the other virtual objects; and

(3) anchoring the virtual objects within the single unit.

34. The method as defined in claim 33 further comprising the step of creating at least one additional virtual object within the virtual environment and adding it to the single unit.

35. A system for generating a garbage matte which transmitting video images of a physical set, said video images being combined with virtual images to thereby generate a composite output image without the undesired objects, said system comprising:

(a) the television camera for creating video images of the physical set;

(b) a video delay means for delaying the video images to enable synchronization of the video images and the virtual images;

(c) a television camera tracker associated with the television camera for monitoring position information of the television camera;

(d) a control computer in communication with the television camera tracker for receiving the position information and for executing software which controls functions of the system;

(e) a graphics computer coupled in communication with the control computer for generating a virtual environment from which the virtual images are taken for combining with the video images to generate the composite output image, wherein a model of the physical set is generated in the virtual environment, and wherein a virtual three dimensional garbage matter enclosure is generated to thereby surround corresponding objects in the physical set that are not to be seen in the composite output image; and

(f) a compositor which receives the video images from the video delay means and the graphics computer to thereby generate the composite output image.