Multi-camera virtual set system employing still store frame buffers for each camera

• US 6,014,163 A
• Filed: 06/09/1997
• Issued: 01/11/2000
• Est. Priority Date: 06/09/1997
• Status: Expired due to Term

• Current Assignee: RT-Set Real Time Synthesized Entertainment Technology Ltd.
• Original Assignee: Evans & Sutherland Computer Corp. (Elevate Entertainment, Inc.)
• Inventors: Houskeeper, James H.
• Primary Examiner(s): Britton, Howard
• Assistant Examiner(s): VO, TUNG T
• Application Number:
  US08/870,752
• Time in Patent Office: 946 Days
• Field of Search: 348/47, 348/48, 348/49, 348/50, 348/39, 348/152, 348/157, 348/159, 348/349, 348/419, 348/425, 348/584, 348/586, 348/589, 348/600, 345/36, 345/8, 358/87, 395/125
• US Class Current: 348/47
• CPC Class Codes: H04N 13/156 Mixing image signals
• CPC Class Codes: H04N 13/239 using two 2D image sensors having a relative position equal to or related to the interocular distance H04N13/243 takes precedence
• CPC Class Codes: H04N 13/246 Calibration of cameras
• CPC Class Codes: H04N 13/275 from 3D object models, e.g. computer-generated stereoscopic image signals
• CPC Class Codes: H04N 13/286 having separate monoscopic and stereoscopic modes
• CPC Class Codes: H04N 13/296 Synchronisation thereof; Control thereof
• CPC Class Codes: H04N 5/222 Studio circuitry; Studio devices; Studio equipment cameras or camera modules comprising electronic image sensors, or control thereof H04N23/00
• CPC Class Codes: H04N 5/2224 related to virtual studio applications
• CPC Class Codes: H04N 5/272 Means for inserting a foreground image in a background image, i.e. inlay, outlay

Abstract

A method and apparatus for producing data in a virtual set memory representing a composite image. The composite image data is produced from a combination of data representing a virtual image and data representing a camera field of view image. The invention includes a plurality of cameras, each disposed for developing data representing a respective field of view and each having a still store frame buffer associated therewith for receiving and storing data representing virtual images. The still store frame buffers are configured for subsequent selective retrieval. The invention also includes a virtual image data source for generating data representing virtual images and for supplying such data to the still store frame buffers associated with each of the plurality of cameras. Finally, the invention includes at least one compositor for compositing data representing the field of view of a selected one of the plurality of cameras and data representing a virtual image retrieved from a still store frame buffer associated with the selected one of the plurality of cameras. The compositor produces data in the virtual set memory.

Claims

1. Apparatus for producing data in a virtual set memory representing a composite image from a combination of a virtual image and a camera field of view image, the apparatus comprising:

a plurality of cameras, each disposed for capturing data representing a respective field of view and each having a frame buffer associated therewith for receiving and storing data representing virtual images, for subsequent selective retrieval;

a virtual image data source for generating data representing virtual images and for supplying such data to the frame buffer associated with a broadcast camera configured for moving spatially, wherein the last image drawn by the image generator is stored in the frame buffer when the broadcast camera stops moving and then virtual image is supplied to another single movable camera; and

a means for compositing data representing the field of view of a selected one of the plurality of cameras and data representing a virtual image retrieved from a frame buffer associated with the selected one of the plurality of cameras, to produce data in the virtual set memory.

2. Apparatus as in claim 1 wherein said plurality of cameras further comprise a camera tracker electrically coupled between each of the plurality of cameras and the virtual image data source, wherein the camera tracker provides, to the virtual image data source, data on camera position and movement, including pan, tilt, zoom, and focus, thereby enabling the virtual image data source to consider camera position data when generating the data representing virtual images.

3. Apparatus as in claim 1 wherein said plurality of cameras further comprise a video delay electrically coupled between each of the plurality of cameras and the means for compositing data, wherein the data representing the field of view of the selected one of the plurality cameras is appropriately delayed for compositing with the data representing a virtual image.

4. Apparatus as in claim 1 wherein the data representing a virtual image comprises data for a three-dimensional graphic having a foreground and a background, and data for a matte channel.

5. Apparatus as in claim 1 wherein said virtual image data source comprises at least one image generator for generating three-dimensional image data to be stored in the frame buffers, the at least one image generator sharing resources with the plurality of cameras.

6. Apparatus as in claim 1 wherein said virtual image data source comprises an external video camera for producing video images to generate the data representing virtual images.

7. Apparatus as in claim 6 wherein the external video camera produces live video.

8. Apparatus as in claim 1 wherein the means for compositing comprises at least one compositor electrically coupled to each of the frame buffers and to each of the video delays of each of the cameras, the at least one compositor for layering the data representing the field of view of the selected one of the plurality of cameras and the data representing the virtual image from one of the frame buffers associated with one of the plurality of cameras, wherein data representing a single composite image is stored in the virtual set memory.

9. Apparatus as in claim 8 wherein the at least one compositor comprises a compositor electrically coupled to each of the frame buffers associated with each of the plurality of cameras and wherein said each compositor is tuned to each camera.

10. Apparatus as in claim 1 wherein the virtual set memory comprises data representing a composite image including a virtual image and a camera field of view image.

17. The apparatus of claim 1 wherein the virtual image data source is a plurality of virtual image data sources wherein each of said data sources is combined with the frame buffer and provides virtual images directly to the frame buffer when the camera is not moving.

11. A method for producing data in a virtual set memory representing a composite image from a combination of a virtual image and a camera field of view image, the method comprising the steps of:

(a) providing a plurality of cameras, each having a field of view and each having a frame buffer associated therewith;

(b) providing a virtual image data source for generating data representing a virtual image;

(c) storing the data representing the virtual image in the frame buffers associated with one broadcast camera while said camera is moving, wherein the last image drawn by the image generator is stored in the frame buffer when the broadcast camera stops moving and the virtual image is supplied to another single moving camera;

(d) selectively retrieving the data representing the virtual image from one of the frame buffers of one of the plurality of cameras; and

(e) compositing data representing a field of view of one of said plurality of cameras and the data representing the virtual image of one of the frame buffers to produce data in the virtual set memory.

12. The method of claim 11 wherein step (a) further comprises providing a camera tracker and a video delay associated with each of the plurality of cameras.

13. The method of claim 11 wherein step (b) comprises providing an image generator for generating a three-dimensional image having a foreground and a background.

14. The method of claim 13 wherein step (b) further comprises providing an image generator for generating a matte channel.

15. The method of claim 11 wherein step (b) comprises providing an external video signal including live video in three-dimensional data sets rendered by processors.

16. The method of claim 11 wherein step (e) comprises providing a virtual set image based on the data in the virtual set memory.

18. A method for producing data in a virtual set memory representing a composite image from a combination of a virtual image and a camera field of view image, the method comprising the step of:

(a) providing a plurality of cameras, each having a field of view and each having a frame buffer associated therewith;

(b) providing a virtual image data source in each frame buffer for rendering data representing a virtual image;

(c) rendering and storing the data representing the virtual image in each of the frame buffers associated with each camera when a camera is not moving;

(d) selectively retrieving the data representing the virtual image from one of the frame buffers of one of the plurality of cameras; and

(e) compositing data representing a field of view of one of said plurality of cameras and the data representing the virtual image in the corresponding frame buffer to produce data in the virtual set memory.