Method and system for rendering and combining images to form a synthesized view of a scene containing image information from a second image
DCFirst Claim
1. A method for image processing comprising the steps of:
- generating a mosaic containing a plurality of first images of a scene and translation parameters for each of said first images, where said translation parameters define an alignment relationship for each of the first images with respect to common image information contained in each of the first images, said plurality of first images including at least an image having a first viewpoint of said scene and an image having a second viewpoint of said scene, said first and second viewpoints being non-coincidental;
generating a second image;
producing an indicium of viewpoint while generating said second image;
rendering, in response to said indicium of viewpoint, a synthesized image of said scene from said mosaic, said synthesized image comprising image information from said first and second viewpoints of said scene; and
compositing said synthesized image with said second image to produce a composite image including parallax information.
1 Assignment
Litigations
0 Petitions
Accused Products
Abstract
An image processing system for imaging a scene to mosaic, selecting a new viewpoint of the scene, and rendering a synthetic image from the mosaic of the scene from that new viewpoint. The synthesized image is then combined with a second image. The combination of the second image and the synthetic image generates a composite image containing a realistic combination of objects in the second image and the scene. Using this system, a production set or other scene need only be created once, then imaged by the system. Thereafter, through image processing, any view of the scene can be synthesized and combined with separately imaged performers or other objects to generate the composite image. As such, a production set or other scene can be repetitively reused without recreating the physical scene.
254 Citations
25 Claims
-
1. A method for image processing comprising the steps of:
-
generating a mosaic containing a plurality of first images of a scene and translation parameters for each of said first images, where said translation parameters define an alignment relationship for each of the first images with respect to common image information contained in each of the first images, said plurality of first images including at least an image having a first viewpoint of said scene and an image having a second viewpoint of said scene, said first and second viewpoints being non-coincidental;
generating a second image;
producing an indicium of viewpoint while generating said second image;
rendering, in response to said indicium of viewpoint, a synthesized image of said scene from said mosaic, said synthesized image comprising image information from said first and second viewpoints of said scene; and
compositing said synthesized image with said second image to produce a composite image including parallax information. - View Dependent Claims (2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16)
providing a plurality of indicia proximate an area where a performance is imaged as said second image; and
imaging said plurality of indicia as a landmark image while generating said second image.
-
-
8. The method of claim 7 wherein said landmark imaging step further comprises the step of generating said second image using a camera.
-
9. The method of claim 7 further comprising a step of imaging a plurality of reference images representing said indicia viewed from a plurality of viewpoints.
-
10. The method of claim 9 further comprising a step of correcting said indicium of viewpoint using said plurality of reference images.
-
11. The method of claim 10 further comprising the steps of:
-
measuring an absolute location of the indicia; and
computing camera pose in response to the absolute location of the indicia and a measurement of camera pose represented by said indicium of viewpoint.
-
-
12. The method of claim 1 wherein said mosaic is a three-dimensional mosaic containing an image mosaic representing a panoramic view of the scene and a shape mosaic representing a three-dimensional geometry of the scene.
-
13. The method of claim 12 wherein said synthesized image is rendered from said three-dimensional mosaic.
-
14. The method of claim 13 wherein said mosaic generating step further comprises the steps of:
-
imaging a scene from a plurality of grid points;
generating a mosaic at each of said grid points; and
storing each of said mosaics with reference to a particular grid point.
-
-
15. The method of claim 14 wherein said step of generating a mosaic at each of said grid points further comprises:
-
generating a plurality of mosaics at each grid point, where each of said mosaics is generated at a different illumination level of the scene;
rendering said synthesized image by matching illumination levels of said synthesized image and said second image.
-
-
16. The method of claim 1 wherein the indicium of viewpoint is produced by a user controlled input device.
-
17. A method for image processing comprising the steps of:
-
generating a mosaic containing a plurality of first images of a scene, said plurality of first images including an image having a first viewpoint of said scene and an image having a second viewpoint of said scene;
generating a second image;
providing a plurality of indicia proximate an area where a performance is imaged as said second image;
imaging, using a tracking camera that is separate from a camera used to generate said second image, said plurality of indicia as a reference image while generating said second image rendering, in response to said indicium of viewpoint;
a synthesized image of said scene from said mosaic, said synthesized image comprising image information from said first and second viewpoints of said scene; and
compositing said synthesized image with said second image to produce a composite image.
-
-
18. A method for image processing comprising the steps of:
-
generating a mosaic containing a plurality of first images of a scene, said plurality of first images including an image having a first viewpoint of said scene and an image having a second viewpoint of said scene;
generating a second image;
providing a plurality of indicia proximate an area where a performance is imaged as said second image;
imaging said plurality of indicia as a reference image while generating said second image;
rendering, in response to said indicium of viewpoint, a synthesized image of said scene from said mosaic, said synthesized image comprising image information from said first and second viewpoints of said scene;
compositing said synthesized image with said second image to produce a composite image;
imaging a plurality of reference images representing said indicia viewed from a plurality of viewpoints;
correcting said indicium of viewpoint using said plurality of reference images;
selecting a reference image that represents a viewpoint that best represents a viewpoint indicated by said indicium of viewpoint;
warping said selected reference image into alignment with a landmark image generated from said viewpoint represented by said indicium of viewpoint;
computing, in response to said alignment, offset parameters for correcting said indicium of viewpoint; and
generating said synthesized image in response to said corrected indicium of viewpoint.
-
-
19. An image processing system comprising:
-
mosaic generation means for generating a mosaic of first images representing a scene and translation parameters for each of said first images, where said translation parameters define an alignment relationship for each of the first images with respect to common image information contained in each of the first images and having a first viewpoint;
viewpoint indicium means for generating an indicium of viewpoint, non-coincidental with said first viewpoint, for a second image;
rendering means, connected to said mosaic generation means and said viewpoint indicium means, for producing a synthetic image derived from said mosaic that represents a view of the scene corresponding to the indicium of viewpoint; and
compositing means, connected to said rendering means and said viewpoint indicium means, for combining said synthesized image and said second image to form a composite image including parallax information. - View Dependent Claims (20, 21, 22, 23, 24, 25)
means for generating a plurality of mosaics, where each mosaic in said plurality of mosaics represents a panoramic view of the scene from a unique location.
-
-
21. The system of claim 19 wherein said viewpoint indicium means further comprises:
-
means for tracking a camera pose as said camera produces said second image; and
means, connected to said tracking means, for generating said camera pose as an indicium of viewpoint.
-
-
22. The system of claim 19 wherein said viewpoint indicium means further comprises:
means for producing said indicium of viewpoint in response to user commands.
-
23. The system of claim 19 wherein said compositing means further comprises:
means for combining said synthetic image and said second image using a chroma-key process.
-
24. The system of claim 21 wherein said camera pose includes indicium of viewpoint selected from the following group of camera parameters:
- three-dimensional position, pan, tilt, roll and zoom.
-
25. The system of claim 19 further comprising:
- second mosaic generation means for generating a second mosaic of images representing a second scene; and
second rendering means, connected to said viewpoint indicium means and said second mosaic generation means, for producing said second image.
- second mosaic generation means for generating a second mosaic of images representing a second scene; and
Specification