Method for creating virtual reality

• US 6,084,979 A
• Filed: 06/20/1996
• Issued: 07/04/2000
• Est. Priority Date: 06/20/1996
• Status: Expired due to Term

• Current Assignee: Carnegie Mellon University
• Original Assignee: Carnegie Mellon University
• Inventors: Rander, Peter W., Narayanan, P. J., Kanade, Takeo
• Primary Examiner(s): Boudreau, Leo H.
• Assistant Examiner(s): Werner, Brian P.
• Application Number:
  US08/671,791
• Time in Patent Office: 1,475 Days
• Field of Search: 382/103, 382/106, 382/153, 382/154, 382/173, 382/275, 382/100, 382/285, 382/293, 345/125, 345/419, 345/420, 345/421, 345/422, 345/423, 345/424, 345/425, 345/426, 345/427, 345/358, 348/42, 348/47, 348/48, 348/552, 434/23, 434/26, 434/37, 434/47, 434/79, 434/32, 434/63, 463/31, 463/32, 463/33, 364/468.04
• US Class Current: 382/154
• CPC Class Codes: H04N 13/117 the virtual viewpoint locations being selected by the viewers or determined by viewer tracking
• CPC Class Codes: H04N 13/128 Adjusting depth or disparity
• CPC Class Codes: H04N 13/15 for colour aspects of image signals
• CPC Class Codes: H04N 13/167 Synchronising or controlling image signals
• CPC Class Codes: H04N 13/189 Recording image signals; Reproducing recorded image signals
• CPC Class Codes: H04N 13/221 using the relative movement between cameras and objects
• CPC Class Codes: H04N 13/243 using three or more 2D image sensors
• CPC Class Codes: H04N 13/246 Calibration of cameras
• CPC Class Codes: H04N 13/25 using two or more image sensors with different characteristics other than in their location or field of view, e.g. having different resolutions or colour pickup characteristics; using image signals from one sensor to control the characteristics of another sensor
• CPC Class Codes: H04N 13/257 Colour aspects
• CPC Class Codes: H04N 13/289 Switching between monoscopic and stereoscopic modes
• CPC Class Codes: H04N 13/296 Synchronisation thereof; Control thereof
• CPC Class Codes: H04N 13/332 Displays for viewing with the aid of special glasses or head-mounted displays [HMD]
• CPC Class Codes: H04N 13/373 for tracking forward-backward translational head movements, i.e. longitudinal movements
• CPC Class Codes: H04N 13/376 for tracking left-right translational head movements, i.e. lateral movements
• CPC Class Codes: H04N 13/38 for tracking vertical translational head movements
• CPC Class Codes: H04N 13/383 for tracking with gaze detection, i.e. detecting the lines of sight of the viewer's eyes
• CPC Class Codes: H04N 13/398 Synchronisation thereof; Control thereof
• CPC Class Codes: H04N 19/597 specially adapted for multi-view video sequence encoding
• CPC Class Codes: H04N 2013/0081 Depth or disparity estimation from stereoscopic image signals
• CPC Class Codes: H04N 2013/0088 : Synthesising a monoscopic image signal from stereoscopic images, e.g. synthesising a panoramic or high resolution monoscopic image

Abstract

A method of virtualizing reality, i.e., a method of creating virtual reality from images of a real event, is comprised of the steps of capturing a plurality of images of each time instant of a real event using a plurality of cameras positioned at a plurality of angles. Each image is stored as intensity and/or color information. A suitable internal representation is computed from these images and the information regarding the camera angles. An image of each time instant may be generated from any viewing angle using the internal representation of it. The virtual viewpoints could be displayed on a single TV screen or using a stereoscopic display device for a true three-dimensional effect. The event thus virtualized can be navigated through, and interacted with, any virtual reality system.

Claims

1. A method of creating virtual reality, comprising:

positioning at least three cameras surrounding a visual object to be virtualized;

synchronizing the operation of said at least three cameras;

generating a plurality of digitized images for each sequence of shots of said visual object by a corresponding one of said at least three cameras;

creating a plurality of discrete depth maps, wherein each depth map is extracted from an associated one of said plurality of digitized images;

generating a plurality of weighted depth maps by attaching reliability values to each of said plurality of discrete depth maps; and

preparing an object-centered model of said visual object including;

fusing said each of said plurality of digitized images and a weighted depth map associated therewith, thereby creating a tessellated surface for said visual object, and

extracting a virtualized surface for said visual object by selectively projecting each voxel in an object-centered three-dimensional space onto said tessellated surface and finding one or more voxels with zero-crossings.

2. The method of claim 1 further comprising storing said object-centered model.

3. The method of claim 1, further comprising manually editing at least one of said plurality of discrete depth maps prior to generating said plurality of weighted depth maps.

4. The method as in claim 1, wherein generating said plurality of digitized images includes:

capturing a plurality of analog images from each of said at least three cameras in parallel;

storing each of said plurality of analog images along with a time-stamp therefor; and

generating said plurality of digitized images by digitizing said each of said plurality of analog images.

5. The method of claim 4, further comprising correlating each of said plurality of digitized images using time-stamps associated with corresponding analog images prior to creating said plurality of discrete depth maps.

6. The method of claim 1, wherein preparing said object-centered model includes:

pre-computing visibility information for each shot of said visual object by each of said at least three cameras; and

storing said visibility information as part of said object-centered model.

7. The method of claim 6, further comprising generating a three-dimensional virtualized image of said visual object from at least one viewing angle using said object-centered model thereof.

8. The method of claim 7, wherein generating said three-dimensional virtualized image includes rendering said three-dimensional visualized image using CAD techniques.

9. The method as in claim 7, wherein generating said three-dimensional virtualized image includes:

initially rendering a reference scene description of said three-dimensional virtualized image while omitting rendering of pixels that map to depth discontinuities; and

subsequently rendering said pixels that map to depth discontinuities using supporting scene descriptions of said three-dimensional virtualized image.

10. A virtual reality system, comprising:

at least three cameras positioned around a visual object to be virtualized;

scene transcription hardware in communication with said cameras, wherein said scene transcription hardware includes;

a sync signal generator in communication with said at least three cameras, wherein said sync signal generator produces a sync signal that synchronizes the operation of said at least three cameras,

at least three storage devices, wherein each of said at least three storage devices is coupled to a corresponding one of said at least three cameras to record analog images supplied by said corresponding one of said at least three cameras on a respective magnetic storage medium,

a time code generator coupled to said at least three storage devices, wherein said time code generator time stamps each of said analog images, and

an A/D conversion device configured to operate on each said magnetic storage medium off-line, thereby converting said analog images into corresponding digitized images; and

an internal representation generator in communication with said scene transcription hardware, said internal representation generator creating a plurality of discrete depth maps by extracting each depth map from an associated one of said digitized images, and wherein said internal representation generator is configured to create an object centered model of said visual object to facilitate generation of a three-dimensional virtualized view thereof.

11. The system of claim 10, further comprising a view generator in communication with said internal representation hardware.

12. The system of claim 11, further comprising a stereoscopic headset in communication with said view generator.

13. The system of claim 11, further comprising a CRT screen in communication with said view generator.

14. The system of claim 10, wherein each storage device is a video cassette recorder and the respective storage medium therefor is a video cassette.