Panoramic image based virtual reality/telepresence audio-visual system and method

• US 5,495,576 A
• Filed: 01/11/1993
• Issued: 02/27/1996
• Est. Priority Date: 01/11/1993
• Status: Expired due to Term

• Current Assignee: Intellectual Ventures Fund 59 LLC (Intellectual Ventures LLC)
• Original Assignee: Kurtis J. Ritchey
• Inventors: Ritchey, Kurtis J.
• Primary Examiner(s): Zimmerman, Mark K.
• Application Number:
  US08/002,582
• Time in Patent Office: 1,142 Days
• Field of Search: 395/119, 395/125, 395/130, 395/154, 395/50, 395/902, 395/2.55, 395/2.6, 395/2.79, 395/2.85, 345/139, 345/158, 345/184
• US Class Current: 345/420
• CPC Class Codes: A63F 2300/1093 using visible light
• CPC Class Codes: A63F 2300/6676 by dedicated player input
• CPC Class Codes: A63F 2300/69 Involving elements of the real world in the game world, e.g. measurement in live races, real video
• CPC Class Codes: G06T 17/00 Three dimensional [3D] modelling, e.g. data description of 3D objects
• CPC Class Codes: G06T 2207/10012 Stereo images
• CPC Class Codes: H04N 13/15 for colour aspects of image signals
• CPC Class Codes: H04N 13/156 Mixing image signals
• CPC Class Codes: H04N 13/161 Encoding, multiplexing or demultiplexing different image signal components for multi-view video sequence encoding H04N19/597
• CPC Class Codes: H04N 13/189 Recording image signals; Reproducing recorded image signals
• CPC Class Codes: H04N 13/194 Transmission of image signals
• CPC Class Codes: H04N 13/239 using two 2D image sensors having a relative position equal to or related to the interocular distance H04N13/243 takes precedence
• CPC Class Codes: H04N 13/243 using three or more 2D image sensors
• CPC Class Codes: H04N 13/257 Colour aspects
• CPC Class Codes: H04N 13/279 the virtual viewpoint locations being selected by the viewers or determined by tracking
• CPC Class Codes: H04N 13/289 Switching between monoscopic and stereoscopic modes
• CPC Class Codes: H04N 13/296 Synchronisation thereof; Control thereof
• CPC Class Codes: H04N 13/305 using lenticular lenses, e.g. arrangements of cylindrical lenses
• CPC Class Codes: H04N 13/324 Colour aspects
• CPC Class Codes: H04N 13/337 using polarisation multiplexing
• CPC Class Codes: H04N 13/344 with head-mounted left-right displays
• CPC Class Codes: H04N 13/361 : Reproducing mixed stereoscopic images; Reproducing mixed monoscopic and stereoscopic images, e.g. a stereoscopic image overlay window on a monoscopic image background
• CPC Class Codes: H04N 13/363 : using image projection screens volumetric display H04N13/388
• CPC Class Codes: H04N 13/366 : using viewer tracking
• CPC Class Codes: H04N 13/383 : for tracking with gaze detection, i.e. detecting the lines of sight of the viewer's eyes
• CPC Class Codes: H04N 19/597 : specially adapted for multi-view video sequence encoding
• CPC Class Codes: H04N 2013/0088 : Synthesising a monoscopic image signal from stereoscopic images, e.g. synthesising a panoramic or high resolution monoscopic image

Abstract

An improved panoramic image based virtual reality/telepresence audio-visual system and method includes panoramic three-dimensional input devices, a computer processor, and a panoramic audio-visual output device. In one embodiment of the system the input devices comprise a sensor assembly including a plurality of positionable radar, camera, and acoustical sensors for recording signatures of all sides of three-dimensional subjects simultaneously. The computer processor integrates the sensor signals, processes signals as a virtual model, updates the model based on participant interaction, and selects and distributes portions of the processed virtual model for presentation on display units and audio speakers. The processor includes participant interactive input devices for instantaneous interaction with the virtual model. The panoramic audio-visual output device includes a head-mounted display or a closed structure having contiguous individual display units mounted in all viewable directions surrounding the participant. Conventional, stereoscopic, autostereoscopic, and holographic display systems are provided to view the panoramic three-dimensional image based model. Computer graphics, artificial intelligence, telecommunications, and vehicle control embodiments of the system are provided.

Claims

1. A display system for virtual interaction with recorded images and comprising:

(a) input means including;

(1) a plurality of positionable sensor means of mutually angular relation to enable substantially continuous coverage by said sensor means of a three-dimensional subject; and

(2) sensor recorder means communicating with said sensor means and operative to store and generate sensor signals representing said subject;

(b) signal processing means communicating with said sensor means and said recorder means, receiving said sensor signals from said recorder means, and operable to texture map virtual images represented by said sensor signals onto a three-dimensional form;

(c) a panoramic audio-visual display assembly means communicating with said signal processing means and enabling display to a viewer of the texture mapped virtual images;

(d) viewer control means communicating with said signal processing means, including at least one interactive input device, and enabling interactive manipulation of said texture mapped virtual images by a viewer thereof operating said interactive input device; and

(e) said signal processing means further including;

(1) a host computer for manipulation of a computer generated world model; and

(2) manipulation software associated with said host computer for assigning action to subjects in said computer generated world model based upon actions by another subject in said computer generated world model.

2. A system according to claim 1 wherein:

(a) the plurality of sensor means are positioned in an inwardly facing mutually angular relation to enable substantially continuous coverage by said sensor means of a given subject.

3. A system according to claim 1 wherein:

(a) the plurality of sensor means are positioned on a housing means in outwardly facing, mutually angular relation to enable substantially spherical coverage by said sensor means.

4. A system according to claim 1 where said sensor means comprises:

(a) a plurality of positionable objective lens means of mutually angular relation to enable substantially continuous coverage by said lens means of a given subject;

(b) light sensitive means optically communicating with said lens means and receiving respective optical images therefrom and generating image signals representing said optical images; and

(c) said sensor recorder means includes image recorder means communicating with said light sensitive means and operative to store said image signals.

5. A system according to claim 1 where said sensor means comprises:

(a) a plurality of microphones positioned in mutually angular relation to enable continuous coverage by said microphones, said microphones providing audio signals representing sounds sensed thereby; and

(b) said sensor recorder means includes audio recorder means communicating with said microphones and operative to store and replay said audio signals.

6. A system according to claim 1 wherein said sensor means includes:

(a) a plurality of three-dimensional digitizing sensor means positioned in mutually angular relation to enable continuous coverage by said digitizer means; and

(b) said sensor recorder means includes three-dimensional digitizer recording means communicating with said digitizing sensor means to store and replay signals representing a three-dimensional representation of said subject.

7. A system according to claim 1 wherein signal processing means includes:

(a) interface apparatus for interfacing video and digital signals to a computer in a telecommunications system to allow a plurality of computer terminals to engage in teleconferencing via a digital data network.

8. A system according to claim 7 including a non-contact sensor array which includes:

(a) a plurality of arrays oriented in an outwardly facing mutually angular relation to enable substantially spherical coverage of a three-dimensional subject.

9. A system according to claim 7 including a non-contact sensor array which includes:

(a) a plurality of arrays oriented in an inwardly facing mutually angular relationship to enable substantially continuous coverage of a three dimensional subject.

10. A system according to claim 1 wherein:

(a) said signal processing means includes;

(1) said host computer including a pixel programmable display generator; and

(2) a video display generator operatively connected to said display assembly means; and

(b) said viewer control means including graphics input means operatively connected to said computer and operated by a viewer;

said computer generating or altering images in response to said graphic input means; and

said display assembly means displaying said generated or altered images.

11. A system according to claim 1 wherein signal processing means comprises:

(a) a plurality of host computers which each process a segment of a panoramic subject for display.

12. A system according to claim 1 wherein:

(a) said signal processing means includes stereoscopic processing means to generate two independent views taken from viewpoints a selected distance apart of a portion of said images mapped onto said three-dimensional form and communicate said views as stereoscopic video signals; and

(b) said display assembly means includes a pair of display units of said assembly means operable to display said signals respectively on said display units as a stereoscopic image to said viewer.

13. A system according to claim 1 wherein said audio-visual display assembly means includes:

(a) an autostereoscopic audio-visual display system in which the participant requires no eye glasses.

14. A system according to claim 1 wherein said audio-visual display assembly means includes:

(a) a holographic display system.

15. A system according to claim 1 wherein the audio-visual display assembly means includes:

(a) a plurality of flat panel display systems.

16. A system according to claim 1 wherein the audio-visual display assembly means includes:

(a) a plurality of cathode ray tubes.

17. A system according to claim 1 wherein the audio-visual display assembly means includes:

(a) a plurality of video projection display units and associated projection screens.

18. A system according to claim 1 wherein the panoramic audio-visual display assembly means includes:

(a) a head mounted display assembly.

19. A system according to claim 1 wherein the panoramic audio-visual display assembly means comprises:

(a) structural support means for securing a plurality of display units about a viewer, supporting a planar floor on which the viewer is situated, and supporting an entry and exit portion of said assembly means;

(b) said plurality of display units being supported by said structural support means such that said display units face inward to the viewer, said display units being operable to display images;

(c) said display units being arranged about a viewer such that a viewer views a respective portion of a composite image in substantially any viewable direction surrounding the viewer;

(d) said planar floor being positioned close to a lower side of said assembly means, said planar floor being transparent and having display units positioned therebelow in such orientations as to enable viewing images displayed on such display units through said floor, and said planar floor supporting said viewer; and

(e) said entry and exit portion of said assembly means being positioned laterally of said viewer.

20. A system according to claim 1 wherein said signal processing means includes image segment circuit means for partitioning an image output by said signal processing means and said audio-visual display assembly means comprises:

(a) a head mounted display assembly receiving a processed video signal which has been processed by said processing means for display on said head mounted display assembly; and

(b) a second audio-visual display assembly including a plurality of display units receiving respective processed video signals representing image segments of a composite image from said segment circuit means; and

(c) each said assembly means being arranged such that a respective viewer views a respective portion of a scene of substantially spherical coverage in any viewable direction.

21. A system according to claim 1 wherein viewer control means includes:

(a) a non-contact participant position sensor system.

22. A system according to claim 1 and including:

(a) a video teleconferencing system interfaced to said signal processing means.

23. A system according to claim 1 wherein the system includes:

(a) a conventional stereophonic radio transmitter which transmits a stereophonically modulated radio signal to a conventional stereophonic radio receiver having conventional stereophonic headphones connected thereto.

24. A system according to claim 1 and including:

(a) view processing means for providing a plurality of independent views of a computer generated world model and transmitting said views as separate video signals to another device for processing or display.

25. A system according to claim 1 wherein the signal processing means includes:

(a) a television production system.

26. A system according to claim 1 wherein the signal processing means includes:

(a) a participant mechanical feedback device, actuated by participant actions and actions by subjects programmed into a virtual model.