Virtual reality television system

• US 5,714,997 A
• Filed: 01/06/1995
• Issued: 02/03/1998
• Est. Priority Date: 01/06/1995
• Status: Expired due to Fees

• Current Assignee: Anderson, David P.
• Original Assignee: Anderson, David P.
• Inventors: Anderson, David P.
• Primary Examiner(s): Au, Amelia
• Assistant Examiner(s): Le, Vu
• Application Number:
  US08/369,885
• Time in Patent Office: 1,124 Days
• Field of Search: 348/39, 348/36, 348/40, 348/42, 348/48, 348/51, 348/578, 348/580, 348/583, 348/586, 395/125
• US Class Current: 348/39
• CPC Class Codes: A63F 2300/69 Involving elements of the real world in the game world, e.g. measurement in live races, real video
• CPC Class Codes: A63F 2300/8011 Ball
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 2027/0198 System for aligning or maintaining alignment of an image in a predetermined direction
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G06T 15/10 Geometric effects
• CPC Class Codes: G06T 2207/10012 Stereo images
• CPC Class Codes: H04H 20/38 Arrangements for distribution where lower stations, e.g. receivers, interact with the broadcast
• CPC Class Codes: H04H 60/07 characterised by processes or methods for the generation
• CPC Class Codes: H04N 13/117 the virtual viewpoint locations being selected by the viewers or determined by viewer tracking
• CPC Class Codes: H04N 13/15 for colour aspects of image signals
• CPC Class Codes: H04N 13/161 Encoding, multiplexing or demultiplexing different image signal components for multi-view video sequence encoding H04N19/597
• CPC Class Codes: H04N 13/167 Synchronising or controlling image signals
• CPC Class Codes: H04N 13/178 Metadata, e.g. disparity information
• CPC Class Codes: H04N 13/189 Recording image signals; Reproducing recorded image signals
• CPC Class Codes: H04N 13/194 Transmission of image signals
• CPC Class Codes: H04N 13/239 using two 2D image sensors having a relative position equal to or related to the interocular distance H04N13/243 takes precedence
• CPC Class Codes: H04N 13/243 using three or more 2D image sensors
• CPC Class Codes: H04N 13/246 Calibration of cameras
• CPC Class Codes: H04N 13/257 Colour aspects
• CPC Class Codes: H04N 13/279 : the virtual viewpoint locations being selected by the viewers or determined by tracking
• CPC Class Codes: H04N 13/289 : Switching between monoscopic and stereoscopic modes
• CPC Class Codes: H04N 13/296 : Synchronisation thereof; Control thereof
• CPC Class Codes: H04N 13/344 : with head-mounted left-right displays
• CPC Class Codes: H04N 13/373 : for tracking forward-backward translational head movements, i.e. longitudinal movements
• CPC Class Codes: H04N 13/376 : for tracking left-right translational head movements, i.e. lateral movements
• CPC Class Codes: H04N 13/38 : for tracking vertical translational head movements
• CPC Class Codes: H04N 13/383 : for tracking with gaze detection, i.e. detecting the lines of sight of the viewer's eyes
• CPC Class Codes: H04N 13/398 : Synchronisation thereof; Control thereof
• CPC Class Codes: H04N 19/27 : involving both synthetic and natural picture components, e.g. synthetic natural hybrid coding [SNHC]
• CPC Class Codes: H04N 19/597 : specially adapted for multi-view video sequence encoding
• CPC Class Codes: H04N 2013/0081 : Depth or disparity estimation from stereoscopic image signals
• CPC Class Codes: H04N 2013/0096 : Synchronisation or controlling aspects
• CPC Class Codes: H04N 7/002 : Special television systems not provided for by H04N7/007 - H04N7/18 still pictures via a television channel H04N1/00098
• CPC Class Codes: H04R 1/406 : microphones
• CPC Class Codes: H04R 2201/401 : 2D or 3D arrays of transducers
• CPC Class Codes: H04R 2420/01 : Input selection or mixing for amplifiers or loudspeakers for hearing aids H04R25/43
• CPC Class Codes: H04R 2430/01 : Aspects of volume control, not necessarily automatic, in sound systems
• CPC Class Codes: H04R 2430/23 : Direction finding using a sum-delay beam-former
• CPC Class Codes: H04S 2400/11 : Positioning of individual sound objects, e.g. moving airplane, within a sound field H04S2420/13 takes precedence
• CPC Class Codes: H04S 2400/13 : Aspects of volume control, not necessarily automatic, in stereophonic sound systems
• CPC Class Codes: H04S 2400/15 : Aspects of sound capture and related signal processing for recording or reproduction
• CPC Class Codes: H04S 2420/01 : Enhancing the perception of the sound image or of the spatial distribution using head related transfer functions [HRTF's] or equivalents thereof, e.g. interaural time difference [ITD] or interaural level difference [ILD]
• CPC Class Codes: H04S 2420/03 : Application of parametric coding in stereophonic audio systems
• CPC Class Codes: H04S 7/304 : For headphones

Abstract

A virtual reality television system includes a video capture and encoding system for capturing and encoding the surface geometries and colors of objects and their background as the objects interact during a live action event taking place in a live action space; an audio capture and encoding system for capturing and encoding spatial sound sources emanating from the live action space as a result of the objects interacting therein, as well as ambient sounds emanating from a periphery area around the live action space and commentators'"'"' voices; and a rendering system responsive to viewer manipulated input devices for rendering a visual and auditory experience of the live action event from the perspective of a viewer selected virtual viewpoint related to the live action space.

Claims

1. An interactive television system providing viewer selectable virtual positions including all points within a three-dimensional space for experiencing an event occurring in said three-dimensional space, comprising:

a capture and encoding subsystem including

means for generating video signals from different viewpoints of said objects within said live-action space such that said video signals communicate information indicative of the locations and three-dimensional shapes of said objects, and

means for processing said video signals to generate encoded data including the locations and three-dimensional shapes of said objects; and

a rendering subsystem including

means responsive to physical actions initiated by a viewer for generating at least one control signal indicative of a virtual position in said three-dimensional space;

means for receiving said at least one control signal, said encoded data, and audio information associated with said event, and generating in response thereof, display signals and audio signals relative to said virtual position;

means responsive to said display signals for providing a simulated viewing experience relative to said virtual position to the viewer; and

means responsive to said audio signals for providing a simulated hearing experience relative to said virtual position to the viewer.

2. The interactive television system as recited in claim 1, wherein said control signal generating means comprises means responsive to physical actions initiated by a viewer for generating at least one control signal indicative of a virtual position and orientation in said three-dimensional space;

wherein said display and audio signal generating means comprises means for receiving said at least one control signal, encoded data, and audio information associated with said event, and generating in response thereof, display signals and audio signals relative to said virtual position and orientation;

wherein said viewing experience providing means comprises means responsive to said display signals for providing a simulated viewing experience relative to said virtual position and orientation to the viewer; and

wherein said hearing experience providing means comprises means responsive to said audio signals for providing a simulated hearing experience relative to said virtual position and orientation to the viewer.

3. The interactive television system as recited in claim 2, wherein said display and audio signal generating means comprises means for receiving said at least one control signal, encoded data, and audio information associated with said event, and generating in response thereof, right and left eye video signals and right and left ear audio signals relative to said virtual position and orientation;

wherein said viewing experience providing means comprises means responsive to said right and left eye video signals for displaying right and left eye images, providing a three-dimensional viewing experience relative to said virtual position and orientation, to the viewer; and

wherein said hearing experience providing means comprises means responsive to said right and left ear audio signals for generating right and left ear sounds, simulating a three-dimensional hearing experience relative to said virtual position and orientation, for the viewer.

4. The interactive television system as recited in claim 1, wherein said receiving means receives a broadcasted signal including said received video and audio information.

5. The interactive television system as recited in claim 1, wherein said at least one control signal generating means comprises a joystick manipulated by the viewer such that left and right velocities are controlled by left and right movement of the joystick, forward and backward velocities are controlled by forward and backward movement of the joystick, and up and down velocities are controlled by up and down movement of the joystick.

6. The interactive television system as recited in claim 5, wherein said joystick is further manipulated by the viewer such that viewer orientation in the yaw axis is controlled by rotary movement of the joystick.