VTV system

• US 7,719,563 B2
• Filed: 12/11/2003
• Issued: 05/18/2010
• Est. Priority Date: 12/11/2003
• Status: Active Grant

• Current Assignee: Angus Richards
• Original Assignee: Angus Richards
• Inventors: Richards, Angus
• Primary Examiner(s): Philippe, Gims S
• Application Number:
  US10/736,582
• Publication Number:
  US 20050128286A1
• Time in Patent Office: 2,350 Days
• Field of Search: 348/36, 348/139, 348/140, 348/157, 702/153, 702/167
• US Class Current: 348/36
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: H04N 21/23614 Multiplexing of additional data and video streams arrangements for simultaneous broadcast of plural pieces of information H04H20/28
• CPC Class Codes: H04N 21/2368 Multiplexing of audio and video streams
• CPC Class Codes: H04N 21/242 Synchronization processes, e.g. processing of PCR [Program Clock References] arrangements for synchronising broadcast or distribution via plural systems in broadcast distribution systems H04H20/18
• CPC Class Codes: H04N 21/4223 Cameras H04N23/00 takes precedence
• CPC Class Codes: H04N 21/42653 for processing graphics
• CPC Class Codes: H04N 21/4305 Synchronising client clock from received content stream, e.g. locking decoder clock with encoder clock, extraction of the PCR packets arrangements for synchronising receiver with transmitter by comparing receiver clock with transmitter clock H04L7/0012; arrangements for synchronising receiver with transmitter wherein the receiver takes measures against momentary loss of synchronisation H04L7/0083
• CPC Class Codes: H04N 21/43072 of multiple content streams on the same device
• CPC Class Codes: H04N 21/431 Generation of visual interfaces for content selection or interaction; Content or additional data rendering
• CPC Class Codes: H04N 21/4341 Demultiplexing of audio and video streams
• CPC Class Codes: H04N 21/4348 Demultiplexing of additional data and video streams
• CPC Class Codes: H04N 21/44218 Detecting physical presence or behaviour of the user, e.g. using sensors to detect if the user is leaving the room or changes his face expression during a TV program methods or arrangements for recognising human body or animal bodies or body parts G06V40/10; methods or arrangements for acquiring or recognising human faces, facial parts, facial sketches, facial expressions G06V40/16; methods or arrangements for recognising movements or behaviour G06V40/20; arrangements for identifying users in broadcast systems H04H60/45
• CPC Class Codes: H04N 21/8106 involving special audio data, e.g. different tracks for different languages
• CPC Class Codes: H04N 5/272 Means for inserting a foreground image in a background image, i.e. inlay, outlay
• CPC Class Codes: H04N 5/602 for digital sound signals
• CPC Class Codes: H04N 5/607 for more than one sound signal, e.g. stereo, multilanguages H04N5/602 takes precedence
• CPC Class Codes: H04N 9/3147 Multi-projection systems displays in general H04N9/12; video walls G06F3/1446, G09G2300/026

Abstract

An electronic system that produces an enhanced spatial television-like audio-visual experience is disclosed. Unlike normal television, the system enables the viewer to control both the viewing direction and relative position of the viewpoint with respect to the movie action. In addition to a specific hardware configuration, this patent also relates to a new video format which makes possible this virtual reality like experience including a system by which multi-channel audio can be recorded in addition to visual information onto a single data channel which is compatible with existing television and audio-visual standards and an optical tracking system to provide both angular and spatial position information to control the display of said imagery and spatial audio.

Claims

1. An optical tracking and processing system configured for communication with a visual display device, the visual display device being configured to display a virtual scene, the virtual scene being visually perceptible by at least one viewer, said system comprising:

a plurality of reference optical data sources each located at respective of a plurality of stationary reference locations spaced apart in a real-world physical environment, the real-world physical environment being suitable to accommodate a person for movement relative to the plurality of reference optical data sources, the plurality of reference optical data sources providing a set of reference optical data;

a portable housing adapted to be supported by a person, the portable housing when supported by a person being movable relative to the plurality of reference optical data sources upon motion of the person in the real-world physical environment;

a video capture device supported by the portable housing for movement in common with the portable housing relative to the plurality of reference optical data sources, the video capture device being configured to collect optical data, the video capture device providing collected video data, the collected video data including the set of reference optical data;

at least one portable image analysis processor supported by the portable housing, the at least one portable image analysis processor being in communication with the video capture device, the at least one portable image analysis processor being configured to be provided on at least a near real-time basis at least the set of reference optical data, the at least one portable image analysis processor being configured to perform at least one series of image analysis preprocessing steps using at least the set of reference optical data, the at least one portable image analysis processor providing on at least a near real-time basis at least one set of analyzed optical data, the at least one set of analyzed optical data including real-world tracking information, the real-world tracking information corresponding to movement of the video capture device relative to the plurality of reference optical data sources,

a wireless communication link, the wireless communication link including a portable wireless transmission unit supported at the portable housing, the wireless transmission unit being in communication with the at least one image analysis processor to receive the at least one set of analyzed optical data, the wireless transmission unit being operable to transmit wireless transmissions, the wireless transmissions conveying from the wireless transmission unit according to a wireless communication protocol the at least one set of analyzed optical data, the wireless communication link including a wireless receiving unit spaced apart from the wireless transmission unit, the wireless receiving unit being configured to receive the wireless transmissions, the wireless transmissions conveying to the wireless receiving unit on at least a near real-time basis the at least one set of analyzed optical data; and

a stationary processing unit in communication with the wireless receiver to receive the at least one set of analyzed optical data, the stationary processing unit including at least one stationary processor configured to perform at least one series of processing steps using the at least one set of analyzed optical data, processing of the at least one set of analyzed optical data providing on at least a near real-time basis virtual scene tracking information, the stationary processing unit being configured to provide for communication to the visual display device virtual scene information, the virtual scene information having incorporated therein the virtual scene tracking information, the virtual scene tracking information relating to at least one changeable characteristic of the virtual scene, the at least one changeable characteristic of the virtual scene being capable of change on at least a near real-time basis in relation to the real-world tracking information.

2. The system of claim 1 and further comprising at least one of the following:

a) wherein the plurality of reference optical data sources is selected from the following;

a plurality of high contrast passive targets each placed at respective of the fixed reference locations, and

a plurality of fixed intensity light sources;

whereby relative angular and spatial data can be determined.

3. The system of claim 1 further comprising:

wherein the plurality of reference optical data sources includes a plurality of retroreflective targets each placed at respective of the fixed reference locations.

4. The system of claim 3 wherein the light sources are color-controllable, whereby the ability of the system to correctly identify and maintain tracking of the individual retroflective targets is improved.

5. The system of claim 1 and further comprising:

wherein the plurality of reference optical data sources includes a plurality of on-axis light sources strobed in synchronization with a capture rate of the video capture device; and

a processor configured to compute absolute angular and spatial data based on the fixed reference locations and relative angular and spatial data collected by the video capture device.

6. The system of claim 5 wherein the light sources are color-controllable, whereby the ability of the system to correctly identify and maintain tracking of the individual retroflective targets is further improved.

7. The system of claim 1 further comprising:

(a) a plurality of controllable light sources synchronized with a capture rate of the video capture device; and

(b) means for utilizing at least one of pulse timing and color of light to transmits spatial coordinates of each beacon to the video capture device;

whereby relative angular and spatial data can be determined by the device and converted into absolute angular and spatial data.

8. The system of claim 1 further comprising a plurality of bi-directional infrared beacons for communicating a unique ID code with the video capture device.

9. The system of claim 1 further comprising:

at least one gravimetric sensor supported by the housing, the at least one gravimetric sensor being adapted to provide vertical orientation tracking information relating to the housing.

10. The system of claim 1 further comprising a means of establishing at least one of the following:

an arbitrary reference orientation,

an arbitrary reference position, and

a combination of both an arbitrary reference orientation and an arbitrary reference position of the portable housing so as to create a reference point from which changes of the same can be compared.