VTV system

• US 7,688,346 B2
• Filed: 09/19/2005
• Issued: 03/30/2010
• Est. Priority Date: 06/25/2001
• Status: Active Grant

• Current Assignee: Angus Duncan Richards
• Original Assignee: Angus Duncan Richards
• Inventors: Richards, Angus Duncan
• Primary Examiner(s): Philippe, Gims S
• Application Number:
  US11/230,173
• Publication Number:
  US 20060082643A1
• Time in Patent Office: 1,653 Days
• Field of Search: 348/36, 348/37, 348/38, 348/39, 348/47, 345/419, 345/474, 364/514
• US Class Current: 348/36
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06T 15/10 Geometric effects
• CPC Class Codes: G06T 2215/08 Gnomonic or central projection
• CPC Class Codes: H04N 21/426 Internal components of the client ; Characteristics thereof H04N5/44 takes precedence
• CPC Class Codes: H04N 7/002 Special television systems not provided for by H04N7/007 - H04N7/18 still pictures via a television channel H04N1/00098
• CPC Class Codes: H04N 7/17318 Direct or substantially direct transmission and handling of requests
• CPC Class Codes: H04N 9/641 Multi-purpose receivers, e.g. for auxiliary information H04N9/642 takes precedence
• CPC Class Codes: H04N 9/8205 involving the multiplexing of an additional signal and the colour video signal

Abstract

The following patent relates to an overall hardware configuration that produces an enhanced spatial television-like viewing experience. Unlike normal television, with this system the viewer is able to control both the viewing direction and relative position of the viewer with respect to the movie action. In addition to a specific hardware configuration, this patent also relates to a new video format which makes possible this virtual reality like experience.

Claims

1. An interactive image capture and display system comprising:

a) an image input means including an array of electronic image capture devices distributed in a horizontal plane such that their fields of view partially overlap and collectively cover a full 360 degrees; and

b) an image storage and playback means compatible with existing television standards;

c) a signal processing means including;

1) a means of producing graphical imagery depicting a panoramic image such that said panoramic image is composed of a plurality of smaller image sections;

2) a means for cropping, distorting and aligning individual images produced by the said image capture devices to produce an overall 360 degree panoramic image with negligible distortion and overlap between the individual image sections and wherein each pixel in the resulting 360 degree panoramic image has the same effective width, where each pixel subtends an equal horizontal angle to the center of said panoramic image;

3) a means for generating an image representing a subset of the said 360 degree panoramic image, whereby the azimuth and elevation of the center of said subset is adjustable by user control;

4) a means for selectively combining and geometrically altering real time imagery from said capture devices and prerecorded imagery to create a composite augmented reality experience;

5) a means for determining the correct location of said image sections within said 360 degree panoramic image utilizing additional information present in the source media;

6) a means for inserting tracking information to describe at least the current orientation of said array of electronic image capture devices into an outgoing video stream;

7) a means for encoding multi-track audio such that it maintains compatibility with standard video storage, playback and transmission systems; and

8) a means for producing orientation-sensitive audio in real-time, utilizing multi-track audio information and controlled by coordinates of a viewport within said panoramic image;

d) an image output means capable of outputting an image in a format compatible with existing television standards;

e) an audio output means capable of outputting at least 2 channels of audio;

f) a display means including at least one display device;

g) a user control means including an input device allowing the user to control said signal processing means; and

h) A tracking means capable of measuring at least azimuth and elevation of said array of electronic image capture devices.

2. The system according to claim 1 further comprising signal processing means for applying distortion correction to the images, wherein each pixel in the resulting 360 degree panoramic image has the same effective height, where each pixel subtends an equal vertical angle to the center of said panoramic image.

3. A system according to claim 1 in which said display means is a conventional television type display device and the user input means is an infrared or radio based manually operated remote control device.

4. A system according to claim 1 in which said display means is a helmet mounted display device and the user input means is an automatic tracking device that calculates at least azimuth and elevation of the user'"'"'s head.

5. A system according to claim 1 which utilizes a modified television protocol comprising a plurality of video fields or frames such that each field or frame includes at least one of graphical data, sound data, and control information, wherein the signal from said image playback means is compatible with at least one widely accepted television standard.

6. A system according to claim 5 wherein said modified television protocol further comprises, within one or more scan lines of a standard video image, additional coded data defining control parameters and image manipulation data for a signal processing means.

7. A system according to claim 5 wherein said graphical data comprises sections of said 360 degree panoramic image.

8. A system according to claim 5 further comprising, within one or more scan lines of a standard video image, additional coded data providing information defining the placement position of image sections within said 360 degree panoramic image.

9. A system according to claim 5 further comprising within one or more scan lines of a standard video image, additional coded data providing information for the generation of four or more real-time audio tracks.

11. A system according to claim 5 further comprising, within one or more scan lines of a standard video image, additional coded data which provides information as to absolute orientation and X-Y-Z position of said capture device array.

12. A system according to claim 1 further comprising

a) means for mathematically combining information about azimuth and elevation of a viewer;

and

b) means for encoding multi-track audio for use with standard video storage and transmission systems such that the combined information can be subsequently decoded by specific hardware to produce a left and right audio channel with spatially correct three-dimensional audio for the left and right ears of a viewer.

13. A system according to claim 1 further comprising means for varying angular field of view of said viewport within said panoramic image responsive to runtime user control.

14. A system according to claim 1 further comprising means for varying the position of a viewpoint within a three-dimensional virtual space responsive to runtime user control.

15. A system according to claim 1 further comprising:

a) a tracking device for continuously calculating a viewer'"'"'s physical position; and

b) means for varying the position of a viewpoint within a three-dimensional virtual space responsive to said position.

16. A system according to claim 1 further comprising means for providing orientation-sensitive audio in real-time, controlled by the direction of the viewer'"'"'s head.

17. A system according to claim 1 further comprising means for providing orientation-sensitive audio in real-time, controlled by coordinates of a viewport within said panoramic image.

18. A system according to claim 1 further comprising means for providing position-sensitive audio in real-time, controlled by the virtual position of a viewpoint within a three-dimensional virtual space.

19. A system according to claim 1 wherein said signal processing means comprises

a) one or more video digitizing modules;

b) one or more memory areas selected from the group consisting of ARM, VRM, and TM;

c) digital processing means for

1) altering address mapping of data held in at least one of ARM and VRM so as to effectively move graphical information from one location to another therein; and

2) mathematically combining and altering data from both a source location and a destination location, thereby achieving the functions of compositing and transformation; and

d) one or more video generation modules.

20. A system according to claim 19 wherein said ARM is mapped to occupy a smaller vertical field of view than said VRM and said TM, thereby reducing the amount of data required for the generation of a high-quality image.

21. A system according to claim 19 further comprising means for mapping ARM, VRM, and TM at different resolutions, whereby pixels in each memory region can represent different degrees of angular deviation.

22. A system according to claim 1 further comprising

a) means for displaying imagery;

b) means for placing said real-time video imagery into ARM and source information from said video playback means into VRM; and

c) means for combining imagery from ARM and VRM according to a pattern of data held in TM into a composite image before display.

23. A system according to claim 1 further comprising:

a) means for displaying imagery;

b) means for placing source information from said video playback means into ARM and VRM; and

c) means for combining imagery from ARM and VRM according to a translation map included in the source media.

24. The system according to claim 1 further comprising

a) means for displaying imagery;

b) means for placing source information from said video playback means into ARM and VRM; and

c) means for combining imagery from ARM and VRM in accordance with a geometric interpretation of said real-time video imagery.

25. A system according to claim 1 further comprising signal processing means for inserting identification information to describe the location of individual image sections that comprise said 360 degree panoramic image into said outgoing video stream.

26. A system according to claim 1 wherein said tracking information also describes the current spatial position of said array of electronic image capture devices into said outgoing video stream.

27. A system according to claim 1 whereby said signal processing means utilizes data received from said away of electronic image capture devices and, by performing a series of image analysis processes, calculates changes in the orientation of said array of electronic image capture devices.

28. A system according to claim 1 whereby said signal processing means utilizes data received from said away of electronic image capture devices and, by performing a series of image analysis processes, calculates changes in the position of said away of electronic image capture devices.

29. A system according to claim 1 wherein said tracking means comprises

a) a plurality of reflective targets placed at predetermined coordinates;

b) a plurality of on-axis light sources strobed in synchronization with the capture rate of said array of electronic image capture devices; and

c) means for computing absolute angular and spatial data based on said predetermined coordinates and relative angular and spatial data determined by said array of electronic image capture devices.

30. A system according to claim 29 further comprising a plurality of color filters positioned over said reflective targets, whereby the ability of said system to correctly identify and maintain tracking of said reflective targets is improved.

31. A system according to claim 29 wherein said light sources are color-controllable, whereby the ability of the system to correctly identify and maintain tracking of said reflective targets is improved.

32. A system according to claim 1 wherein said tracking means incorporates active beacons which utilize at least one of pulse timing and color of light to transmit spatial coordinates of each beacon to said array of electronic image capture devices, whereby relative angular and spatial data can be determined by said array of electronic image capture devices and converted into absolute angular and spatial data.

10. A system according to 5 further comprising, within one or more scan lines of a standard video image, additional coded data providing

a) audio information for generation of four or more real-time audio tracks;

and

b) data descriptive of a number of employed audio tracks, an employed audio data format, an employed audio sampling rate, and track synchronization, whereby said signal processing means can decode the audio information into position and orientation sensitive sound.