Telepresence using panoramic imaging and directional sound and motion

• US 20060028542A1
• Filed: 08/01/2005
• Published: 02/09/2006
• Est. Priority Date: 07/30/2004
• Status: Active Grant

• Current Assignee: 360ai Solutions LLC
• Original Assignee: EyeSee360, Inc.
• Inventors: Herman, Herman, Singh, Sanjiv, Rondinelli, Michael, Stentz, Anthony
• Granted Patent: US 8,730,322 B2
• Time in Patent Office: Days
• US Class Current: 348/36
• CPC Class Codes: H04N 21/41415 involving a public display, viewable by several users in a public space outside their home, e.g. movie theatre, information kiosk
• CPC Class Codes: H04N 21/43074 of additional data with content streams on the same device, e.g. of EPG data or interactive icon with a TV program
• CPC Class Codes: H04N 21/47217 for controlling playback functions for recorded or on-demand content, e.g. using progress bars, mode or play-point indicators or bookmarks specific graphical features in visual interfaces H04N21/4312
• CPC Class Codes: H04N 23/698 for achieving an enlarged field of view, e.g. panoramic image capture
• CPC Class Codes: H04N 5/04 Synchronising for television systems using pulse code modulation H04N7/56
• CPC Class Codes: H04N 5/144 Movement detection for video coding H04N19/503; analysis of motion in general G06T7/20
• CPC Class Codes: H04N 5/76 Television signal recording
• CPC Class Codes: H04N 5/772 the recording apparatus and the television camera being placed in the same enclosure
• CPC Class Codes: H04N 7/18 Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast
• CPC Class Codes: H04N 7/181 for receiving images from a plurality of remote sources
• CPC Class Codes: H04N 9/89 Time-base error compensation

Abstract

An imaging system comprising a panoramic visual image display, an associated directional sound playback device, and an associated motion reproduction device is disclosed. The imaging system conveys visual, sound and motion information related to a particular viewing direction to provide a realistic experience for the viewer. The imaging system can also comprise a panoramic visual image recording device capable of recording panoramic images, an associated directional sound capturing device capable of recording sound, and an associated directional motion capturing device capable of recording motion. Recorded panoramic images, sound and motion can be synchronously recorded to a common time code for simultaneous playback.

Claims

1. An apparatus comprising:

a panoramic visual image capturing device capable of recording panoramic visual images;

a sound capturing device capable of recording sound associated with viewing directions for the visual images; and

a motion capturing device capable of recording motion associated with viewing directions for the visual images.

2. The apparatus of claim 1, further comprising at least one additional sound capturing device positioned remotely from the panoramic visual image capturing device.

3. The apparatus of claim 1, further comprising at least one additional motion capturing device positioned remotely from the panoramic visual image capturing device.

4. The apparatus of claim 1, further comprising a processor for processing signals from the sound capturing device and the motion capturing device to determine a location of a source of sound and motion.

5. A method comprising the steps of:

recording panoramic visual images;

recording sound associated with viewing directions for the visual images; and

recording motion associated with viewing directions for the visual images.

6. The method of claim 5, wherein the sound associated with viewing directions for the visual images is recorded at a position remote from a visual image capturing device.

7. The method of claim 5, wherein the motion associated with viewing directions for the visual images is recorded at a position remote from a visual image capturing device.

8. The method of claim 5, wherein the sound associated with viewing directions for the visual images is recorded on two or more channels, and a temporal and spatial correspondence is established between the panoramic visual image and the recorded sound.

9. The method of claim 5, wherein at least one of the panoramic visual images, the recorded sound and/or the recorded motion is synthetic.

10. The method of claim 5, wherein a spatial correspondence between the panoramic images and recorded sound is achieved by localizing sources of the sound and embedding localized information in a data stream that contains both panoramic image data and sound data.

11. The method of claim 10, wherein the sources of sound are localized by measuring both the loudness and phase of the sound, and by computing an estimate of the location of the sources.

12. The method of claim 5, wherein the step of recording motion associated with viewing directions for the visual images includes the steps of:

sensing motion in different areas of an environment; and

coupling the sensed motion to the panoramic visual images.

13. The method of claim 5, wherein the sound and motion are recorded over time and synchronized to a common time code.

14. The method of claim 13, wherein the common time code is generated and stamped on digital samples of the panoramic visual image, the recorded sound, and the recorded motion.

15. An image playback system comprising:

a panoramic visual image display for displaying panoramic visual images;

a sound playback device for reproducing sound associated with viewing directions for the panoramic visual images; and

a motion reproduction device for reproducing motion associated with viewing directions for the panoramic visual images, wherein the panoramic visual images, sound and motion are synchronously reproduced.

16. An image playback method comprising the steps of:

displaying panoramic visual images;

reproducing sound associated with viewing directions for the panoramic visual images; and

reproducing motion associated with viewing directions for the panoramic visual images, wherein the panoramic visual images, sound and motion are synchronously reproduced.

17. The image playback method of claim 16, further comprising the step of:

changing the viewing directions for the panoramic visual image by utilizing a pre-determined computer program, or by a user manipulating a haptic device.

18. The image playback method of claim 16, wherein as the viewing direction changes, the strength of the sound and the motion parameters vary as a function of the angle between the viewing direction and a direction to a source of the sound and motion.

19. The image playback method of claim 16, wherein the sound is modulated according to the alignment of a sound source with respect to a current viewing direction.

20. The image playback method of claim 16, wherein the phase and loudness of the sound is modulated to emulate a position of a sound source with respect to a current viewing direction.

21. The image playback method of claim 16, wherein a user chooses a direction of view that defines an offset from a reference frame, and the sound and motion are played back in a modified manner based on the offset from the panoramic visual image reference frame.