Internet system for virtual telepresence

• US 6,573,912 B1
• Filed: 11/07/2000
• Issued: 06/03/2003
• Est. Priority Date: 11/07/2000
• Status: Expired due to Fees

• Current Assignee: Zaxel Systems, Inc.
• Original Assignee: Zaxel Systems, Inc.
• Inventors: Suzuki, Norihisa, Kanade, Takeo
• Primary Examiner(s): Jankus, Almis R.
• Application Number:
  US09/708,293
• Time in Patent Office: 938 Days
• Field of Search: 382/154, 345/419, 345/420, 345/424, 345/426, 345/427, 345/582, 345/619, 345/629, 345/630, 345/632, 345/633, 345/473, 345/474, 345/716, 345/719, 345/720, 345/721, 345/722, 345/738, 345/744, 345/747, 345/751, 345/756, 345/757
• US Class Current: 715/757
• CPC Class Codes: G06T 17/10 Constructive solid geometry [CSG] using solid primitives, e.g. cylinders, cubes
• CPC Class Codes: G06T 2207/10012 Stereo images
• CPC Class Codes: G06T 7/55 from multiple images
• CPC Class Codes: G06T 7/564 from contours
• CPC Class Codes: H04N 13/194 Transmission of image signals
• CPC Class Codes: H04N 13/243 using three or more 2D image sensors
• CPC Class Codes: H04N 19/597 specially adapted for multi-view video sequence encoding
• CPC Class Codes: H04N 2013/0081 Depth or disparity estimation from stereoscopic image signals
• CPC Class Codes: H04N 2013/0092 Image segmentation from stereoscopic image signals

Abstract

An interactive virtual telepresence system allows an Internet user to view three-dimensional objects from any perspective in a studio at a remote server. The system includes several video cameras trained on the three-dimensional objects and a computer that calculates the voxel-representation of the solid. The server broadcasts each camera view and voxel information. A user selects the viewing perspective desired, and the computer at the user'"'"'s site receives the nearest camera views to the requested perspective and voxel information for the interpolation and renders the selected view.

Claims

1. An Internet system for virtual telepresence, comprising:

a plurality of video cameras for monitoring a scene, and wherein each provides a respective real-perspective video stream from a unique viewpoint;

a volume-element processor connected to each of the plurality of video cameras, and for providing a calculation of a voxel database that represents a 3D image in said scene; and

a network server connected to receive each said respective real-perspective video stream from the plurality of video cameras, and connected to receive said voxel database from the volume-element processor;

wherein, a particular rendering of a novel perspective is left to be done by a network client and an information transfer for such is provided by the network server.

2. The system of claim 1, further comprising:

a network client that can be provided with a novel perspective selection by a user, and that provides for a transmission of information to the network server to send less than all said respective real-perspective video streams from the plurality of video cameras.

3. The system of claim 2;

wherein;

the network client is provided with a novel perspective selection N that is forwarded over the Internet to the network server; and

the network server is such that it responds to the particular network client with only those respective real-perspective video streams from the plurality of video cameras that are needed to render said novel perspective selection N.

4. The system of claim 2, wherein:

the network client is provided, with a novel perspective selection N that is forwarded over the Internet to the network server; and

the network server is such that it responds to the particular network client with only a single real-perspective video stream from the plurality of video cameras that can best be used to render a viewpoint represented by said novel perspective selection N.

5. The system of claim 2, further comprising:

a renderor connected to the network client for generating a viewpoint represented by said novel perspective selection.

6. The system of claim 2, wherein:

the network client is provided with a data compressor for transmitting said voxel database over the Internet; and

the network server includes a matching data decompressor for receiving said voxel database over the Internet.

7. The system of claim 1, further comprising:

a plurality of video capture devices each respectively associated with the plurality of video cameras and providing for said real-perspective video streams.

8. The system of claim 7, further comprising:

a plurality of background subtractors each respectively associated with the plurality of video cameras, and providing for said real-perspective video streams.

9. The system of claim 8, further comprising:

a silhouette processor connected to the background subtractors associated with the plurality of video cameras, and providing for a silhouette solution representing a 3D object in said scene.

10. The system of claim 9, further comprising:

an intersection processor between the silhouette processor and the volume-element processor.

11. A virtual telepresence system for use on the Internet, comprising:

a plurality of video cameras for monitoring a scene, and wherein each provides a respective real-perspective video stream from a unique viewpoint;

a volume-element processor connected to each of the plurality of video cameras, and for providing a calculation of a voxel database that represents a 3D image in said scene;

a network server connected to receive each said respective real-perspective video stream from the plurality of video cameras, and connected to receive said voxel database from the volume-element processor, and is such that it responds to the particular network client with only those respective real-perspective video streams from the plurality of video cameras that are needed to render said novel perspective selection N, and such that it responds to the particular network client with only a single real-perspective video stream from the plurality of video cameras that can best be used to render a viewpoint represented by said novel perspective selection N;

a network client that can be provided with a novel perspective selection by a user, and that provides for a transmission of information to the network server to send less than all said respective real-perspective video streams from the plurality of video cameras, and is provided with a novel perspective selection N that is forwarded over the Internet to the network server, and is provided with a novel perspective selection N that is forwarded over the Internet to the network server;

a renderor connected to the network client for generating a viewpoint represented by said novel perspective selection;

a data compressor included in the network client for transmitting said voxel database over the Internet; and

a matching data decompressor included in the network server far receiving said voxel database over the Internet;

wherein, a particular rendering of a novel perspective is left to be done by the network client and an information transfer for such is provided by the network server in response to a request.

12. A method for communicating 3D imaging information over the Internet, the method comprising the steps of:

capturing several video streams in parallel from a plurality of strategically placed cameras around a scene;

computing a voxel database for a 3D object within said scene;

transmitting said voxel database and at least one of said several video streams over the Internet from a network server; and

rendering a novel perspective at a network client from a reception of said voxel database and at least one of said several video streams.

13. The method of claim 12, further comprising the steps of:

accepting a user selection of a novel perspective at said network client;

communicating said novel-perspective user selection to said network server; and

eliminating a transmission of one or more of said several video streams from transmission aver the Internet by said network server to a particular network client associated with said novel-perspective user selection.

14. A method for communicating 3D imaging information over the Internet, the method comprising the steps of:

capturing several video streams in parallel from a plurality of strategically placed cameras around a scene;

computing a voxel database for a 3D object within said scene;

compressing said voxel database to produce a compressed voxel datastream;

transmitting said compressed voxel datastream and at least one of said several video streams aver the Internet from a network server;

decompressing at a network client said compressed voxel datastream to produce a voxel database;

rendering a novel perspective at said network client from said voxel database and a reception, of at least one of said several video streams;

accepting a user selection of a novel perspective at said network client;

communicating said novel-perspective user selection to said network server; and

eliminating a transmission of one or more of said several video streams from transmission aver the Internet by said network server to a particular network client associated with said novel-perspective user selection.

15. An Internet system for virtual telepresence, comprising:

a plurality of video cameras far monitoring a scene, and wherein each provides a respective real-perspective video stream from a unique viewpoint;

a volume-element processor connected to each of the plurality of video cameras, and far providing a calculation of a voxel database that represents a 3D image in said scene;

a network server connected to receive each said respective real-perspective video stream from the plurality of video cameras, and connected to receive said voxel database from the volume-element processor; and

a background subtractor for eliminating any background information from said video streams, and such that encoding is limited to any foreground objects;

wherein, a particular rendering of a navel perspective is left to be done by a network client and an information transfer far such is provided by the network server.

16. The system of claim 15, further comprising:

a data compressor providing for a gassy compression of said video streams to reduce network bandwidth requirements.

17. A method far communicating 3D imaging information over the Internet, the method comprising the steps of;

capturing several video streams in parallel from a plurality of strategically placed cameras around a scene;

computing a voxel database for a 3D object within said scene;

subtracting any background information represented to said scene;

transmitting said voxel database and at (east one of said several video streams aver the Internet from a network server; and

rendering a novel perspective at a network client from a reception of said voxel database arid at least one of said several video streams.

18. The method of claim 17, further comprising the step of:

compressing said video streams with a lossy compressor to reduce the bandwidth requirements of a network.