Internet system for virtual telepresence

• US 20030231179A1
• Filed: 03/14/2003
• Published: 12/18/2003
• Est. Priority Date: 11/07/2000
• Status: Active Grant

• Current Assignee: Zaxel Systems, Inc.
• Original Assignee: Zaxel Systems, Inc.
• Inventors: Suzuki, Norihisa
• Granted Patent: US 6,864,903 B2
• Time in Patent Office: Days
• US Class Current: 345/423
• CPC Class Codes: G06T 17/10 Constructive solid geometry [CSG] using solid primitives, e.g. cylinders, cubes
• CPC Class Codes: G06T 2207/10012 Stereo images
• CPC Class Codes: G06T 7/55 from multiple images
• CPC Class Codes: G06T 7/564 from contours
• CPC Class Codes: H04N 13/194 Transmission of image signals
• CPC Class Codes: H04N 13/243 using three or more 2D image sensors
• CPC Class Codes: H04N 19/597 specially adapted for multi-view video sequence encoding
• CPC Class Codes: H04N 2013/0081 Depth or disparity estimation from stereoscopic image signals
• CPC Class Codes: H04N 2013/0092 Image segmentation from stereoscopic image signals

Abstract

The present invention is an interactive virtual telepresence system that allows an Internet user to view three-dimensional objects from any perspective in a studio at a remote server. The system includes several video cameras trained on the three-dimensional objects and a computer that calculates the depth information of the solid. The server broadcasts each camera view and depth information for each view. A user selects the viewing perspective desired, and the computer at the user'"'"'s site receives the nearest camera views to the requested perspective and depth information of each of the views for the interpolation and renders the selected view. The invention also applies to virtual viewpoints where a virtual three-dimensional model image is stored on a network server. The server uses virtual cameras that obtain a rendered image and depth information for a particular viewpoint and sends the rendered image and the depth information to a user.

Claims

1. An Internet system for virtual telepresence, comprising:

a plurality of video cameras for monitoring a scene, and wherein each provides a respective real-perspective video stream from a unique viewpoint;

a depth calculation processor connected to each of the plurality of video cameras, and for providing a calculation of depth information for each video camera; and

a network server connected to receive each said respective real-perspective video stream from the plurality of video cameras, and connected to receive said depth information from the depth calculation processor;

wherein a particular rendering of a novel perspective is left to be done by a network client and an information transfer for such is provided by the network server.

2. The system of claim 1, further comprising:

a network client that can be provided with a novel perspective selection by a user, and that provides for a transmission of information to the network server to send less than all said respective real-perspective video streams and associated depth information from the plurality of video cameras.

3. The system of claim 2, wherein the network client is provided with a novel perspective selection N that is forwarded over the Internet to the network server, and wherein the network server is such that it responds to the particular network client with only those respective real-perspective video streams from the plurality of video cameras and associated depth information that are needed to render said novel perspective selection N.

4. The system of claim 2, wherein the network client is provided with a novel perspective selection N that is forwarded over the Internet to the network server, and wherein the network server is such that it responds to the particular network client with only a single real-perspective video stream from the plurality of video cameras and associated depth information that can best be used to render a viewpoint represented by said novel perspective selection N.

5. The system of claim 2, further comprising:

a renderor connected to the network client for generating a viewpoint represented by said novel perspective selection.

6. The system of claim 1, further comprising:

a plurality of video capture devices each respectively associated with the plurality of video cameras, and providing for said real-perspective video streams.

7. The system of claim 1, further comprising:

a plurality of infrared video cameras for placed near or collocated with each video camera used for monitoring a scene;

wherein each infrared video camera captures a pattern projected onto said scene by an infrared source; and

wherein said depth calculation processor analyzes each captured pattern by measuring pattern size on areas of said scene to calculate depth information.

8. The system of claim 1, further comprising:

a plurality of infrared video cameras for placed near or collocated with each video camera used for monitoring a scene;

wherein each infrared video camera captures a pattern projected onto said scene by an infrared source; and

wherein said depth calculation processor triangulates captured patterns on areas of said scene to calculate depth information.

9. The system of claim 1, further comprising:

a data compressor providing for a lossy compression of said video streams to reduce network bandwidth requirements.

10. A virtual telepresence system for use on the Internet, comprising:

a plurality of video cameras for monitoring a scene, and wherein each provides a respective real-perspective video stream from a unique viewpoint;

a depth calculation processor connected to each of the plurality of video cameras, and for providing a calculation of depth information for each video camera; and

a network server connected to receive each said respective real-perspective video stream from the plurality of video cameras, and connected to receive said depth information from the volume-element processor, and is such that it responds to the particular network client with only those respective real-perspective video streams from the plurality of video cameras and associated depth information that are needed to render said novel perspective selection N, and such that it responds to the particular network client with only a single real-perspective video stream from the plurality of video cameras that can best be used to render a viewpoint represented by said novel perspective selection N;

a network client that can be provided with a novel perspective selection by a user, and that provides for a transmission of information to the network server to send less than all said respective real-perspective video streams from the plurality of video cameras and associated depth information, and is provided with a novel perspective selection N that is forwarded over the Internet to the network server, and is provided with a novel perspective selection N that is forwarded over the Internet to the network server;

a renderor connected to the network client for generating a viewpoint represented by said novel perspective selection;

a data compressor included in the network client for transmitting said depth information over the Internet; and

a matching data decompressor included in the network server for receiving said depth information over the Internet;

wherein, a particular rendering of a novel perspective is left to be done by the network client and an information transfer for such is provided by the network server in response to a request.

11. A method for communicating 3D imaging information over the Internet, the method comprising the steps of:

capturing several video streams in parallel from a plurality of strategically placed cameras around a scene;

calculating depth information for each video camera; and

transmitting at least one of said several video streams and associated depth information over the Internet from a network server; and

rendering a novel perspective at a network client from a reception of at least one of said several video streams and associated depth information.

12. The method of claim 11, further comprising the steps of:

accepting a user selection of a novel perspective at said network client;

communicating said novel-perspective user selection to said network server; and

eliminating a transmission of one or more of said several video streams and associated depth information from transmission over the Internet by said network server to a particular network client associated with said novel-perspective user selection.

13. The method of claim 11, further comprising the step of:

compressing said video streams with a lossy compressor to reduce the bandwidth requirements of a network.

14. A method for communicating 3D imaging information over the Internet, the method comprising the steps of:

capturing several video streams in parallel from a plurality of strategically placed cameras around a scene;

calculating depth information for each video camera; and

compressing each of said depth information;

transmitting at least one of said several video streams and associated compressed depth information over the Internet from a network server;

decompressing at a network client said associated compressed depth information;

rendering a novel perspective at said network client from a reception of at least one of said several video streams and associated decompressed depth information;

accepting a user selection of a novel perspective at said network client;

communicating said novel-perspective user selection to said network server; and

eliminating a transmission of one or more of said several video streams and associated depth information from transmission over the Internet by said network server to a particular network client associated with said novel-perspective user selection.

15. A method for communicating 3D imaging information over the Internet, the method comprising the steps of:

providing a plurality of virtual viewpoints for monitoring a virtual three-dimensional model image stored on a server, and wherein each virtual viewpoint provides a respective real-perspective video stream from a unique viewpoint;

calculating depth information for each of the plurality of video viewpoints; and

providing a network server connected to receive each said respective real-perspective video stream from the plurality of virtual viewpoints, and connected to receive said depth information for each of the plurality of video viewpoints;

wherein a particular rendering of a novel perspective is left to be done by a network client and an information transfer for such is provided by the network server.

16. The method of claim 15, further comprising the step of:

providing a network client that can be provided with a novel perspective selection by a user, and that provides for a transmission of information to the network server to send less than all said respective real-perspective video streams from the plurality of virtual viewpoints and associated depth information.

17. The method of claim 16, wherein the network client is provided with a novel perspective selection N that is forwarded over the Internet to the network server, and wherein the network server is such that it responds to the particular network client with only those respective real-perspective video streams from the plurality of virtual viewpoints and associated depth information that are needed to render said novel perspective selection N.

18. The method of claim 16, wherein the network client is provided with a novel perspective selection N that is forwarded over the Internet to the network server, and wherein the network server is such that it responds to the particular network client with only a single real-perspective video stream from the plurality of virtual viewpoints and associated depth information that can best be used to render a viewpoint represented by said novel perspective selection N.

19. The method of claim 16, further comprising the step of:

providing a renderor connected to the network client for generating a viewpoint represented by said novel perspective selection.

20. The method of claim 15, further comprising the step of:

providing a data compressor providing for a lossy compression of said video streams to reduce network bandwidth requirements.