Three-dimensional collaboration

• US 9,595,127 B2
• Filed: 04/22/2013
• Issued: 03/14/2017
• Est. Priority Date: 12/22/2010
• Status: Active Grant

• Current Assignee: zSpace Inc.
• Original Assignee: zSpace Inc.
• Inventors: Champion, Clifford S., Hosenpud, Jonathan J., Vesely, Michael A.
• Primary Examiner(s): Hallenbeck-Huber, Jeremaiah C
• Application Number:
  US13/867,245
• Publication Number:
  US 20130234934A1
• Time in Patent Office: 1,422 Days
• Field of Search: 348/14.01, 348/42
• US Class Current: 1/1
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06F 3/03545 Pens or stylus
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06T 15/00 3D [Three Dimensional] image rendering
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2207/20064 Wavelet transform [DWT]
• CPC Class Codes: G06T 2207/30201 Face
• CPC Class Codes: G06T 2219/024 Multi-user, collaborative environment
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G06T 7/262 using transform domain methods, e.g. Fourier domain methods
• CPC Class Codes: G06V 10/245 by locating a pattern; Special marks for positioning
• CPC Class Codes: G06V 10/56 relating to colour
• CPC Class Codes: G06V 40/161 Detection; Localisation; Normalisation
• CPC Class Codes: G06V 40/171 Local features and components; Facial parts eye characteristics G06V40/18; Occluding parts, e.g. glasses; Geometrical relationships
• CPC Class Codes: H04N 13/194 Transmission of image signals
• CPC Class Codes: H04N 13/366 using viewer tracking
• CPC Class Codes: H04N 7/14 Systems for two-way working H04N7/12, H04N7/173 take precedence
• CPC Class Codes: H04N 7/157 defining a virtual conference space and using avatars or agents computer conference optimisation or adaptation H04L12/1827

Abstract

Remote collaboration of a subject and a graphics object in a same view of a 3D scene. In one embodiment, one or more cameras of a collaboration system may be configured to capture images of a subject and track the subject (e.g., head of a user, other physical object). The images may be processed and provided to another collaboration system along with a determined viewpoint of the user. The other collaboration system may be configured to render and project the captured images and a graphics object in the same view of a 3D scene.

Claims

1. A system, comprising:

a first three-dimensional (3D) collaboration system comprising;

a first display configured to project a stereo 3D scene; and

a first processing subsystem configured to;

receive processed captured images of a first subject and a viewpoint of a second user of a first graphics object from a second 3D collaboration system;

render the processed captured images of the first subject and the first graphics object in the stereo 3D scene; and

capture a 3D scene from a viewpoint of a virtual camera, wherein the viewpoint of the virtual camera is based on a position of a user input device associated with the first 3D collaboration system;

one or more first tracking sensors configured to track a six degree of freedom viewpoint of a first user associated with the first 3D collaboration system, wherein the first processing subsystem is configured to adjust a projection of the rendered processed captured images of the first subject and the first graphics object such that the projection substantially matches a viewpoint of the first user; and

wherein the first processing subsystem is further configured to provide the viewpoint of the first user, the captured 3D scene, and a captured image of the first user to the second 3D collaboration system, wherein the second 3D collaboration system is configured to;

display the captured image of the first user and the first graphics object from the viewpoint of the first user substantially concurrent to the first 3D collaboration system displaying the first graphics object from the viewpoint of the first user in a first portion of a display; and

display the captured 3D scene in a second portion of the display.

2. The system of claim 1,

wherein said rendering of the processed captured images of the first subject or a first graphics object are substantially confined to a portion of the first display.

3. The system of claim 1,

wherein said rendering of the processed captured images of the first subject are substantially confined to a first portion of the first display, and wherein said rendering of a first graphics object is substantially confined to a second portion of the first display.

4. The system of claim 1,

wherein the first processing subsystem configured to:

receive processed captured images of a third subject and/or a viewpoint of a third user from at least a third 3D collaboration system, and

render the processed captured images of the third subject and/or a third graphics object in at least a first portion of the first display.

5. The system of claim 1,

wherein the first processing subsystem is configured to adjust a change in projection of the rendered processed captured images of the first subject and a first graphics object such that the changed projection substantially matches the tracked changed viewpoint of the first user.

6. The system of claim 1,

wherein the first processing subsystem is configured to render the stereo 3D scene as monoscopic and/or stereoscopic images.

7. The system of claim 1,

wherein the viewpoint of the second user from the second 3D collaboration system is selected by the second user using a handheld device or hand.

8. The system of claim 7,

wherein the handheld device or hand comprises a control system which when activated by the second user selects the viewpoint, wherein the viewpoint defines a localized scene within the stereo 3D scene which becomes stored for later retrieval.

9. The system of claim 1,

wherein said rendering the processed captured images includes rendering the processed captured images as a texture on a flat graphic.

10. The system of claim 1,

wherein said rendering of the graphics object is according to a tabletop view.

11. The system of claim 10,

wherein to project the 3D scene, the first display is configured to project the rendered processed captured images by projecting the flat graphic coincident with a conveyed screen edge of the first display that coincides with a zero parallax setting from which the captured images were captured.

12. The system of claim 1,

wherein the first graphics object is one of a plurality first graphics objects that are within a graphic space, wherein the graphic space is a computer model of objects and where the rendering is the rendering of a subset of the graphic objects within a proximity to each other as seen from the perspective captured by the second 3D collaboration system.

13. A method for three-dimensional (3D) collaboration between a first stereo 3D collaboration system and at least a second stereo 3D collaboration system, comprising:

receiving captured images of a first subject and a viewpoint of a second user from the second stereo 3D collaboration system using the first stereo 3D collaboration system;

rendering the captured images and a first graphics object in a stereo 3D scene using the first stereo 3D collaboration system;

projecting the stereo 3D scene using a first display of the first 3D collaboration system;

capturing a 3D scene from a viewpoint of a virtual camera, wherein the viewpoint of the virtual camera is based on a position of a user input device associated with the first 3D collaboration system; and

tracking a six degree of freedom viewpoint of a first user associated with the first 3D collaboration system using one or more first tracking sensors;

adjusting a projection of the rendered processed captured images of the first subject and the first graphics object such that the projection matches the viewpoint of the first user using the first processing subsystem; and

providing the viewpoint of the first user, the captured 3D scene, and a captured image of the first user to the second 3D collaboration system, wherein the second 3D collaboration system is configured to;

display the captured image of the first user and the first graphics object from the viewpoint of the first user substantially concurrent to the first 3D collaboration system displaying the first graphics object from the viewpoint of the first user in a first portion of a display; and

display the captured 3D scene in a second portion of the display.

14. The method of claim 13, further comprising:

confining said rendering of the processed captured images of the first subject or a first graphics object to a portion of the first display.

15. The method of claim 13, further comprising:

confining said rendering of the processed captured images of the first subject to a first portion of the first display; and

confining said rendering of a first graphics object to a second portion of the first display.

16. The method of claim 13, further comprising wherein:

receiving processed captured images of a third subject and/or a viewpoint of a third user from at least a third 3D collaboration system using the first processing subsystem configured, and

rendering the processed captured images of the third subject and/or a third graphics object in at least a first portion of the first display using the first processing subsystem configured.

17. The method of claim 13, further comprising:

rendering the stereo 3D scene as monoscopic and/or stereoscopic images using the first processing subsystem.

18. The method of claim 13,

wherein the viewpoint of the second user from the second 3D collaboration system is selected by the second user using a handheld device or hand.

19. The method of claim 13,

wherein the handheld device or hand comprises a control system which when activated by the second user selects the viewpoint, wherein the viewpoint defines a localized scene within the stereo 3D scene which becomes stored for later retrieval.

20. A system, comprising:

a first three-dimensional (3D) collaboration system comprising;

a first display configured to project a stereo 3D scene; and

one or more first tracking sensors configured to track a six degree of freedom viewpoint of a first user associated with the first 3D collaboration system; and

a first processing subsystem configured to;

receive processed captured images of a second user, a captured 3D scene, and a viewpoint of the second user of a first graphics object form a second 3D collaboration system, wherein the captured 3D scene is from a viewpoint of a virtual camera of the second 3D collaboration system, wherein the viewpoint of the virtual camera is based on a position of a user input device associated with the second 3D collaboration system;

render the processed captured images of the second user and the first graphics object in the stereo 3D scene according to the viewpoint of the second user in the first portion of the display; and

render the captured 3D scene according to the viewpoint of the virtual camera in a second portion of the display.